Will Webberley's Blog

Cross-publishing web content to Gemini

Thu, 01 Jun 2023 21:38:00 +0000

My personal website is generated using Hugo, which allows me to write nearly all of the actual content itself in plain markdown files.

I also maintain a Gemini capsule (hosted at gemini://wilw.capsule.town). For a while I’ve wanted to be able to add more content to this capsule, and to try and keep it updated more consistently over time. However, I don’t really have the capacity to duplicate the time taken to maintain the site (and its blog posts and notes) in order to do so.

Whilst Gemini text format is not markdown, it is similar enough that I wondered if I could script a simple translation flow to turn my *.markdown files into valid .gmi ones.

At the end of the day, it is up to the Gemini clients themselves to decide how to render content, and so the main thing I really needed to handle were links (which can’t be rendered in-line in Gemini) and images (which cannot be displayed).

Using the python-frontmatter package, I was able to write a quick builder script which now helps to populate my Gemini capsule.

The script creates new Gemini files for each of my blog posts and notes, and also generates index files to allow navigation to this content. In each file, links are extracted out of the main text, and replaced with indexed references (e.g. [3]), which are then placed as valid Gemini links after each paragraph.

If an article has links referencing another blog post or note, then the links are generated using the gemini:// protocol, allowing the visitor to stay in Gemini as much as possible.

The script is executed as part of my Woodpecker CI pipeline to automate the generation of the Gemini capsule on each git push. The output is then automatically deployed to gemini://capsule.town in order to make the changes live.

It’s great knowing that I can now keep the capsule up-to-date as part of my usual workflow.

CI/CD with Woodpecker and Gitea

Sun, 23 Apr 2023 16:51:00 +0000

Some of my personal projects are beginning to get larger and more complicated, often involving different front-ends and services that all need to be separately built and deployed. Managing all of these is taking away more of my personal time, and I’ve been on the look-out for a good CI/CD automation system for these projects. I primarily use Gitea as a git server, and have been struggling to find a system that suited my needs and works well with Gitea.

I know of Jenkins’ Gitea plugin, and other similar alternatives, but many of these tools are ones I’ve tried in the past and didn’t get on with for one reason or another.

There is also the upcoming Gitea Actions feature, which aims to provide similar functionality to GitHub’s own Actions. However this is still experimental and my early tests did not (yet) allow me to settle on this reliably.

More recently I stumbled across Woodpecker, which has in-built support for Gitea and a super-easy setup (as I briefly document in this Woodpecker note).

Once Woodpecker is running, one can simply login with single sign-on (SSO) provided via the configured Gitea instance. This then allows Woodpecker to automatically fetch your repos and, when attached, create the required webhooks.

Adding repos to Woodpecker is straight forward through the web UI, and each repository can be configured with secrets (e.g. Docker credentials) and any extra settings. Setting up the build pipeline instructions themselves is as simple as creating a .woodpecker.yml file in your project root, following the simple syntax.

I now use Woodpecker for a number of projects and services, and for a variety of tasks – including building and deploying websites/apps, Docker images, and even Flutter apps. For interest, the pipeline for this Hugo website can be viewed here.

From all the continuous development options I’ve tried, Woodpecker is certainly my current favourite, and I feel it is one that will stick. I find the UI simple and intuitive, and also blazing fast, with updates showing in real-time.

The build speeds themselves are, of course, only as quick as the underlying resources allow. Currently I run Woodpecker on a dedicated VPS with 4GB memory, and this performs fine for my needs.

🤘

Thoughts on book logging

Fri, 24 Mar 2023 18:24:00 +0000

I’ve been a member of Goodreads since 2013. I follow a few of my friends and family on there, and whilst it was nice to see the types of things people are reading, I only really ever used the service as a way for logging what I had read. The other social aspects didn’t keep me coming back and I personally didn’t find the home feed interesting.

As I started to get more into the Fediverse back in 2021, I joined BookWyrm (as @wilw@bookwyrm.social). I was able to export my reads as a CSV from Goodreads and import them into BookWyrm, which I continued to use as a method for recording reads and listens. BookWyrm is excellent, easy-to-use, and less clunky than Goodreads. I can certainly recommend it if you’re looking to join or build a community around books.

However, as with Goodreads, I just never really seemed to get involved with the other features or engaged with people on the platform. Like most Fediverse services, BookWyrm is primarily designed to be social. People discuss books, write reviews for others, and get inspiration from the people in their community.

I recently came to the realisation that reading doesn’t feel like a social activity to me. One of the great things about books is that they are inherently personal – fiction provides an escape from reality and evokes your own imagination, and other works can (among other things) help to educate and improve your life.

I do enjoy discussing books with others from time to time (and usually in person) but, for now, all I really needed was a simple method for logging reads and listens.

As such, I now simply include a log on my books note. There’s not much to it, but it helps me keep track of what I am reading and the frequency. To create the log, I include a CSV section in the note (which I can easily add to) and I use a shortcode (based on the very helpful work by Brian Wisti) to get Hugo to render a table from the CSV content.

A question about encryption for self-hosting

Sun, 05 Mar 2023 09:25:00 +0000

I self-host a number of services - Nextcloud, FreshRSS, PhotoPrism etc. - at home on a Raspberry Pi. Attached to this Pi is a large SSD to hold the service data and configuration, and all of this is periodically backed-up via Restic to a remote site.

The SSD is simply formatted with ext4, and the directory containing all of these services and data is currently encrypted using fscrypt. I (mainly) want to encrypt the data in order to protect against the case of break in and theft in my house, however likely or unlikely that is to occur.

However, the downside of this mechanism is that each time the power drops in the house (a fairly frequent event these days with constant building work going on), the directory is unable to auto-decrypt and so the services do not launch properly until I manually unlock the directory using a passkey.

Whilst this works fine, the approach feels a bit clunky. I wondered if anyone had any alternative solutions to managing this. I’m certainly not an expert in encryption mechanisms, but below I outline three approaches I’ve wondered about.

Option 1: fscrypt PAM decryption

I notice from the documentation that directories can be automatically unlocked at login-time through PAM. Is there a secure way to auto-login a user after boot (and drive-mounting) such that the directory is ready by the time the Docker daemon restarts the service containers?

Option 2: Full-disk encryption with LUKS

In the past, I’ve used LUKS to encrypt entire volumes, and crypttab to handle the unlock automatically at boot-time. I could use this to encrypt the entire volume on the SSD.

My concern with this is that (in my own way of thinking about it!) the encryption key must be stored unencrypted on the root/boot filesystem in order to facilitate the automatic unlock. If the device was to be stolen, this key would be available to anyone checking the filesystem. I may be misunderstanding the encryption approach here, or perhaps over-estimating an average thief’s technical ability.

Option 3: Resistance to power drops

My final thought on this is to address the root cause of the issue, and to find a solution that keeps the Pi booted when there is no mains power. For example, this could be accomplished with a UPS (Uninterruptible Power Supply).

My ask

I understand that this must not be a rare problem. If you’ve had similar experiences and have found a solution that works well for you, I’d love to hear from you!

Please reach out to me on Mastodon (@wilw@fosstodon.org) or via email (blog@wilw.dev).

Window management workflows on macOS

Sat, 10 Dec 2022 15:40:00 +0000

macOS window management

Every Mac user seems to have a different way of managing their open applications and windows.

Some people prefer to view each window in “full” mode, in which they take up the entire display and the user can cycle apps or use the dock to change the active window. Other people use full-screen mode and/or swipe between desktop spaces to find their apps, or a mixture of several approaches.

Personally, I only tend to use one desktop workspace and I keep all my application windows open, which I click or cycle between with ⌘-tab (or ⌘-` for windows in a single application). At any one time, I usually (at least) have open Mail, Calendar, Spotify, Firefox, Nova, Slack, Teams, and Sorted. And although I do minimise windows (to the dock) occasionally, things can get messy very quickly.

Some may think I’m crazy, but I’ve lived like this for over a decade now, and things have been mostly fine so far. However, I’ve recently noticed that app windows behind my currently-active one can get distracting (for example, new Slack messages, or emails arriving), causing me to lose focus on my current task.

The visual complexity of the screen may also be having a toll on my ability to concentrate or find the things I’m looking for, especially since the visual hierarchy of the current window is interrupted by everything else going on around it.

The window-hiding workflow

These factors weren’t anything I was particularly concerned about, but a semi-recent episode of the Accidental Tech Podcast, which I frequently listen to, talked about this exact problem. The hosts discussed the concept of “hiding” windows as part of their normal workflow.

Window-hiding in itself isn’t a new concept to me, and I even knew about the cmd-H keybinding before hearing the episode. However, listening to how people use this feature as part of their usual workflows really interested me.

The general idea for this type of workflow is that, once a task is completed (e.g. choosing a new Spotify playlist or sending an email), one “hides” the app until it is needed again. This can be achieved by using the cmd-H keybinding (which hides the app and automatically switches the focus to the next-most recent app in the stack) or option-clicking another window (which leaves and hides the current app).

The effect of this workflow is that one only has a single (or very few) number of application windows showing at any one time. When you need to re-show a hidden app, you can just cycle to it in the app-switcher or select it from the dock.

I’ve been using this workflow now for a few weeks, and have noticed a big improvement in my focus on tasks that, when combined with an effective to-do list system, has had a big positive hit on my work effectiveness.

Earlier in the year I blogged about how I spend more time working on iPads these days, which is partially down to a preference of the single-screen context afforded by iPadOS. I can certainly see why I enjoy replicating this process on macOS too.

Addendum: Stage Manager

Stage Manager was recently released along with macOS Ventura. I haven’t given this a proper go yet, but I plan to soon since it looks like workflows based on this could also help boost productivity and focus.

From Google and Apple Photos to Photoprism

Fri, 25 Nov 2022 19:35:00 +0000

A history of media storage solutions

Back in 2021 I blogged about how and why I wanted to switch from Google Photos as a storage solution (and source of truth) for my life’s photo and video library. The post compared several solutions, such as Piwigo, Mega, and Nextcloud.

In that time I’ve tried several further options, starting with pCloud (as described in that post), Nextcloud backed by S3, and plain-old Apple Photos.

The primary way by which I capture photos is through my phone’s camera, and my main issue with pCloud was the reliability of its mobile app photo auto-uploader. It would often get logged-out and miss several days or even weeks of sync, without any notification. After logging-in again, the app offered no way to catch-up specifically on the missed syncs, and so it would involve having to manually fill-in the gaps myself.

Nextcloud is an excellent option for self-hosting files, and I make strong use out of this myself. However, I found it a bit slow and clunky for managing images and videos. It would often grind slowly to the point of unresponsiveness whilst it performed background tasks - I imagine related to indexing or creating thumbnail previews of media content. Its image viewer also isn’t as useful as other systems.

I was sinking a lot of time into this, and so in the end I gave up and resorted to simply using Apple Photos - it’s easy, cost-effective, and naturally works well across all of my Apple devices.

Apple Photos

However, I couldn’t sit still, and I was constantly worried about the single point of failure in having everything stored in a single “basket”. What would happen if I lost access to my Apple account? How can I make sure things are backed-up? What if I chose to move away from the Apple ecosystem?

For a short while I made use of Backblaze’s Mac backup solution to try to backup my photo library on my Mac. However, despite having configured the Photos app on my Mac to download the full versions of everything in the library, the file size differences in the photo library file on my Mac vs. that reported by iCloud on the Mac (and the storage settings panes on my iPhone) just didn’t add up. I also recently had similar issues with the opaqueness of iCloud in a slightly different context.

Either way, I was worried I was missing something and that the backups would not be complete. Also, the solution depended on my Mac being awake and receiving the photo sync in order for its local library to be successfully backed-up with all of the recent changes.

Photoprism

More and more I’ve heard talk on the grapevine about alternative self-hostable options for photos and media. Some of these can reportedly perform super well, including the Google Photos-like Immich.

Photoprism is another option that had come up a few times, and I resonated well with its feature list. It supports face and object recognition, automatic categorisation, favourites, moments, albums, and much more. It also has a rich suite of settings and advanced import capabilities.

The disadvantage with self-hosting a service like this is the amount of storage required. Images and videos aren’t small files, particularly when taking into account the quality produced by modern smartphones. I alluded to this back in my blog post from last year, but since then my mindset has shifted much more to prioritising data sovereignty over ease of use and surrendering privacy.

I deployed Photoprism and documented my setup in this note. I’ve now been using it successfully for a few months and have really enjoyed it. My library backs-up nicely with Restic to Backblaze B2 and everything feels smooth and safe. The app is only available via my Tailscale network for extra security and the storage volume is encrypted.

My library is over 200GB big, and so I needed a hefty volume attached to my VPS. I also wanted a machine powerful enough to process the images and videos, and to ensure things didn’t become too slow when scrolling through thousands of images. As such, deploying a service like this is certainly not the cheapest option, but it is one that works well for me.

For automatic photo sync from my iPhone I use the Photosync app. This has been reliable and I can certainly recommend it. I’ve not had a single upload failure during the months I’ve been using the app. My setup for this is also included in that same note.

Photosync also supports pCloud as a sync target, but having tried Photoprism I haven’t looked back - its experience is far superior.

So far, this has been my most positive photo management experience, and I am really happy with the setup. To me, it is more fully-featured and much more performant than Google Photos. I saved the Photoprism PWA to my phone’s homescreen for quick access to my photos.

Here are a few of the other features I enjoy about Photoprism:

Speed: I find it super fast to use and navigate around. Images and videos load quickly. The webapp loads more quickly than the native Apple Photos app does.
Apple live photos are supported out-of-the-box and the UI for viewing these is better than in Apple Photos.
The libary review process is great for ensuring that only the photos I actually want to keep make it into my library. Photoprism is good at detecting the types of files that should be reviewed.
Photos marked as “favourites” on Apple Photos are automatically reflected into the Favourites album on Photoprism.

Having the images stored safely on a volume I have direct access to means I can pretty easily switch to a new self-hosted app if I wanted to without a tedious export from a commercial system. I don’t know if this is the end of the road (yet) for my photo management solution adventure, but it’s definitely in a good place for now!

Tailscale: multi-service HTTPS on a single machine

Thu, 27 Oct 2022 18:48:00 +0000

HTTPS on Tailscale

Tailscale’s HTTPS feature is an excellent tool for adding TLS connections to web services exposed over the tailnet.

Although traffic over the tailnet is encrypted anyway due to the nature of Tailscale itself, some web-based services work better when served over HTTPS. Since the browser does not know that you are accessing the service over a secure connection, it may enforce limits on connected web services when accessing them in - what feels like - an insecure context.

Enabling HTTPS for Tailscale machines is as simple as running tailscale cert on each machine you want to issue a certificate for. This command will cause Tailscale to request a TLS certificate using Let’s Encrypt for the machine and to write the certificate and key file to the filesystem.

In an ideal world, this is great, since you can then just plug these files into your reverse-proxy in order to add HTTPS capability to your web service.

Limitations in multi-service contexts

However, Tailscale can only issue one certificate per machine. If your tailnet is named “my-tailnet.ts.net” and one of your machines is called “machine1”, then running the command on this machine will generate a certificate for “machine1.my-tailnet.ts.net”. Since Tailscale doesn’t (currently) support machine sub-names (e.g. for “app1.machine1.my-tailscale.ts.net”), then it becomes tricky to use a single reverse-proxy to terminate TLS for several services at once from the single machine.

Reverse-proxy web servers typically support routing based on host (i.e. subdomains), path, headers, and more. But in this case, host-based routing wouldn’t work since there is only one host name for which the certificate is valid. Path-based routing can become a nightmare to manage (and many services do not expect to be served from a non-root path), and header/parameter-based routing is not accessible at all from a number of devices (e.g. mobile browsers).

My experiences

I host a number of personal services on a single VPS, which is a machine on my tailnet. Generally, I do not bother using TLS for these services, since they are happy to be served over plain old HTTP on a port I assign them, and the traffic is secured anyway by Tailscale itself.

However I recently started using 2FAuth as a self-hosted alternative to Authy, and while 95% of it works fine using my normal pattern, browsers block the camera (for scanning QR codes) when working in a non-HTTPS context. Since this is (for me, at least) a crucial part of the 2FA setup experience, this became a major blocker.

I do not want to expose this service outside of my tailnet and so I needed to find a method for serving it over HTTPS using the Tailscale certificate approach.

One option would be to spin-up a new instance (and Tailscale machine) dedicated to the service, but this would not be worth the cost for such a small service. Alternatively, I could make this the only HTTPS-served service on the machine, in which case 2FAuth would be the sole user of the certificate issued for the entire machine. This feels like a dirty and non-scalable approach; for example, I may have other services in the future that also rely on an HTTPS connection.

The solution I went with in the end depends on the fact that the browser doesn’t care which port secure traffic is served from on the machine. In particular, this means that the machine can use the same Tailscale-issued certificate to expose HTTPS traffic simultaneously on ports 8001, 8002, 54321, or anything else. In practice, this would typically involve using a deidcated reverse-proxy server for each service that requires HTTPS, and with each server being provided with the same key and certificate.

To make this work in my case, I simply stuck an nginx container, provided with the certificate and key, in front of the service, as documented in this note. For another service, I would use an additional nginx container, mount the same certificate and key, and then that service would also be served over HTTPS - just on a different port.

To keep things neat and network-separated, I would include the Nginx container definition in the same docker-compose.yml file as the service itself, as also documented in the same note.

Conclusion

I don’t know if this is the most elegant or resource-effective solution, but it is one that works for me for now. I hope that, eventually, Tailscale will be abe to support sub-names for machines in order to improve this experience.

If you’ve come across this issue yourself and found an alternative/better way to manage “virtual hosts” on a single Tailscale machine, then I’d love to hear from you.

Note-taking apps: Bear and Joplin

Sun, 02 Oct 2022 19:55:00 +0000

The Bear notes app has been my go-to notes app for Mac, iPhone, and iPad for some time now. It’s got a great UX, a customisable UI, and is one of those apps that feels like a (clichéd) “delight” to use.

Some of my gripes with Bear (and CloudKit in general)

Bear is written exclusively for Apple devices, and uses CloudKit to sync notes between devices via iCloud. In theory, this isn’t too much of a problem. However, I’ve recently found CloudKit-reliant apps to become a little unreliable.

Frequently, I’d start a quick note on my phone and the changes wouldn’t come through to my Mac in any reasonable time for me to continue writing. I don’t know if this is a limitation with CloudKit or a factor with the Bear app itself, but I find that many apps built on CloudKit tend to not display any kind of progress bar or confirmation that things have been successfully synced.

Also, despite things being “securely stored in my iCloud account”, there seems to be no way to actually view the notes themselves. As with all CloudKit apps, they take up space in iCloud, but without any meaningful way to manage this storage.

iCloud preferences list it as an app using iCloud, but that’s pretty much the limit, as far as I can see.

I’m sure this level of opacity is designed to make things easier for the user (and probably safer!), but I personally find that it makes it hard for me to reliably organise it alongside the rest of my digital life.

Furthermore, in Bear, notes are categorised by tags rather than into a folder-like hierarchy. Similar to the Gmail approach to using labels instead of traditional IMAP-style folders, this allows notes to sit in multiple tag “folders” at once. As such, I can imagine it is tricky for the Bear developers to represent this sensibly in a tree. Indeed, when exporting Bear notes, they are exported as a single flat directory of files.

This is different to other Notes apps I’ve used in the past, such as Obsidian and SimpleNote, which are perfectly happy to directly use the filesystem as a data store. However, I do understand where syncing this with iOS could pose difficulties.

This approach facilitates app “lock-in”, since I can only ever use Bear (and its “Pro” plan at that) in order to continue to sync and edit the notes, since exporting them loses the “folder” hierarchy.

Despite all this, Bear is certainly a great app, and I have no problems recommending it to Apple product users. I just had a feeling that a different solution might work better for me.

Why I prefer Joplin

Joplin is an open-source note taking app available on a number of platforms.

It’s not quite as slick as Bear, but Joplin is still quick and easy to use and it maintains notes in a sensible hierarchy (divided into “notebooks”), whilst also supporting tagging. It is also much more configurable, with lots of app styles, plugins, and more. Exporting Joplin notes using the app also maintains the folder structure.

Joplin supports a number of sync targets, such as Nextcloud, WebDav, and S3. It also supports Joplin Server - a self-hosted server option. This is what I use and I’ve noted my setup. I access and sync my notes securely via Tailscale and back them up to Backblaze B2.

The Joplin app also features a sync progress and success indicator for that extra peace of mind 😀.

Both Bear and Joplin are great note-taking experiences, but I feel safer with Joplin and it meets my current needs more.

Re-Building my Website with Hugo

Sat, 03 Sep 2022 15:52:00 +0000

GatsbyJS

For several years I’ve been using GatsbyJS to generate the static site content for this website. Gatsby is a great tool and produces blazing-fast websites through the use of an interesting combination of technologies.

In Gatsby, pages are simply React components, and developers can make use of the entire JavaScript and React ecosystems to craft their sites. Config files can be used to create pages that don’t “exist” in your filesystem (e.g. an index page for each tag used in a blog) and GraphQL queries are used to surface content and query data from across the website. Gatsby templates and standard React composition patterns allow for excellent re-use of components.

When building the site, Gatsby produces standard HTML files for each of your pages, such that they can (in a limited sense) be accessed by JavaScript-less browsers. When JavaScript is enabled on the browser, Gatsby pre-fetches content automatically in the background and routes requests internally to make navigating through the pages extremely quick.

Gatsby has served me well for a number of years and I’ve written a number of posts on this blog around tweaking and enhancing setups to give even better experiences.

Why consider a move?

For a small personal website, like mine, however, Gatsby felt a bit overkill. I shouldn’t need to write JavaScript to present simple static content, and dealing with the JS dependency graph and a huge default node_modules/ is a little like hitting a nail with a sledgehammer.

When updating Gatsby or any other dependencies, I’d always end up fighting against issues with incompatibilities, taking my time away from actually producing any content. GitHub’s Dependabot alerts would email me frequently about insecure packages somewhere in the graph, which I wouldn’t mind so much if I wasn’t simply needing just a small bunch of HTML files.

I find the Gatsby config files hard to understand, read, and write. I’d often end up blindly copying bits from other examples without really understanding what they were doing. The same applies to the GraphQL queries; I never really got my head around these in the Gatsby context, and I ended up reducing the site’s functionality because I couldn’t achieve what I wanted to. Sure - I could spend more time learning about these concepts but I just couldn’t bring myself to do so when my target needs of the system are so simplistic.

As the site got bigger I found that the build times also increased. Towards the end of my usage, the site would take 20-30 seconds to boot the hot-reloading develop mode, and building the production-ready site would often take a good minute or more. I use Vercel to host the site, and so I imagine it has to run a yarn install each time it builds too.

When in production, the site’s assets - whilst not huge - would be pretty sizeable given the actual content being displayed. Visitors shouldn’t need to load a React runtime just to view the homepage (though I do understand that the JavaScript components are there to improve the quality of the UX).

All in all, I felt that Gatsby was too big/heavy/complex for my needs, and with too many overheads in terms of up-keep and maintenance.

Switching to Hugo

Hugo has been on my radar for some time now, but I never got around to giving it a go. I understood its aims and characteristics (especially given my past experience with Jekyll), and it always seemed as though it would be a good fit for my needs.

Given my growing concerns about Gatsby, and having a couple of days of work downtime last month, I thought I’d give Hugo a proper try by attempting to port my personal website over to it.

I began by installing the hugo tool and creating a new site according to the documentation. I knew that Hugo content is mostly written in markdown, and since my blog posts and notes were already in markdown from their Gatsby days anyway, I could copy them straight over into the relevant directories under content/ in the new Hugo site.

Although there was now content in the site, Hugo would still render blank pages. This is because I didn’t have a theme configured. I had a quick look through the available themes on the website, but none of them really took my fancy. Instead, I started to build out my own basic theme by creating HTML files in the layouts/ directory.

My goal was to try and replicate my existing site as much as possible. Partly because this would be quicker (I wouldn’t need to come up with new designs and layouts), but also because I like its simplicity.

I gradually migrated all of my content over, making use of content organisation, indexes, layout files, and simple styling. The documentation is easy to follow and very intuitive. There’s no GraphQL to learn - it’s all just basic HTML, markdown and some simple Hugo functions.

Before long, I had the whole site copied over and built, and I also thoroughly enjoyed the process. I use none of my own JavaScript (aside from a couple of pieces I include, for which I’ll justify below), and so everything feels much simpler and cleaner. I now look forward to adding more content and continuing to tweak the site going forward.

Things I love ❤️

Here are some of the things I’ve found to love about Hugo:

Partials - like React components, these are great for re-using layouts and markup across a website.
Pipes - I use these for processing and minimising SCSS, and also for resizing and manipulating images.
Configuration - Super simple, from a single file. Changes automatically get reflected in the site when hot-reloading.
Speed - Dev environment boots in milliseconds, and the whole production site builds in <500ms.
Simplicity - No dependencies, other than the Hugo binary itself.
Content-focused - Super quick and easy to add new content, rebuild, and deploy.

The minimal styles, plus the fact that I use generally-available system fonts and scaled images (thanks to Hugo pipes) mean that the total size of the homepage, when downloaded and uncompressed, is less than 20kB.

This means that the site is now a Green Team member of the 512KB Club, and most of its pages are also measured to result in the production of extremely low CO2 per visit. Some included JavaScript on the page re-measures every now and again to keep this up-to-date.

I look forward to building more with Hugo and the community, and perhaps working on and releasing a theme sometime.

Making Tax Digital and Plain-Text Accounting

Sat, 30 Jul 2022 22:25:00 +0000

HMRC’s Making Tax Digital

Making Tax Digital (MTD) is part of the UK Government’s plan for modernising the tax system for both businesses and individuals.

For years, HMRC (the Government’s tax department) has had an online tax system that is infamously complicated and slow to use and update such that even accomplishing simple tasks can be long and painful processes. Part of this is due to the laughably complicated UK tax system itself (rather than the fault of the technology), but some of it can certainly be attributed to the antiquated tooling.

The main idea behind MTD is that tax-payers make use of accounting software - such as Xero or Quickbooks - to automate their tax returns. These commercial systems are often more nicely-designed and well-maintained than HMRC’s own, and so it does make sense to use them if you are looking for a more pain-free tax experience.

The positive effect of using such a tool is also amplified if you already use one of these systems for your bookkeeping anyway. In these cases, the accounting system is naturally aware of all your transactions and tax liabilities, such that submitting returns via MTD can be as simple as a couple of button clicks.

If you’re not currently using this types of software for bookkeeping (for example, if you’re a smaller business just using some simple spreadsheets), then you’ll have to change your business processes. And since the MTD-compatible tools mostly seem to be commercial SaaS systems, this will likely come at an additional cost.

The impact on plain-text accounting

Those people that follow this blog will know I’m an advocate for the simplicity, flexibility, and transparency provided by plain-text accounting systems - in particular, those compatible with Ledger and its peers.

MTD-compatible software must be approved by HMRC, which results in a walled garden and restricts people’s freedom. I don’t know what the approval process involves, but I would imagine successful systems would include some form of integration secret or private authentication mechanism with HMRC. Such arrangements don’t lend themselves well to distributing FOSS or self-managed tools since they would depend on the end-user managing the secrets themselves.

As such, I fear that the introduction of MTD will rule-out a swathe of plain-text methods and open-source tools for those doing their accounting and bookkeeping in the UK. Sure, one can use Ledger to periodically export CSV data and import this into an “approved” system for completing a tax return, but do people really have time for this? And why should they continue to use the non-approved methods when they already have to pay for the MTD-compatible ones?

I am fully on-board with modernising the tax system, but I feel that if this was instead achieved through providing open APIs (authenticated with OAuth2 or similar and without the requirement for app-wide secrets), then it would enable the open-source community to develop systems and tools to compete with the commercial offerings and help to keep things fair for everyone. Governments mandating the use of specific for-profit companies for non-optional activities just doesn’t feel right. Many people in the UK are still wary of this after the Government approach to the COVID pandemic, with dubious contract awards for PPE and testing.

Making Tax Digital is still relatively young. In fact, as far as I know, it’s only currently mandatory in some sitations relating to VAT for businesses. However, there is an objective to gradually roll it out more comprehensively for business and individual tax over the coming years, and I really hope that the HMRC APIs can be developed in a fair and non-exclusive way.

Switching-up my workouts 🏋️‍♀️

Wed, 20 Jul 2022 19:56:00 +0000

Starting to workout

I’ve worked out every day - pretty much - now since March 2020. The only days I’ve missed were because of illness (Covid), or another reason that made it physically (e.g. my dislocated shoulder) or logistically (e.g. during travel) impossible.

These workouts have pretty much always been “at home”, or wherever I happen to be staying at during the time. They started during the first wave of UK Covid lockdowns, in which I realised I was not getting the exercise I was previously used to when walking the 3 kilometres daily to work and back.

My brother happened to be living with me at the time, and he had recently got into using the Fitbod app to help guide his exercise. He recommended it to me, and - since I happened to have a pair of dumbells at home - I started giving it a go.

The app lets you configure your workout goals (e.g. muscle growth, toning, etc.), frequency, and duration. You can also provide it with the types of equipment you have available and any exercises you want to avoid (for example, lunges are hell for my knees, so I avoid these at all costs). Once setup, it creates a workout regime for you, targeting different muscle groups whilst respecting the muscle recovery. Depending on your goal, the app also automatically adjusts the intensity/weight/duration/reps of your exercises as it learns what your current limits are.

Fitbod keeps things varied, and so I ended up using it daily for over 18 months. I learned a lot about different types of exercises, and how these target the various areas of the body. When I was away from home and the weights, I configured workouts that would rely only on body weight or strap workouts.

On a personal level, I also noticed hugely positive changes in my body shape. My increased fitness also resulted in better sleep and a more positive daily existence. This all happened more quickly than I had anticipated, too, and I soon got to the point where my body felt like it was almost protesting its own laziness on those rare days I was unable to exercise.

Change of routine

When I dislocated my shoulder back in October 2021, I had to stop working out for a short while. Even though it was only a shoulder injury, my arm was in a sling which made any form of exercise virtually impossible. After a few weeks, I could lose the sling, but I still couldn’t really exercise it whilst things healed.

Fitbod didn’t have the capacity to understand shoulder injuries, and so ploughed on in reminding me to workout every day and suggesting now-impossible workouts. It was frustrating, so I uninstalled the app with the aim of re-downloading it at a later time.

I was soon able to introduce small exercises every day. For example, some core exercises and squats. I used my learning from all the types of workouts Fitbod had introduced me to in order to create my own balanced workouts that worked around my needs as my body got better.

During the weeks and months in which I recovered, I gradually built up my own workouts further, and I realised that I didn’t need Fitbod anymore. I had enough of my own momentum to ensure I kept up the daily routine, and I was able to choose the exercises and routines I enjoyed the most, whilst respecting my muscle recovery.

I’m not a serious weightlifter or gym-goer, but I’m now at the point where I am confident in selecting the workouts that are right for me, without needing to rely on being told what to do by an app. I take inspiration from friends who suggest new exercises, and I shake things up myself to keep things interesting. I still notice progress and I think I’m at the point now where I mainly just want to maintain my level of fitness and body shape.

A changed person? 🤔

If you had said - five years ago - that I would now be the type of person that enjoys and actively works out every day, I (along with my friends, family, and others that know me) would have laughed at you. It’s amazing how much I feel I have changed over the last two years.

This has also come with a greater understanding of my mental health and wellbeing, and how to help manage these aspects of myself. Altogether, I definitely feel that I am much better equipped to develop and adapt as life goes on. ✌️

My setup for coding on iPad

Sun, 12 Jun 2022 11:42:00 +0000

Since getting a Magic Keyboard for my iPad Pro, I’ve been using the iPad for many areas of work for which before I would have needed a laptop.

In fact, last week I was able to use the iPad full-time when at work in our new office space, and I didn’t need to reach for my MacBook once. When I can, I prefer working on the iPad due to its flexibility, brilliant display, speed, battery life, and more.

I also prefer working in the single-context view iPadOS provides. On a Mac I would have dozens of windows open at any one time, offering lots of distraction. On iPad, I generally use one screen at a time, allowing me to keep focus. In in-person meetings, the iPad’s form-factor also feels less of a psychological barrier between myself and my colleagues. In video meetings, the iPad also holds up well, and its screen-sharing functionality is fantastic.

My role has changed in recent years and, as a result, I do less coding in my day-to-day work at my job. However, this is one area where native iPad working does fall down, since the system does not offer native command line tooling or the ability to install and run development software. During the past week there were a couple of times where it would’ve been useful for me to contribute some code with my team and - not wanting to have to return to the MacBook just for these moments - I spent a little time setting-up a development environment on my iPad.

In this post I will talk a little about my approach to setting-up a development workflow from my iPad, using a VPS, Tailscale, Termius, and Code Server.

✌️ As a bonus, all of the steps in this post can also be completed on an iPad. This post was also written, and the site was built and deployed, from my iPad.

1. Provision a VPS

Since the iPad itself does not have the tooling needed to write, run, and build applications, I provisioned a Linux cloud server that would do most of the heavy lifting.

I spun up a new Ubuntu 22.04 server on Linode with sufficient resources needed to run and build my apps. I went for 4GB memory and 2 vCPUs, but Linode lets you change this later if you realise you need less or more resources. My intention is to use this machine solely for development work.

2. Configure Docker and initial server set-up

The next steps were to access the new server and set-up the environment.

Termius is an excellent app for managing remote SSH and file-transfer sessions, and it’s even better on iPad with Magic Keyboard. It’s free to use but their premium subscription is worth it too.

Using Termius, I connected to the new server (using my SSH key) and firstly installed Docker and Docker Compose (which I will use to run the Code Server instance):

# apt update
# apt install docker docker-compose
# service docker start

Next, I created a user that I would use for management and for running the Docker containers:

# useradd will # ... go through setup
# usermod -aG docker will

I recommend editing /etc/ssh/sshd_config to harden your SSH setup, restrict root login and only permit key-based authentication. Don’t forget to restart the SSHD service after changing the settings. You should also add any needed SSH keys to your non-root user’s home directory so that they can login later.

3. Configure Tailscale and firewalls

Tailscale is a fantastic service that allows you to setup a VPN mesh network for all of your devices, based on the Wireguard protocol. By leveraging Tailscale, we can ensure that access to the server is restricted only to devices you’ve added to your Tailnet.

First, create an account with Tailscale if you haven’t got one already (it’s free for up to 20 devices!) and then install Tailscale on the server:

# curl -fsSL https://tailscale.com/install.sh | sh
# tailscale up

Follow the instructions to authorise the machine with your Tailscale account, and then you’ll be able to see the server listed on your Tailscale’s account webpage.

Next, ensure you have the Tailscale app installed and connected for your iPad. Your iPad should then also become listed on the Tailscale website.

Finally, configure your firewalls to block all public access. If you use Linode, you can make use of their cloud firewall service to drop all inbound traffic.

Finally, verify that you can still connect to the development server by modifying your VPS IP address in Termius to the Tailscale one shown on the website (or the DNS name if you’ve enabled MagicDNS!).

You can now be sure that only your authorised Tailscale devices can talk to your server.

4. Run Code Server

Use Termius to log back into your development VPS with your non-root user. We’ll now set-up Code Server on the server.

Code Server is a web-based implementation of Microsoft’s VSCode. linuxserver.io maintain a community Docker image, which is what I use.

Create a docker-compose.yml file on your server with these contents:

version: "2.1"
services:
  code-server:
    image: lscr.io/linuxserver/code-server:latest
    container_name: code-server
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
      - PASSWORD=password
      - SUDO_PASSWORD=password
      - PROXY_DOMAIN=development
      - DEFAULT_WORKSPACE=/config/workspace
    volumes:
      - ./code:/config
    ports:
      - 8443:8443
    restart: unless-stopped

Change the values for the password (used to access the Code web interface) and the sudo password (which you can use to add extra packages to the container later).

When happy, bring up the Code Server with docker-compose up -d. After a few seconds, you should be able to visit your Code Server instance in Safari by navigating to the Tailscale IP address (or MagicDNS entry) on port 8443.

You can now use Code in much the same way as you would with the desktop app.

5. Post-setup tasks

Whilst you can now write code via Safari on your iPad, extra setup will be needed to actually do anything with the code.

When managing the Code Server container, I recommend simply opening a Terminal window from within the Code Server web interface itself. You can use the SUDO_PASSWORD you configured earlier for tasks that require root access.

For example, to set up remote git properly, you’ll want to create a new SSH keypair and add the public key to your git host.

If you need extra software - for example, Node/yarn for JS apps, Python, Ruby, or anything else - then install these via apt. Such changes should persist properly between container restarts, meaning that you can build out your development environment in the way that suits you best.

If you want to be able to run your apps - for example a Python Flask app on port 5000 - and access via a browser, then just add a mapping for the port to your docker-compose.yml file.

Optional next steps

Although Tailscale is encrypting your traffic, you may want to provision a TLS certficate and use a reverse-proxy, like Traefik, to serve Code Server.

To do so, I recommend following the instructions on the Tailscale website.

Conclusion

Whilst the approach in this post will provide a working development environment for writing, building, and deploying many types of software, it isn’t suitable for all apps.

For example, mobile development (in which a connected device or simulator might be required) will be tricky. Similarly, coding for Apple devices (macOS or iOS) requires XCode, which is currently only available on MacOS.

Even web development can be a little fiddly without direct access to developer tools in a browser. However, for light usage, I really enjoy this approach and I am excited about the possibilities as the ecosystem continues to develop.

Parcel to the rescue

Wed, 25 May 2022 17:59:00 +0000

Earlier this week I needed to make some changes and re-deploy an old Vue app. I hadn’t touched the codebase in over a year, and my experience with the rate of change in the front-end web space made me dread what would happen if I tried to re-awaken this thing.

Sure enough, after running a yarn install and launching the app using the scripts in package.json, a number of errors were displayed about Node/Webpack/Vue incompatibilities, and I didn’t really know where to start. I don’t use Vue on a daily basis these days, and so I don’t usually need to make an effort to keep fully up-to-date on its developments, but I knew I was several versions behind on vue, vue-loader, as well as all the sass and babel toolings. This wasn’t going to be a quick fix.

From the errors displayed on the console, I could see that the problem was largely related to the relationships between the various build tools. When I worked on this project more full-time (or even still today?) the vue-cli would manage the build process for you, and would create a number of files and directories containing various loader configuration setups and Webpack build files.

The problem was probably to do with something in those files no longer working - which is frustrating given the version-locked dependencies that are supposed to stop this kind of thing from happening. I didn’t have the time, expertise, or energy to try and debug this complex setup, and I couldn’t find many resources that helped me in migrating the build tools.

Could Parcel help?

In another project, I had recently used Parcel to manage webapp build processes. The tool promises to be “zero config” out of the box and - in my experience - it had been.

I saw that Parcel supports Vue and thought I’d see if it could help me solve my problems. I made a few tweaks in the codebase structure and small changes to make the app Vue 3 compatible (this was relatively straight forward) and I bumped a couple of other packages (such as the router).

I added the Parcel dependency and also removed a number of other dependencies, such as Webpack and babel-related ones, along with the various -loaders needed by Webpack. I launched the app using parcel and… it built! I got a hot-reloading development environment without any fuss. Awesome.

I was able to make the changes I needed to the app, build, and deploy it as a static webapp - all using Parcel and now with a cleaner and leaner project structure and dependency list.

Alternatives

I understand that, these days, Webpack 5 is supposed to also be “config-less” for most setups, but their docs explain that this is only the case for js and json files. If you want anything else, such as images or stylesheets, you’ll need extra loaders in your dependencies and extra configuration files to write and maintain.

More recently, I’ve heard great things about Vite too, but I’ve not yet had a chance to test this out.

Either way, if you want to build and manage your web projects without a fuss, then I can certainly recommend taking Parcel for a spin.

Simple Ledger visualisations using Python

Sun, 24 Apr 2022 14:39:00 +0000

If you’re a current follower of this blog then you may already know that I’m a bit of a fan of using plain text accounting for managing finances.

I mainly use the Ledger text file format and CLI tool for bookkeeping and reporting on finances. This works great, and I can quickly and easily generate different kinds of reports using a range of simple Ledger commands.

For example, to generate a quick income/expenses balance sheet for a particular date range I can run ledger balance income expense -b 2022/03/01 -e 2022/03/31, which produces something along the lines of the following:

             £381.50  Expenses
             £129.87    Hosting
             £185.00    Services:Accountancy
              £66.63    Software
          £-1,000.00  Income:Sales:Contracting
--------------------
            £-618.50

Whilst this approach allows for easy numeric reporting for accounting purposes, by default there are no built-in ways for visualising the flow of money.

In this post I’ll talk a little about how to use Python to generate a simple visualisation of financial flow when provided with a Ledger export. This consists of three key steps:

Outputting data from Ledger in a readable format;
Processing the data;
Outputting a diagram that visually represents the data.

Step 1: Outputting Ledger data

Ledger provides a number of commands that allow for exporting and reporting on a ledger file. I recommend reading the documentation for a full idea. However, in this case, we can simlpy use the csv command to generate an output that can be easily piped into a Python program.

For example, ledger csv income expense -b 2022/03/01 -e 2022/03/31 produces an output with each transaction on its own row in CSV format, along the lines of below:

"2022/03/01","","Linode","Expenses:Hosting","£","40.58","",""
"2022/03/01","","Company1","Income:Consultancy","£","-120.5","",""
"2022/03/01","","Accountancy Company","Expenses:Services","£","200.28","",""
"2022/03/03","","Company2","Income:Sales","£","-900.0","",""
...

We can write some simple Python now to receive this data into our program. Create a new Python source file (e.g. viz.py) and add these contents:

import sys

# Define a Transaction class
class Transaction:
  def __init__(self, account, value):
    self.account = account # The account name for this transaction
    self.value = 0 - float(value) # Income is negative

# Prepare a list of transactions by reading from stdin
transactions = []
for line in sys.stdin:
  line = line.replace('"', '').split(',')
  transactions.append(
    Transaction(line[3], line[6])
  )

The code above reads in lines from stdin and instantiates Transaction objects for each line. We can invoke this code by piping the output from Ledger directly into the program:

ledger csv income expense -b 2022/03/01 -e 2022/03/31 | python viz.py

In this post, I’m only really interested in the financial flow (i.e. the values of transactions), and so we don’t need to capture payees or other information from the transactions.

Step 2: Data processing

The next step is to process the transaction data. My aim is to achieve a Sankey diagram that shows the flow from income to expense accounts, and to illustrate any profit.

As such, we need to build some account data for the various accounts in our Ledger file.

To do so, add the following code to your viz.py file:

# Instantiate an Account class
class Account:
  def __init__(self, name, value):
    print(name, value)
    self.name = name
    self.value = value

  # In the Sankey diagram, show income coming in from above and expense going downwards:
  @property
  def orientation(self):
    if self.value < 0: return -1
    if self.value > 0: return 1

accounts = []
for transaction in transactions:
  t_accounts = transaction.account.split(':')
  top_level = t_accounts[0]
  account = next(filter(lambda a: a.name == top_level, accounts), None)
  if not account:
    account = Account(top_level, 0)
    accounts.append(account)
  account.value += transaction.value

# Calculate profit
profit = sum(map(lambda t: t.value, transactions))
if profit > 0:
  accounts.append(Account('Profit', 0 - profit))

In the above code, we declare an Account class and then iterate through our transactions to build up a list of Account objects, with each having a total value representing the sum of all transactions involving that account. In this code, for now, we only consider the top level accounts (i.e. Income, Expense) rather than deeper levels (like Income:Sales).

In the last few lines above, we work out if the reporting period is profitable. If so, we manually create a Profit account with the left over value so that this can be visualised in the diagram.

Step 3: Producing the diagram

We’ll make use of the matplotlib package for creating its own version of a Sankey diagram. Install this package in your environment (pip install matplotlib) and then import the required modules at the top of viz.py (below the sys import):

from matplotlib.sankey import Sankey
import matplotlib.pyplot as plt

Then, at the bottom of your viz.py file, add the following code:

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, xticks=[], yticks=[], title="Ledger Financial Data")
sankey = Sankey(ax=ax, unit='£')
sankey.add(
  flows=list(map(lambda a: a.value, accounts)),
  labels=list(map(lambda a: a.name, accounts)),
  orientations=list(map(lambda a: a.orientation, accounts)),
  patchlabel="Income and expense"
)
sankey.finish()
plt.show()

In the above code we instantiate a new figure, axis, and Sankey diagram. We then add a new section to the diagram (using add()) and to this pass a bunch of parameters:

flows: a list of total values in each account
labels: a list of account names for each account
orientation: a list of orientations for the Sankey arrows
patchlabel: a title for the Sankey section

We then finalise the diagram and ask for the figure to be shown.

If you now run your code, as described in “Step 1”, you should see something like the following being produced.

The diagram shows us some basic information about income, expense, and profit, but it would be nice to get a bit more of a breakdown.

Step 4: Add sub-accounts to the visualisation

To display more of a granular breakdown in the diagram, we need to make changes to the processing step (Step 2) and the display step (Step 3) above.

To begin with, modify the Account class so we can provide it with a reference to the account’s parent (if it exists):

class Account:
  def __init__(self, index, name, value, parent=None):
    self.name = name
    self.value = value
    self.parent = parent # A reference to another Account object
    ...

Next, modify the loop in which we create the accounts to detect parent accounts:

accounts = []
for transaction in transactions:
  t_accounts = transaction.account.split(':')
  for level, a in enumerate(t_accounts):
    account_name = ':'.join(t_accounts[:level + 1])
    parent_account_name = ':'.join(t_accounts[:level])
    account = next(filter(lambda a: a.name == account_name, accounts), None)
    if not account:
      parent_account = None
      if parent_account_name:
        parent_account = next(filter(lambda a: a.name == parent_account_name, accounts), None)
      account = Account(len(accounts), account_name, 0, parent_account)
      accounts.append(account)
    account.value += transaction.value

The code above performs various additional lookups to try and find parent accounts, and instantiates all accounts at each level in the Ledger file.

Finally, we can adjust the diagram-producing code to make use of the various accounts:

sankey = Sankey(ax=ax, unit='£')
for parent_account in filter(lambda a: not a.parent, accounts):
  children = list(filter(lambda a: (a.parent and a.parent.name) == parent_account.name, accounts))
  if len(children):
    sankey.add(
      flows=list(map(lambda a: a.value, children)),
      labels=list(map(lambda a: a.name, children)),
      orientations=list(map(lambda a: a.orientation, children)),
    )
  else:
    sankey.add(
      flows=[parent_account.value],
      labels=[parent_account.name],
      orientations=[parent_account.orientation],
    )
sankey.finish()
plt.show()

This code now iterates through each parent account (i.e. each account without a parent of its own). It then looks up the children of these accounts and creates a new Sankey segment containing the child accounts (or a single account for the parent if there are no child accounts).

Running the script again with the Ledger data should yield something like the following diagram.

We now get more useful information being displayed, and we can see - at a glance - where the money is going.

Conclusion

Although we have made improvements to the diagram, there are still possible further enhancements that you may wish to explore, depending on your needs. For example, the current approach only goes two levels deep; the display stage would need to be amended to recursively navigate account levels to get an even finer-grained view.

I can recommend taking a look through the matplotlib Sankey documentation for more information on how you can tweak the diagram, or the documentation in general should you wish to explore other types of visualisations that you can pass the processed transactions in to.

On alcohol

Tue, 12 Apr 2022 17:55:00 +0000

Whilst my days of binge drinking as a student are thankfully far in my past, alcohol is still an ongoing, yet much more minor, part of my life.

Like many millennials (and I’m sure it must be the same for other generations too), we got used to it as a mechanism for socialising. Whether this is meeting friends after work, going out for dinner with family, or spending time in the pub or at home with a significant other.

Throughout my life I’ve been lucky enough to meet or work with people who just don’t drink. And although it’s taken me a while, I’m certainly starting to see why. The thought of feeling that you have to drink in order to enjoy one’s self is clearly ridiculous, and if someone does find this to be the case then it may be a sign of more fundamental issues.

Whilst I understand and appreciate the relaxing effect of alcohol, I’ve recently begun to wonder about whether this outweighs the down-sides. Some of the negative effects I’ve begun to personally notice after consuming alcohol, or related in general, are;

Sleep: my sleep quality (and subsequent potential the following day) is significantly reduced. It’s known that alcohol can inhibit the brain’s ability to “deep sleep” even if you appear to be asleep.
Fitness: my workouts are less impactful, and things become harder than they should. This inhibits progression.
Weight: generally, alcohol is calorific and does not fill you up like food does.
Hangovers: whilst only after a (thankfully) less-frequent particularly heavy night, these just cancel out the next day. Even “light” hangovers can severely impact morning productivity. These are also accompanied by depression, shame, and other negative emotions I don’t need or want.
Mindfulness: my mood, ability to deep think, and motivation are impacted.

There are also things I can’t observe directly myself that I am sure must be affected, such as the effect on my health (both physical and mental).

I don’t intend to give it up entirely (it’s always nice for celebrating with others or for special occasions), but I’m going to spend at least a few weeks alcohol-free and see how things go with social events and seeing family. It will alter my routine a little (e.g. instead of meeting friends in the pub I will suggest coffee instead), and perhaps it’ll change the way others in my peer and family group see me, but I’m excited to hopefully see and feel some day-to-day benefits that will enable me to be more informed in weighing-up some of my decisions going forward.

Self-hosting apps and services using Traefik reverse proxy

Sat, 09 Apr 2022 13:22:00 +0000

I often talk about self-hosting on this blog, and I’m certainly a big fan of being able to control my own data and systems wherever possible (and feasible). I’ve recently switched from using Nginx to Traefik as a reverse proxy for my server and for terminating TLS connections.

In this post I’ll talk a little about why and how I made this change.

My (until recent) setup

I self-host a number of services; including Nextcloud for file storage and sync, Gitea for git syncing, FreshRSS for RSS feed aggregation and management, Monica for relationship organisation, and a few other things too.

To date, I’ve mostly relied on Nginx for providing a reverse proxy, enabling me to run many of these services on a single VPS. I run everything, including Nginx, using Docker containers, which I find much more convenient and easy to manage.

Individual services are managed using Docker Compose, which allows me to keep services that rely on multiple containers (such as those that depend on both a web server and a backend database server) logically separated and organised. I use a single nginx.conf file, used by my Nginx container, to manage these services as virtual hosts and for configuring TLS certificates.

Why I wanted a change

This setup works well and has served me well for several years. However, it isn’t without its drawbacks;

The basic system relies on manual and semi-tedious config entries to my nginx.conf, including HTTP -> HTTPS redirects, enumerating virtual hosts, and other deeper Nginx stuff I don’t always fully understand.
TLS certificate provisioning and renewal: I either need to do this manually or remember to setup (and then trust) a cron job to renew things properly.
Domain verification for Let’s Encrypt: configuring webroots and interception of ACME requests for each virtual host.
Needing to restart (or automate the restart) of the webserver after receiving new certificates.

Most of these pain points are related to the reverse proxy setup itself, rather than the individual services (which can happily run indefinitely). To me, it felt like the webserver side of things should be the most “boring” and least time-consuming, when in reality it is the opposite.

I wanted to find a new solution that would help me solve these issues and allow me to focus more on maintaining and working on the services themselves.

## Traefik as a reverse proxy

Traefik provides a proxy system that I’ve used before in Kubernetes setups. It acts as a reverse proxy, TLS terminator, load balancer, and much more. It’s designed to work well with microservices running in containers, and whilst Nginx can certainly achieve all of this too, the nice thing about Traefik is that it does this for you dynamically and automatically.

It can easily be configured using Docker Compose, and all the TLS provisioning is handled automatically via Docker labels. The documentation covers the setup well and in detail, but below I will run through (roughly) how I made the switch from Nginx to Traefik.

### Step 1: Firstly, stop all of your running services

If you use Compose, you can simply run docker-compose down to take your services down. At the very least, make sure you are shutting down your Nginx container.

Note: If your user is not in the docker group you may need to use sudo to run docker-compose commands.

Step 2: Next, write a new Docker Compose file

Create a new directory to house your Traefik setup, and in here write a new docker-compose.yml file, as shown below. In this example I am using Traefik to route requests through to Nextcloud and FreshRSS instances:

version: '3'

services:

  # The Traefik service
  reverse-proxy:
    image: traefik:v2.6
    command:
      # Tell Traefik that we're working in a Docker environment
      # This allows it to dynamically pull out the configuration
      - "--providers.docker"
      # Standard ports for HTTP/S
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      # ACME TLS certificate resolver setup. Remember to change your email address here
      - "--certificatesresolvers.myresolver.acme.email=me@example.com"
      - "--certificatesresolvers.myresolver.acme.storage=/etc/traefik/acme/acme.json"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
    ports:
      # Map our web traffic ports to the host network
      - "80:80"
      - "443:443"
    volumes:
      # Mounting this volume is important to ensure certificates can be persisted
      - ./acme:/etc/traefik/acme
      # This volume is to enable Traefik to communicate with Docker
      - /var/run/docker.sock:/var/run/docker.sock

  # The FreshRSS service
  # (see https://github.com/FreshRSS/FreshRSS/tree/edge/Docker for more info)
  freshrss:
    image: freshrss/freshrss
    environment:
      - "CRON_MIN=3,33"
      - "TZ=Europe/London"
    volumes:
      - ./freshrss_data:/var/www/FreshRSS/data
      - ./freshrss_extensions:/var/www/FreshRSS/extensions
    labels:
      # Here we add a label to tell Traefik how to route to this service by domain
      # Remember to change this host value.
      - traefik.http.routers.freshrss.rule=Host(`rss.example.com`)
      # These two labels tell Traefik to setup TLS for this service
      - traefik.http.routers.freshrss.tls=true
      - traefik.http.routers.freshrss.tls.certresolver=myresolver
    restart: always

  # The Nextcloud service
  # (see https://hub.docker.com/_/nextcloud for more info)
  nextclouddb:
    image: mariadb:10.5
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - ./nextcloud-data/db:/var/lib/mysql
    environment:
      # Remember to change these values:
      - MYSQL_ROOT_PASSWORD=
      - MYSQL_PASSWORD=
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
  nextcloud:
    image: nextcloud:22
    volumes:
      - ./nextcloud-data/storage:/var/www/html
    restart: always
    labels:
      # Here we add a label to tell Traefik how to route to this service by domain
      # Remember to change this host value.
      - traefik.http.routers.nextcloud.rule=Host(`nextcloud.example.com`)
      # These two labels tell Traefik to setup TLS for this service
      - traefik.http.routers.nextcloud.tls=true
      - traefik.http.routers.nextcloud.tls.certresolver=myresolver

The above may seem quite verbose, but this is literally all you need. After running docker-compose up -d (and you’ve changed your domains and passwords, etc.) your services will be up and running with TLS certificates automatically provisioned.

I’ve added comments to the file to explain some of the concepts further, but below I’ll add a few extra notes:

I’ve setup four services: the Traefik reverse proxy itself, a freshrss container, and two containers for Nextcloud: nextcloud (the software itself), and nextclouddb (the MariaDB database for Nextcloud).
Since nextclouddb doesn’t need to (and shouldn’t, for security) be exposed to the internet, we do not add any Traefik routing labels to this service.
Adding a mount for the acme.json storage is important in your Traefik setup to avoid hitting Let’s Encrypt rate limits.
I added volumes for FreshRSS and Nextcloud so that data can be persisted.
Notice how all Traefik routing and TLS configuration on services is simply handled with labels.

Whilst the above setup achieves what we need, it is simple. There is much more you can do, such as route using specific paths and manage load balancing. For more information, please see the documentation.

More complex services

As mentioned earlier, I also run Gitea as a service on my VPS. Gitea serves a GitHub-style web UI in addition to a git endpoint. Since I interact with the git endpoints over SSH, this service ends up relying on two ingress points: one for HTTP web traffic and another for SSH git traffic.

When I first set this up, Traefik was routing web traffic through to Gitea’s SSH port, which obviously caused problems. As such, I needed to add extra configuration to tell Traefik which port to use, as described below:

...

  gitea:
    image: gitea/gitea:latest
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
    volumes:
      - ./gitea_data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    expose:
      - "3000"
    ports:
      # Ensure we map port 22 to the host for incoming SSH git traffic
      - "22:22"
    labels:
      # The below three labels are as described earlier
      - traefik.http.routers.gitea.rule=Host(`git.example.com`)
      - traefik.http.routers.gitea.tls=true
      - traefik.http.routers.gitea.tls.certresolver=myresolver
      # Add a fourth label to tell Traefik where to route web traffic to
      - traefik.http.services.gitea.loadbalancer.server.port=3000

After running docker-compose up -d again, your Gitea instance will be running, along with its web UI and git endpoint.

Note: if you use port 22 for standard SSH to your host server, you can map a different port for your SSH git traffic. For example, 2200:22.

Conclusion

Having made this switch, I find things much more logically organised and robust. The reverse proxy is just as performant (at least, for my use) as Nginx and I get extra peace of mind in that I can trust Traefik to handle web traffic, routing, and certificate renewals without any manual intervention.

Interacting with a Nextcloud instance deployed with Docker

Fri, 28 Jan 2022 19:18:00 +0000

If you’ve ever run your own Nextcloud before, you may have noticed screens like the following in your instance’s settings pages.

The messages advise a number of maintenance procedures to help ensure the smooth running of your instance. These could be to run database migrations or to update schemas in response to installing new apps.

Often these steps might involve running occ commands. occ is Nextcloud’s command-line interface, and is so-called because of its origins in ownCloud.

If you deploy your Nextcloud using Docker, then it isn’t immediately obvious how to begin invoking the occ command.

Invoking `occ` on a running Nextcloud Docker container

Luckily, Docker makes this straight forward if you run Nextcloud in a container.

Assuming you manage your Docker orchestration using Docker Compose, if you wanted to run the occ db:add-missing-indices command from the screenshot above on a Nextcloud Docker container, you could run the following from your project’s directory:

$ docker-compose exec --user www-data nextcloud php occ db:add-missing-indices

Where nextcloud is the service’s name in your docker-compose.yml file

If you don’t use Docker Compose, the same can be achieved using docker exec directly:

$ docker exec --user www-data 0e4c3hd9s049 php occ db:add-missing-indices

Where 0e4c3hd9s049 is the ID for your Nextcloud container, which you can find by running docker ps first.

In both cases, the command is structured the same. You may wish to note the --user flag. In my case, I tell Docker to run the command as www-data as that is the user that owns the Nextcloud config/config.php file in my setup. If you use a different user, then update this value in your command.

You may also need to run the docker-compose and docker commands as a superuser, depending on whether your normal user is in the appropriate group.

The future of the decentralised web: thoughts on web0 and web3

Fri, 21 Jan 2022 08:27:00 +0000

I recently signed the web0 manifesto, which embodies many of the values I consider to be important when it comes to technology - and the web in particular.

web0 is the decentralised web… web0 is web3 without all the corporate right-libertarian Silicon Valley bullshit.

– The web0 manifesto website

Essentially web0 is around empowering a decentralised web that:

Is available and truly accessible to everyone;
Minimises the barriers to entry for all;
Enables people to own and control their own data (sovereignty);
Enables people to move, remove, or modify their own data;
Avoids the naturally-occurring centralisation when utilising big-tech services;
… And therefore puts the web back in the control of individuals rather than a small number of huge companies.

In practice this could mean owning your own domain name and taking part by hosting a website or through getting involved in other communities, such as in the tildeverse. The key thing is that participants own and can control their own data and that things are accomplished without needing to rely on big-tech.

Otherwise, there are no clearly defined objectives specified in the manifesto, which in itself demonstrates the freedom in the values it is trying to promote. Furthermore, web0 is not in itself aiming to become a trend or a recognised movement - it is intended more as a guiding set of principles that rejects the use of some technologies in order to achieve its goals.

Decentralisation and web3

The fundamental principles behind web3 share some similarities with those exhibited by the ideas of web0. Notions such as removing the reliance on “big tech”, making tools accessible to all, and helping to facilitate the return of data ownership and control back to the individuals involved are certainly shared amongst the two movements.

And in terms of the underlying technology itself relevant to web3 - i.e. permissionless distributed blockchain and smart contracts - this is essentially achieved, too. The technologies are already well-proven in a number of fields - from finance and insurance through to legal and supply-chain, but their implementation as a core web technology still faces a number of issues. This “web3” aims to replace traditional web components (servers, databases, accounts, etc.) with blockchain-native concepts

The issues - rather than the technologies themselves (though these can be problematic too) - are part of the motivations for publishing the web0 manifesto, and are largely related to the way these tools are being leveraged, and the creep of financially powerful individuals and organisations into the space.

With early adopters and VCs understanding that by growing enthusiasm and intrigue in the world of web3 and crypto they can hugely maximise their return on investments, self-worth, and perceived positions in the crypto echelons, they naturally want to drive additional growth in mid-to-late adopters. It is a consequence that those towards the end of the “hype train” lose out by engaging at a time when the decentralised digital assets (and the transactions that power their creation and movement) have gained massively inflated prices.

Some still choose to engage at this time as they are convinced by the excitement (marketing?) and assume they will also be able to join in on the riches and reward so publicly flaunted by their peers and celebrities alike.

Trends such as including ENS (.eth) names in Twitter profiles and the use of NFT artwork as avatars and imagery across social platforms only serve to develop the hype. Some of these communities (“crypto bros”) even use toxicity to further their cause and will aggressively rebuke any motions made against crypto ethics - no matter how well founded the argument.

The upshot is that now most people are completely priced out of getting involved. .eth domains are a nice way of providing a basic public identity to your web3 interactions; compared to traditional DNS names they’re a bargain at $5/year, but given the gas fees (at the time of writing) mean you need to fork out hundreds of dollars worth of ETH just to complete the transactions required in order to own one, this doesn’t scream openness and accessibility.

NFT marketplaces have been hailed as a way to revolutionise art and to put the power in the hands of the creators. If this is the case, then that’s fantastic, but I worry about up-and-coming artists losing out financially when they realise that the fees they need to pay in order to transfer the work after a purchase may far exceed the sale price itself.

Many of these issues are related to the limitations of the blockchain technology currently most relevant to web3 - Ethereum. Various technical factors result in a limited transaction throughput, and with increased hype in the space, the demand (and thus cost) for computation on the network also increases. Of course, other technologies exist that try to solve these problems; Solana has lots of promise but still has some way to go in competing with Ethereum in terms of popularity. There have also been a few questions raised about Solana’s stability.

Of course, things may change a little when Ethereum’s proof-of-stake merge completes later this year, anyway. People with less disposable funds may then be able to participate, but asset prices and NFTs are unlikely to decrease because of this. Either way, the technical know-how required to set-up wallets and trade crypto assets presents intrinsic friction which, when combined with the fact that there’s essentially zero protection if passphrases or private keys are lost, presents alien concepts to new users.

web3 and web0 might both purport to be focussed on decentralisation and accessibility, but it is clear that there is a difference in the type of people that have the resources and capability to actively get involved in each.

How decentralised is web3, really?

Moxie Marlinspike‘s recent and excellent post on web3 highlighted several issues that hint at a couple of growing trends that essentially centralise web3 principles.

The first is that nearly all crypto wallets and dApps (decentralised apps, in which the blockchain is the database) interface with the blockchain through a handful of well-known providers, such as Infura or Alchemy.

The second is that these wallets also use the Opensea API for retrieving and displaying NFTs. This means that if Opensea chooses to delist your NFTs for any reason (as they did in Moxie’s post), then suddenly they cease to exist in wallets too - even if they continue to live as tokens on the chain itself. Like any other company, Opensea can make its own decisions about how it acts, and is also subject to external pressures (government, police and financial included) to act in different ways, which could have dramatic effects on the NFT economy.

As before, this has a greater impact on the creative industry, where the visual aspect of the tokens (and thus where and the way they are rendered) is more important than the technical effects of having the token in one’s wallet.

Both of these factors are antithesis to what web3 is supposed to - and what web0 certainly does - stand for. Suddenly people are back in the position of losing their data sovereignty to a small number of tech companies that are gathering increasingly more control over the ecosystem. If things continue down this route, as it surely will with more and more large companies becoming involved in the space - then it won’t be long until the blockchain component is completely forgotten and everything boils down to centralise around a handful of platforms running the whole show. Sound familiar?

Unstoppable Domains is another example of a VC-invested company getting in on the web3 action. They let you buy domains that sit as NFTs on the Polygon protocol (an Ethereum layer 2 solution). Whilst I certainly agree that domain squatting is bad practice, preventing people from purchasing domains because they happen to have a name similar to a “notable individual” is also an example of corporate control over the web3 ecosystem - no matter what the intention is.

I totally get that traditional DNS names can also be seized and controlled, but web3 companies should be pushing past these restrictions.

Apparently their support team has also admitted that their goal is “adoption” (i.e. higher sales) rather than “permissionless” (i.e. fully decentralised).

Whilst, of course, we shouldn’t necessarily be against companies trying to advocate for decentralisation, I think they just need to be wary about undermining the very world they are trying to build. Their priority is always their shareholders; not the general good.

Some closing thoughts

Whilst I back decentralisation and the accessibility and availability-to-all ideologies of web0 wholeheartedly, I appreciate that there could eventually be avenues for collaboration between the underlying concepts from both movements.

Some people say that web3 is still in its infancy. It’s hard to say how true this is when the technology itself has been in mainstream use for over a decade, but if there is hope for a dramatic reduction in the barrier-to-entry for this suite of technologies, and if the ethical and human issues can be resolved, then I can see some potential for a decentralised transfer of tokens and uniqueness of assets being useful in a web context.

For example, if a truly decentralised and permissionless asset system to represent user accounts can involve tokens being transferred and used anonymously between people-oriented services, and which doesn’t in itself exclude people based on position or financial capability, then I think that would be a positive outcome.

Either way, blockchain tech, digital assets, and the entire ecosystem is certainly not going anywhere, and the uses for these tools are expanding every day. Though until the issues around adaptation to the web and mainstream accessibility can be resolved then they may not yet be ready to form the foundations for a web that is truly available to all.

web0, on the other hand, is perfectly positioned as a concept powering an ecosystem that anyone and everyone can participate in. We’ve already seen a significant recent growth in the people and organisations adopting small web principles and I am excited to see how this can help form a healthy web for the future.

Completing the #100DaysToOffload challenge

Thu, 13 Jan 2022 19:19:00 +0000

Twelve months ago - in January 2021 - I started my attempt at the #100DaysToOffload challenge. I had set myself a new year’s resolution to try to write more and, around the same time, I noticed the hashtag for the challenge circulating on Mastodon. It seemed like the perfect opportunity to fulfil my resolution. The challenge is to post 100 times on a personal blog during the space of one year.

Whilst I had a blog (on this website) before starting the challenge, I only really wrote on it sporadically. In this post I’ll talk a little about how I approached the challenge, and about how I came to be able to post more frequently during the year (and hopefully beyond!).

Interacting with other challengers

I began by visiting the challenge website to follow the feeds of other bloggers that had previously completed the challenge. I also searched the hashtag in the fediverse to discover people currently attempting it.

My aim was to follow and interact with others in order to gain inspiration and ideas for posts. However, I quickly realised that people wrote about things that meant something to them - a personal blog is exactly that: a series of posts about things the author is interested in or wants to share. This is the entire point of the challenge in the first place; it has to be completed on a personal blog.

Despite this, following other frequent bloggers certainly provided encouragement and motivation to get started and to keep going, which was in itself invaluable.

Planning “when” to write

As someone who typically blogged only a handful of times a year previously, I needed to plan an approach that would enable me to write more frequently.

To get 100 posts published in 2021, I’d need to write about two per week. I set myself a “Write blog post!” Reminder and configured it to repeat twice a week: on Wednesdays and Saturdays. I use the Sorted app, which made this easy, but any software that supports repeated “reminders” would work.

This way, even if I was unable to write my post on a particular Wednesday in which I was particularly busy, the reminder would still show as “todo” on the Thursday. This helped me to stay on track.

Planning “what” to write

Planning when to write is the easy part. The problem is then working out what to write about.

I always keep a “scratchpad” note on my notes app in which I keep random links, bookmarks, and other things to eventually think about, visit, or to filter through to a better record-keeping system.

I made a section at the top of this note called “Blog post ideas” and began to populate it with a handful of short bullet points for potential topics. I added sub-points to each topic in order to add extra context or ideas related to that topic that I could also include.

I use the Bear app, which meant I could easily note down ideas as they came to me when out and about, but any app that syncs well between devices would do the job.

To start with, the list contained some software dependency packages I’d recently discovered and found interesting. My intention was to write short posts to explain how to use the packages, describe their purpose, and some thoughts about why I had decided to use them.

I then began to think of my own life, and what was potentially noteworthy from that angle. I began to bullet about some interesting books I’d recently read, some podcasts I subscribe to, and the fact I had recently got a dog. I had also recently started exercising more, working from home, and had ideas about achieving balance in those areas.

Before too long, I had a bank of around 20 post ideas in my scratchpad note. This made the task seem much less daunting!

The ideas themselves weren’t very exciting, but I had angles I could use to make them interesting or useful to others whilst keeping them personal.

Getting over the inertia of writing & some learnings

I’m not a natural writer. One of the reasons I blogged so infrequently before 2021 was a fear and apprehension about others reading - and judging - what I write. This made it hard to initially put “pen to paper” and get started.

This fear also made the process long. Short posts that should take 20 minutes to draft, check, and publish would often take me hours. I’d proof-read again and again and try to take on additional viewpoints. This made the act of then physically posting the article even more difficult, and even after posting it I’d re-read it and continue to make post-publish edits.

The volume and frequency of writing required of the challenge meant I didn’t have time for all of this anymore. I still wanted a life and the ability to work on other things in my spare time (where it exists!), and so spending several hours twice a week on blog-writing just wouldn’t fit.

As such, I quickly learned to just start writing. After I was a few posts into the challenge I began to treat each one as if it were a more casual email or as if I were simply drafting a toot (or a tweet).

It was a new mental model that allowed me to write without fear, and I quickly began to look forward to Wednesdays and Saturdays and to getting down to writing and publishing a new post.

Before I knew it I was 10 days, then 20, and then 30 days into the challenge. I’d continue to add new ideas to the scratchpad to keep up the backlog, and the whole process began to take on a rhythm of its own.

Takeaways

I now feel much more confident in my writing. I know that not everyone will find my posts useful or interesting (far from it), but I’ve begun to enjoy writing simply for the sake of it. I also get a few comments from others who read and subscribe to my blog, which is nice.

I still maintain my “Blog post ideas” list, and continue to add to it. I hope to carry on working through these in 2022. I found I’ve re-learned my ability to explore the web and all it has to offer. I take the time to read about articles, posts, and new developments, and to form opinions and thoughts that - in turn - spark the ideas for new thought pieces of my own.

I’ve also loved reading and engaging with the posts published by others taking part in the challenge. It’s a great mix of personal content, technical reviews and tutorials, and lots more. My RSS feed is always full of interesting posts (which reminds me that I need to create a blog roll for my website soon…), and I feel that I learn a huge amount from getting involved.

Writing my blog also encouraged me to spend more time on my personal website. I wanted to make it more accessible, readable and engaging, whilst keeping things as simple as possible. I added new pages (such as a “notes” page), and tried to link to other projects and inspirations where possible.

As a result of the blog and the changes to my site I noticed a 20-30x increase in web traffic. Some posts got popular enough for them to get shared and boosted more widely, or even cross-shared by others to other websites and I’d receive 250x my usual pre-challenge daily traffic.

Whilst I won’t immediately take part in the challenge again at the start of 2022, I hope to certainly do so again in the future. Either way, I will be writing more frequently and with improved confidence.

The #100DaysToOffload challenge is certainly something that I can recommend to anyone who has a little time to spare a couple of times a week. Thank you to Kev for setting it up and for being so encouraging throughout!

Browser push notifications in a Flutter web application

Sat, 18 Dec 2021 10:03:00 +0000

🎉 This is post 100 in my attempt at the #100DaysToOffload challenge!

For a couple of years I have been writing mobile apps using the Flutter framework, having previously been a React Native advocate. Flutter is a great tool for writing applications that target multiple platforms and architectures from one code base - and not needing to write any JavaScript is definitely a bonus!

I use and recommend Firebase Cloud Messaging to handle push notifications in these applications. There’s also a great library for Flutter - Flutterfire - to handle the setup and receipt of these messages, along with the requesting of push permissions on iOS. The set-up takes away the pain of managing cross-platform notifications in Android and iOS applications.

I recently needed to publish a front-end web version of one of these apps. The back-end system (web API) would remain the same, but I was initially daunted by the task of needing to write a whole new web application mirroring the mobile client functionality.

However, I then realised I could (pretty much) simply issue a flutter build web command to get a web-based version of my mobile app. This worked beautifully and it made me remember why I enjoy using this framework so much! As a result of the command, a fully-responsive web app is built and is instantly deployable.

The only piece that didn’t quite work so well out of the box in the web version was the Firebase push notification system - the webapp would run fine but would crash with errors when trying to request push permission or interact with the Firebase SDK in other ways.

It seemed that additional setup would be required to allow for web-based push notifications.

Initializing a Flutter app

Both the Flutterfire documentation and the Firebase console include guides and documentation on setting-up push notifications in applications, however a little deviation is required for this to work seamlessly for both web and mobile applications from a single Flutter codebase.

In this post I’ll cover my approach to the additional setup required to getting this to work.

I’ll assume that you have an existing Flutter application to which you want to add web push functionality. I’ll also assume that you’ve already gone through the initial Firebase configuration and installation (i.e. that you have push working on mobile devices). If you don’t yet, the documentation I linked to above covers this better than I can and so I recommend going through that first!

Setting-up push for web

First off, we need to enable web-based builds for the existing Flutter app. If you haven’t already got a web/ directory in your project, run the following command in the root of your project:

flutter create .

You should now have a web/ directory with files such as index.html and manifest.json inside.

Next, in the Firebase console, go through the process of creating a new “web project” for your application. Once setup, navigate to the “General” tab of your “Project Settings” and select the newly-created Web App. In the “SDK setup and configuration” section, select “Config” and copy the displayed firebaseConfig object as shown in the image below.

Create a new empty service worker file named firebase-messaging-sw.js in the web/ directory of your Flutter project and paste the firebaseConfig object into this new file:

importScripts('https://www.gstatic.com/firebasejs/8.6.1/firebase-app.js');
importScripts('https://www.gstatic.com/firebasejs/8.6.1/firebase-messaging.js');

const firebaseConfig = {
  // Copy from Firebase as described above
};

firebase.initializeApp(firebaseConfig);

We now need to reference and register this service worker. To do so, open up the automatically-generated web/index.html file in your Flutter project and add this script block:

<script>
  if ('serviceWorker' in navigator) {
    window.addEventListener('load', function () {
      navigator.serviceWorker.register('/firebase-messaging-sw.js');
    });
  }
script>

Finally, we need to slightly alter the way we request push notification permission from the client to allow this to work in a browser-based context. To do so, we need to provide the browser with a web certificate. Luckily, this is easy to do in Firebase.

Back in the Firebase console navigate to the “Cloud Messaging” tab of your project’s settings. In the “Web configuration” section, create a new Web Push certificate and, once done, make a note of the key pair displayed:

Somewhere in your Flutter app’s Dart code, you probably have something along these lines when requesting push notification permission for your iOS apps:

FirebaseMessaging _firebaseMessaging = FirebaseMessaging.instance;
await _firebaseMessaging.requestPermission(
  alert: true,
  badge: true,
  sound: true,
);
String _pushToken = await _firebaseMessaging.getToken();

Make a change to that final line such that we now pass in the keypair you just created in Firebase:

// ... Previous code
String _pushToken = await _firebaseMessaging.getToken(
  vapidKey: ''
);

And that’s it! Your application will now request permission for push notifications when it is delivered over a secure connection. Build the web version of your Flutter app (flutter build web) and then deploy the web/ directory to your host of choice.

If permission is granted then the browser will receive and display notifications when they are sent - even if the browser tab for your app is currently closed! The system should also function on supported mobile browsers too, though please take a look through Caveats below.

Caveats

Although browser web push is gaining more and more support (and is already available in nearly all modern desktop browsers), they aren’t yet ubiquitously available. Lookin’ at you, Safari.

On Safari (either on my Mac or iPhone) the request permission flow would cause the app to throw an error and hang. To alleviate this I wrap the Firebase block in a try/catch and show a dialog to warn the user when push is unavailable.

try {
  // Firebase init and getToken code
}
catch (error) {
  showCupertinoDialog(
    context: context,
    builder: (BuildContext modalContext) => CupertinoAlertDialog(
      title: new Text('Unable to request push permissions'),
      content: new Text('Push may not be supported on your device.'),
      actions: <Widget>[
        CupertinoDialogAction(
          isDefaultAction: true,
          child: Text('OK'),
          onPressed: () => Navigator.of(modalContext).pop(),
        ),
      ],
    ),
  );
}

There may be a more elegant solution to detect this up-front, but I was unable to find one that worked nicely across web and mobile. Please let me know if there is a better approach to handling Safari and other unsupported browsers!

Conclusion

In this post I’ve briefly walked through the set-up of Firebase push notifications in a Flutter web app, with an approach that should allow you to maintain a single codebase and maintain functionality across web and native mobile app builds.

Although I enjoy using it, I feel that I’m still relatively inexperienced with Flutter. I am continually learning about new and interesting concepts and I’ll try and cover more about these as I discover them!

Moving my Matrix identity to Element One

Wed, 15 Dec 2021 21:51:00 +0000

For as long as I’ve been using Matrix I’ve hosted my own homeserver on my own VPS and at my own domain.

Why I chose to move

I previously wrote about how I self-host my homeserver with the help of the Synapse project. Although this set-up is quite straight forward, it’s an extra system to maintain with all of the associated overheads.

One of the reasons I don’t host my own mail server is that I fear missed messages and silent bounces. I trust dedicated mail providers (particularly Fastmail) more than myself in providing a robust enough service to ensure things get through. Equally, if I am telling other people my Matrix handle, then I want to make sure that messages they send (and those that I send) actually get delivered without any problems.

The Matrix project, and Element too, are moving at such a pace that my own setup quickly feels old-fashioned. They also introduce new features for which I would need to take time learning about and setting-up in a robust way.

So, last week, I made the decision to move my primary Matrix identity over to the Element One service. I pay $5 a month for their basic plan.

The migration process

Moving was relatively simple. I just created a new account on the Element One website and, once in, I re-joined the rooms I had previously been in on my old server. I also logged-into my new account on the Element One homeserver in my Element apps.

I then messaged those people that I talk to frequently to let them know my new username. Unlike Mastodon, there is no way to seamlessly transfer and redirect conversations between homeservers, so this was a bit of a manual job (though not too time-consuming).

I am keeping my old homeserver up for a few weeks in case any new messages come through, but will eventually archive the messages and shut this down.

My thoughts so far

I’m really enjoying the fact that I don’t need to worry about the ongoing maintenance of my personal homeserver, and the knowledge that I can trust Element in that they know what they’re doing.

The bridges that come with Element One - for Telegram, Whatsapp, and Signal - are fantastic. They work seamlessly; I get read receipts, can reply to messages, and see when people are typing. Contact profile information (such as avatar images) get pulled through too with no problems.

This means I can chat to people and groups on Matrix, Telegram, and Whatsapp all from one place, which is super convenient! Sure - the chats are no longer end-to-end encrypted (yet), but I rarely use secure chats on Telegram anyway and Whatsapp’s claims have always been spurious.

On the flip-side, Element One restricts you to their own domain name for their homeserver. As such, my new handle is @wilw:one.ems.host, and I’ve lost the “vanity” associated with using my own domain. This also means that I need to go through the migration process again if I ever move away from Element.

EMS (Element Matrix Services) does offer the ability to bring your own domain, but this service is aimed more at groups or businesses, since the smallest plan comes with a minimum of 5 users at $3 each. For now, this is overkill for just me.

This is the only down-side I notice at the moment, and it isn’t a big killer for me since most of my activity is room-based anyway.

Conclusion

In general, I’m glad to have made the move. Things work smoothly, I get to use modern Matrix features, and I trust the robustness and security of the professionally-hosted platform.

I’m not emotionally tied to Element One, and am always interested to hear about other Matrix homeservers and communities that may be more of a fit for me. However, for now, it’s a great alternative to self-hosting.

Nextcloud for photos: using cheaper object storage

Sat, 11 Dec 2021 10:54:00 +0000

Some people may remember my quest a few months back to find a good alternative to Google Photos for image storage and backup.

At the time, I talked about Piwigo, Mega and pCloud as potential candidates. I also briefly touched upon Nextcloud in that post - a service I use (and self-host) anyway for all of my other storage needs, but I did not consider it further due to the high cost of the associated block storage required to house a large number of images.

In this post I want to explore a different approach to storing files using Nextcloud: one that uses object storage (instead of block storage) as a backend.

Briefly: block storage vs object storage

This is a fairly obvious point these days in the cloud world, but I thought I’d include it in case the reader isn’t familiar with these terms.

Block storage is a facility offered by cloud and infrastructure-as-a-service (IaaS) providers. It is a system that enables you to create volumes that can then be attached and mounted by your cloud virtual machines. Once attached, they can be formatted with a filesystem of your choice and used just like any other filesystem mounted by your machine.

Block storage volumes can often be detached and re-attached to other instances, or even persist without being attached to any instance. Most modern providers use high-performance SSDs to power this facility, making it a great option for data-processing on cloud servers.

Object storage, on the other hand, is more of a service offered by the cloud provider. Users can create containers known as buckets, and in these store any number of arbitrary files (“objects”). Buckets don’t get attached to instances, and the objects they contain can be accessed completely independently over the web or via SDKs.

As such, object storage can be super useful for storing and retrieving user-uploaded content in a service, or for storing large collections of arbitrary data.

From the viewpoint of this post, the key difference between the two is the pricing. I use Linode as my cloud provider of choice, and - to give you an idea of the scale - 1TB of block storage would cost you over $100 a month. The same amount on object storage would be $20 a month.

Another factor to consider is that block storage must be pre-provisioned. I.e., once you create the volume you begin to be charged even if the volume is unformatted and empty. In object storage the cost scales depending on the amount of data actually contained by your buckets.

Object storage: competing with the photos alternatives

Based on the above, object storage is naturally a sensible choice if one were to choose out of the two for storing large volumes of images, since it’s the one that competes the most with other commercial offerings in terms of pricing.

Neither Apple iCloud or Google Photos offer a 1TB option, but their 2TB storage options are priced at around $10 a month.

Whilst this is essentially quarter the price of the Linode object storage pricing (which, in itself is great), the commercial offerings must also be pre-provisioned (i.e. you pay that amount even if you only upload one photo). You also become more locked into their relevant walled gardens, with all of the limitations I spoke about previously.

As such, I believe that a system based on your own managed block storage solution is a viable alternative.

Nextcloud for image storage

Nextcloud offers a great web interface and mobile app for accessing your files and viewing stored images. The mobile app also has an “auto-upload” facility for photos, which is super useful.

This, combined with the object storage, provides a more cost-effective way of managing files and photos stored in your Nextcloud.

Some people have reported issues with Nextcloud when viewing large folders of images (since the thumbnail-generating task is quite CPU-intensive). However, I am keen to give this a go to see if it could be a viable option.

Configuring Nextcloud to use object storage

To set-up a Nextcloud to use object storage I recommend creating a fresh new instance, since changing your existing one will make your previously-stored files unavailable.

Create a bucket and access credentials

In this post I will use Linode’s object storage as an example, but the process should be similar for any S3-compatible backend (e.g. Amazon S3, DigitalOcean Spaces, Backblaze B2, etc.).

To begin, login to the Linode dashboard and navigate to the “Object Storage” page. Then click “Create Bucket”, give your new bucket a name and choose a geographic region for its storage.

Next, navigate to the “Access Keys” tab on the “Object Storage” page and click “Create Access Key”. Give your key a label to remember it by, and select to give it limited access such that it can only read/write to the bucket you just created.

When created, a one-time dialog will popup to display the details of the new key. Make a note of both the “Access Key” and the “Secret Key”, and then close the dialog.

At this stage, you should now have:

A bucket name
An access key
An access key secret

You can now close the Linode dashboard.

Creating a new Nextcloud instance

I use Docker to run my services, so I recommend creating a new docker-compose.yml file with these contents:

version: '2'

services:
  nextcloud:
    image: nextcloud:23
    volumes:
      - ./nextcloud-data/storage:/var/www/html
    restart: always

Feel free to add any network configurations as required (e.g. to an Nginx host for TLS termination) and then bring up your instance: docker-compose up -d. You can also specify a database container in your compose file if the default SQLite option isn’t appropriate for your needs.

Configuring Nextcloud to use your new bucket

Once up and running, Nextcloud should create its files and directories in the nextcloud-data directory. To configure your instance to use object storage for its primary storage, you’ll need to add some lines to nextcloud-data/storage/config/config.php:

php
$CONFIG = array (
  ... // Omitted for brevity
  'objectstore' => [
    'class' => '\\OC\\Files\\ObjectStore\\S3',
    'arguments' => [
      'bucket' => 'BUCKET_NAME',
      'key'    => 'ACCESS_KEY',
      'secret' => 'ACCESS_KEY_SECRET',
      'hostname' => 'eu-central-1.linodeobjects.com',
      'port' => 443,
      'use_ssl' => true,
    ],
  ],
  ...
);

You’ll need to change the BUCKET_NAME, ACCESS_KEY, and ACCESS_KEY_SECRET to the values you noted earlier. You may also need to change the hostname if you use a different storage region or a different provider.

You can now save and exit the file.

Testing the set-up

To test the set-up, navigate to the URL you’ve configured for your instance. If this is the first time you’ve visited it there will be a few bits of setting-up to do, which Nextcloud will guide you through.

Once you’re through and logged-in, you can try adding files using the web interface and then use the Linode dashboard to verify that Nextcloud has created new files in the bucket. Note that the file names and extensions will be changed, since Nextcloud manages metadata and naming in its own database.

If something goes wrong then just check the logs of your running Nextcloud container: docker-compose logs.

Enabling automatic photo upload

Finally, when you’re ready, you can try importing your photos or enabling the Nextcloud app auto-upload feature on your phone. This feature can be found under the app’s settings.

I recommend trying it for a while and seeing how you get on with regards to performance and stability.

Wrapping-up

In this post I’ve covered the advantages of object storage, and how this mechanism can be applied to Nextcloud to provide a self-hosted alternative to Google and Apple photos.

Personally, I am still not 100% on the performance of Nextcloud in terms of photo-viewing. However, I would like to give it a chance and it could be viable anyway as a secondary backup to an existing primary system.

Google’s own image search and navigation is so convenient that creating a real competitor is difficult. However, I am keen to hear how others get on in this space!

The Idiot Brain by Dean Burnett

Wed, 08 Dec 2021 18:33:00 +0000

Dean Burnett’s The Idiot Brain is an interesting insight into why people think the way they do, personality, emotion, and the biology of the brain.

The author (who happens to live in the same city as me: Cardiff), covers a wide range of examples of human behaviour and relates them to brain function. Often these are based on defence mechanisms developed over the vast time of human evolution, and it’s amazing how our perception of fear and “uncertainty” can have an impact on other feelings and emotions too - such as embarrassment.

I really enjoyed learning about how seemingly-complex traits and personalities in other people can be driven by relatively simple innate characteristics and biology, and how relationships between traits can form - such as how more intelligent people are typically less confident.

I was also surprised by how much we - as humans - have come to understand about the brain, although of course there is still much to be discovered.

The book is well-written and uses very witty examples throughout. It’s a great read and I can certainly recommend it.

Parse and process incoming emails in a web application

Sun, 05 Dec 2021 13:15:00 +0000

Most applications include some sort of outbound transactional email as part of their normal function. These email messages could be to support account-level features (such as password-resets) or to notify the user about activity relevant to them.

In the latter case, such emails might be read, and then archived or deleted by the user, without further direct action. They aren’t typically designed to be something one actions or replies to - they’re mostly there to bring you back into engaging with the platform.

However, what if your users could interact with your platform via email to some extent? For example, if someone were to send you a message on the platform - and you receive an email notifying you about the message, along with its contents - then the natural thing would be for you to be able to click “reply” to the email and write a message back.

It is probably kinder to the user to provide this option, rather than always sending from a noreply@ type address (since everyone has email clients wherever they go), and it also means that your service sees more interaction (even if indirectly).

In this post I will cover an approach to enabling users to be able to simply click “reply” to transactional emails, and to allow the replies to be processed by the platform.

Transmitting and receiving emails

Luckily, many current transactional email providers make it quite straight-forward to support this sort of functionality.

I will talk about Sendgrid as a transactional mail provider for the benefit of this post, and will use a Python Flask app demo for handling the inbound webhooks from Sendgrid.

Essentially, the flow will be the following:

User “A” sends user “B” a message in our platform
The platform sends an email to user “B” containing the message, with a special Reply-To header
User “B” writes a message back to user “A” by directly replying to the email
Sendgrid processes the incoming email and sends a POST request with the details back to our platform
Our platform receives the webhook from Sendgrid, reads the message and headers and takes action.

Configuring the transactional mail provider

The first thing to do is to set-up a mail provider we can use for outgoing and incoming email messages. As mentioned, in this post we’ll discuss Sendgrid, but others would also do the job (such as Mailgun).

Set up an account on Sendgrid and validate the domain name you’ll use to carry the email messages via your DNS manager. In this post we’ll assume the domain is mail.example.com. Copy and paste the needed connection details (such as API tokens) and configure your app to send mail in the documented way (I won’t cover that in this post, as it is out of scope).

You’ll also need to configure your domain’s DNS with an MX record to enable Sendgrid to receive email sent to addresses at your domain. The details are as follows:

Host: mail.example.com (replace this with your actual domain)
Type: MX
Server/target: mx.sendgrid.net.
Priority: 10

Next, we need to tell Sendgrid to handle the inbound messages. For this, navigate to Settings -> Inbound Parse on the Sendgrid dashboard (or use this direct link). Click “Add Host & URL” and enter the details as needed. In the example below, I have configured Sendgrid to handle all email sent to anything@mail.example.com and to POST the parsed email to our app (https://app.example.com) at the /webhooks/email route.

Once that’s done, we can close the Sendgrid manager.

Sending notification emails

Next, we need to make a change to the way we send our notification emails to users. For now, we’ll just focus on the “new message” event, but the same process could also be applied to other types of events in our platform.

The Reply-To field of an email message contains an email address, and tells standards-compliant email clients to use this address (instead of the From address) when users click the “Reply” button on the email.

We will need to add this field to outbound emails notifying about new messages such that Sendgrid can pick them up and we can process them later. There are various ways to achieve this, depending on your existing approach to sending transactional mail, but the below is an example in Python (given we know the message_id of the message we are notifying about):

msg = MIMEMultipart('alternative')
msg.attach(MIMEText(text, 'plain')) # add other fields to the email
msg['Subject'] = subject # add a subject line
msg['To'] = ', '.join(to) # add "to" addresses (there can be more than one!)

msg['Reply-To'] = f'message.{message_id}@mail.example.com'

# Continued mail logic...

In this example, we have specified a Reply-To address that will look a little like this: message.xxxyyyzzz@mail.example.com. This means that when a reply to this email comes in, we can use the To field of the reply email to work out what the message is about - in this case a reply to the message with an id of xxxyyyzzz. We’ll cover this later.

Since we’ve already configured Sendgrid to receive all emails sent to @mail.example.com, then we know that it will try and POST the data back to our app once they’ve been received.

Handling the inbound emails

The final step is to ensure our app can handle the inbound webhooks from Sendgrid.

As we saw earlier, we told Sendgrid to send the details of incoming emails to our app’s /webhooks/email route, and so we will need to set that up. We also know that it will be a POST request and the payload will be standard multipart form data.

Below I have included an example of how we might handle this in a Python Flask app, but the same should be similar for other frameworks and languages:

@app.route('/webhooks/email', methods=['POST'])
def parse_email():
  to_address = request.form.get('to')
  from_address = request.form.get('from')
  text = request.form.get('text')
  subject = request.form.get('subject')
  print(to_address, from_address, subject)
  return jsonify({'processed': True})

We could then deploy our app, send a couple of emails to addresses the app can receive (e.g. test.1234@mail.example.com) and view the app’s logs to make sure everything is coming through OK.

Once we’re happy with the basic setup, we can make changes to the parse_email function to actually do something useful. For example:

  # function definition, etc.
  email_user = to_address.split('@')[0]
  email_topic = email_user.split('.')[0] # `message`
  if email_topic == 'message': # Branch code depending on the type of message
    email_topic_id = email_user.split('.')[1] # `xxxyyyzzz`
    message = db.getMessageWithId(email_topic_id) # Replace with your own database logic
    from_user = db.getUserWithEmail(from_address) # Get the originator user
    if not message or not from_user:
      # Handle not found cases
    if from_user.id not in message.participants:
      # Handle case where the user isn't allowed to reply to the message
    db.createNewMessage(from_user.id, text, in_reply_to = message.id) # Create the message

  return jsonify({'processed': True})

In this example we have included some checks to ensure the user exists and has the permission needed to reply to the message. We parse the relevant message details from the From field as discussed in the previous section.

Conclusion

We’ve now run through the complete end-to-end set-up; users can now click “reply” to emails in order to send content back through to our platform.

Of course, this approach isn’t perfect. For example, changes would be needed to handle the following:

Newly-created messages may include quoted content in the email that we’d need to remove (and things like signatures).
We should add extra text to our outbound email telling users that they can reply to it.

However, hopefully this post has set the scene for how one might go about implementing such a system in their own software!

Open-sourcing projects

Thu, 02 Dec 2021 16:01:00 +0000

I maintain a small number of projects in my spare time. The amount of time I get to work on and maintain these varies depending on my other workloads.

The projects were never designed to be a means of making additional income, and were usually created simply to solve a need that I (or somebody else I know) had!

By open-sourcing them I hope that others will read through the code, and even check for (and report) problems or potentially contribute. The projects are licensed quite liberally under the BSD license.

The following projects are affected by this change:

Treadl (👨‍💻 source code)
SSO Tools (👨‍💻 source code)
Capsule Town (🌍 via web portal) (👨‍💻 source code)
Dotty (👨‍💻 source code)

Server back-ups with Restic

Sat, 27 Nov 2021 10:20:00 +0000

A while ago I posted about how I back-up my personal servers to Backblaze B2. That approach involved adding all of the files and directories into a single compressed archive and sending this up to an encrypted bucket on B2.

Whilst this does achieve my back-up goals (and I use Telegram to notify me each time it completes), it felt inelegant. Every time the back-up executed, the entirety of the back-up file - several gigabytes - would be built and transferred.

In this post I will talk about my new approach, which uses a tool called Restic.

Restic, repositories, and Backblaze B2

Restic is an open-source cross-platform program to back-up files and directories. It works on the basis of snapshots and repositories.

A single repository can contain multiple “snapshots” representing various back-ups. Each snapshot is timestamped and can be filtered by the path of the back-up and the host that the back-up came from.

One of the most useful features of Restic is that each snapshot only stores the representation of what changed since the last snapshot. This means that although the first back-up might take a little while, subsequent snapshots are much quicker.

Repositories can be local (i.e. on a filesystem accessible by the host), or remote. Restic supports a number of remote repository types - from SFTP and RESTful interfaces through to S3-compatible backends and storage services operated by cloud providers.

Restic also supports Backblaze B2 out of the box, which is the repository type that I use.

In the rest of this post I will run through how this can be set-up and automated.

Preparing a Restic repository on B2

First, we need to install Restic. There are instructions for installation on a range of systems on the Restic website.

Next, in the Backblaze B2 console, create a new bucket. The bucket will need a unique name. Whilst there, create a new “App Key”, with read/write access to the bucket. Make a note of the keyID and applicationKey once created.

You can close the B2 console. You now need a file that can be sourced to tell Restic how to access your repository. Create a new file (e.g. in the home directory of the host you want to back-up), called resticenv, with these contents:

export B2_ACCOUNT_ID=""
export B2_ACCOUNT_KEY=""
export RESTIC_REPOSITORY="b2:"
export RESTIC_PASSWORD=""

You’ll need to replace some of the parts of the file as described below:

: the keyID of the B2 key you created earlier
: the applicationKey of the B2 key you created earlier
: the name of the B2 bucket you created earlier
: a strong and private password

Each Restic repository is secured with a password. This is used to encrypt the repository, so if this is lost then so is the associated data. I use my password manager to generate a strong password. It goes without saying that this resticenv file should be kept very safe.

Finally, we need to initialise the repository. To do so, first source the environment file, and then use the init command:

$ source ~/resticenv
$ restic init

You’re now ready to create back-ups.

Creating a new back-up

Creating a new back-up is as easy as running the following (as long as your shell still has the sourced environment as described above):

$ restic backup /path/to/directory

If the target directory is particularly large then the first back-up may take a while. However, subsequent back-ups will be much quicker.

Once done, you can run the following to view a list of all the snapshots held by the repository. You’ll see your recently-created back-up listed:

$ restic snapshots

ID        Time                 Host        Tags      Paths
-----------------------------------------------------------------------
6b25863c  2021-11-27 08:39:44  myhost                /path/to/directory
-----------------------------------------------------------------------
1 snapshot

The next time you run restic backup again, you’ll notice new snapshots are created.

Pruning snapshots

In some scenarios you may not want to keep back-ups around forever. For example, if you provide a service to users you may be in breach of your own Privacy Policy if you keep backed-up data of old deleted accounts.

Luckily it’s straight forward to “forget” old snapshots. The command below tells Restic to hold onto the last 24 hours’ worth of hourly snapshots and the last 7 days’ worth of daily snapshots:

$ restic forget --prune --keep-hourly 24 --keep-daily 7

Recovering snapshots

Back-ups are useless unless you can later recover them! Rustic makes this easy. You just need the ID of the back-up you want to recover (as provided by the restic snapshots command from earlier):

$ restic restore 6b25863c --target ~/restored

The ~/restored directory should now contain the files included in the back-up at that snapshot’s point in time.

Automating back-ups

Finally we can cover a way to automate back-ups. For this we can simply use cron. To begin, install a suitable cron service (if your system doesn’t come with one). I use cronie.

Once ready, add an entry to your user’s crontab by running crontab -e and adding the following line:

15 * * * * . /home//resticenv; /usr/bin/restic backup -q /path/to/directory

The above job will run at quarter-past the hour for every hour the system is running. Note that cron requires that we include full paths to files. The -q flag suppresses output. You’ll need to change parts of the line to suit your needs (run which restic to see where Restic is installed on your system).

Conclusion

I have much more confidence in a tool like Restic to handle my back-ups than a home-grown solution. It’s great to have the peace of mind!

Nightfall City

Thu, 25 Nov 2021 18:36:00 +0000

From the hills of Dusk’s End to the small alleys of Main Street, you feel drawn to the lights of this vibrant metropolis in an uncharted internet territory. The sign reads “Nightfall”.

– Nightfall City

The Nightfall City Gemini capsule (also available via the web) is an internet community in which people can engage with each other and write blog posts and other long-form content.

The community is divided into different districts of Nightfall City: Main Street, Dusk’s End, and Writer’s Lane. These districts allow members (or “citizens”) to post links to their blog posts, zines, or other online articles. From what I can see, people tend to participate in the district or community that best fits them as an individual.

There is also the Midnight Pub - in a small alley just off Main Street. Here one can write posts and thoughts directly.

It’s late. You are seconds away from the main street in a small alley. It’s quieter here, but you can still hear the sound of chatter, footsteps, and cars from busy downtown. The city is buzzing, the streets are like arteries. You see an intriguing place in the alley, with a moon on its door. It reads “The Midnight Pub”.

– The Midnight Pub

I won’t say much more about Nightfall City, or its various districts, as I can recommend taking some time to explore through yourself. I enjoy coming back every now and again to read some of the recent posts. It’s another of those great - yet quirky - corners of the internet.

Webzines

Sat, 20 Nov 2021 14:40:00 +0000

I’ve really enjoyed my recent discovery of a couple of traditional-style webzines. Webzines (sometimes referred to as online magazines, or - in this instance - simply “zines”) are a way of distributing periodic content through the web.

I’m not referring to modern-day online media outlets, but those publications which are typically written by a small number of individuals (often “netizens”) and where the focus is not on advertisements, clickbait, or the mass production of content.

Such outputs differ from blogs, in the sense that they often feature contributions from a number of people and sources, and potentially exhibit some form of editorial process.

They can be used to distribute news, updates, opinion, or anything else that you might find in any other type of periodical.

I like them because they reflect the values of the slow web, they’re informative, can be quite informal and quirky, quick and easy to consume, and they help with discovering new and interesting content from around the web.

I’ve only discovered a few examples so far - such as the OpenBSD Webzine and the Yesterweb Zine - but I look forward to staying on the look-out for more.

Rebel Ideas by Matthew Syed

Wed, 17 Nov 2021 12:56:00 +0000

I feel that this book really resonated with my own thoughts around the importance of diversity in groups and teams.

Matthew Syed’s Rebel Ideas: The Power of Diverse Thinking is a book that examines how effectiveness and output can be dramatically altered through building teams that contain diverse thinkers.

The book considers a number of examples - from both the past and present - and compares and contrasts scenarios where teams expressing different proportions of diverse thinkers can change their performance; sometimes with life-threatening consequences.

One of the most interesting concepts discussed by the author is that around problem spaces. When solving challenges, there is a cognitive space that must be considered and approached in order to produce a more effective outcome. By recruiting diverse thinkers, there is less overlap in each individuals’ own representation of the space, and therefore more of the space can be explored.

Overlap in the space can even be dentrimental to the performance of the group. If there is sufficient overlap, then those members will simply reaffirm each others’ thoughts and ideas to the point where these opinions become dominant and drown out the inputs from members occupying other areas of the space.

In relation to this, the book considers the example of a sprint relay team. In this type of scenario, you’d want to involve the four fastest sprinters you can find - no matter their background - in order to win the race. However, in cognitive challenges - for example, to accurately create financial forecasts - teams consisting of a number of people from different financial backgrounds, models, and ways of thinking, out-performed the individuals known to be “the best in the game”.

Interestingly, the author also analyses how cultural diversity can (but doesn’t always) impact cognitive diversity. In many scenarios, people from different cultural backgrounds may develop their own ways of thinking through their families, social groups, and education.

On the other side of this, two people from different cultural backgrounds, but who attended the same university and were tought the same models by the same lecturers, are likely to inhabit the same area of the problem space, and thus overlap in their ability and knowledge.

There are many more examples considered by the book, and I can strongly recommend reading it - particularly if you manage or frequently work in a team yourself. The general upshot is that the rebel ideas - i.e. those ideas that are truly innovative and ground breaking - are more likely to emerge from teams of people that think different.

On websites and creativity

Sat, 13 Nov 2021 11:13:00 +0000

I’ve recently been reminiscing about the “old” days of the web. They felt much more like expressions of personality and creativity.

These days, most people have social media accounts on mainstream services that act as their sole representation of themselves online. Whilst the content can be different, everyone’s own pages end up looking the same, with avatar images, feeds, and other components having layouts and “look and feel"s controlled by the service - the creativity is lost and things become bland.

This looks like the current trend in such platforms. Previously, MySpace offered some nice ways to customise things on individuals’ pages, but before that the main way to fully control your representation online was to own your space - i.e. have an actual website you can control.

Personal websites are still very much a thing, and in many ways are making a bit of a comeback. The concepts are described nicely in Aral Balkan’s blog, in which there are also arguments that personal websites can interconnect - an initiave also made possible through technologies like ActivityPub.

Either way, modern personal websites tend to all be quite minimalistic and sleek. People often share tools and software (e.g. CSS libraries), which are good ways to get something off the ground or to help enable those that are less tech-able. Whilst this is also great for providing useful reading experiences, enhancing accessibility, and encouraging mobile-first principles, I worry that perhaps something is lost in the expression of creativity.

I recently stumbled across the Yesterweb webring. For those not familiar with the concept, by joining your site to a webring it gives your website additional exposure, since each member of the ring publishes a link to the next site along. As a visitor, they’re a great way to explore and discover new sites.

This particular webring showcases websites with very distinctive 90s vibes - many of which are hosted on Neocities. They feature bright colours, lots of scrolling banners, animated GIFs and more. Some also bring back concepts like visitor counts, guestbooks and chatboxes. It’s a refreshing and fun experience.

It’s great to see and to explore! I will show and link to a few below, but I recommend taking a look through yourself too.

sadgrl.online

bytemoth.neocities.org

cinni.net

auzziejay.com

haxrelm.neocities.org

Our DIY bathroom project

Wed, 10 Nov 2021 21:04:00 +0000

I’ve recently posted about our home, in which we’ve completed a few DIY projects, such as renovating the garden and building a small loft conversion (amongst other things!). Today I’m writing about a project we did on the bathroom in the house.

When we first moved into the house, which was an old student dwelling in disrepair, the first floor had a stange configuration containing a separate toilet and shower room. Both of these rooms were tiny, old, and damp. There was no space for a bath and also no window in the shower room and the extractor fan didn’t work.

In this post (complete with images) I will walk through the changes we made in order to turn this into a more liveable bathroom. We are by no means experts, but we managed to do the project pretty much by ourselves, with a fair bit of Googling and only one plumbing mishap!

Initial state

These images were taken from when we first viewed the house and also the first day we moved in. You can see the grotty toilet, shower, and the small, gloomy rooms.

Knocking down the walls

One of the aims of the project was to create one single room containing a toilet, shower, bath, and sink. This meant that the first thing for us to do was to join the existing toilet and shower rooms together, as well as to take part of the bedroom next door.

Luckily these were all stud walls, so aside from the mess, this wasn’t too much of an issue given a bit of muscle and a sledgehammer.

In the image below you can see where the previous toilet and shower rooms were. And the remains of where the two walls were. We took an extra window from the bedroom too.

Sorting out the plumbing

To knock the walls down we needed to turn the mains water off to the house in case we damaged any pipes.

We then needed to remove the shower fittings and tray properly, and correctly cap all exposed water inlet pipes.

Building the new wall structure

We needed a wall between the new bathroom space and the bedroom (for obvious privacy concerns). For now we just built the structure and would come back to finishing it off later with the plumbing.

Trying to get the bath in

Probably one of the most challenging parts of this project was getting our ridiculously-shaped new bath into the room.

In the picture below you can see the two doors into the bathroom. We would later build over the one we (eventually) got the bath through!

Taking down the old plasterboard

Removing the walls had caused damage to the old plasterboard. We also wanted to replace some of this with moisture board instead in order to make the walls more water resistant.

We broke and stripped off the plasterboard from the inner side of the landing wall. In the images below you can also see the structure of the new wall we had started building where the old shower room door was.

This required a bit of cleaning up. Notice the bath now (somehow) in the room!

Continuing with the wall

With that out of the way, we could then add new plasterboard and moisture board to all of the new and exposed walls.

Some drainage plumbing

We built a small raised platform that would more easily enable us to run drainage under the floor, which you can see in the picture below. We also got the bath plumbed in.

Finishing off the floor

To complete the floor we added plywood and some simple lino flooring.

Proving that the bath works! With inlet and outlet plumbing done.

Tiling

To complete the wall around the shower, we added tiles in a herringbone style. Naturally we left the hardest bits (i.e. the edges) to last.

We also added in some of the shower plumbing through the wall so we could tile around it.

Installing the new shower tray and glass

Next, we needed to install the shower tray and glass. The shower would be a walk-in shower without a door.

We also grouted the tiles and sealed the shower tray.

Building the sink

We bought a nice sink bowl and some hairpin legs to build the support for the sink, as you’ll see in the pictures below.

We also added some skirting rail, dado rail, and some panelling effect for improved aesthetics.

We plumbed the sink through the wall - just like the shower.

Adding some paint

The next jobs were to plaster the remaining exposed plasterboard and add some paint. We also added some additional panelling to the wall.

As you’ll see from these photos, we went for a dark blue colour for the main part of the wall to match the tiles.

Some more tidying up!

Closing off the door

Remember that old shower room? The wall to that still needed to be completed. We patched the hole with some plasterboard, plastered it and its surround, and painted it the same colour as the landing.

Once done, the old doorway was nowhere to be seen.

Finishing touches

Finally, we bought some mirrors, pictures, furniture, and some fittings (such as hand-towel rails) for the walls. We also added a rug and some hanging plants, which sort of water themselves from the shower’s condensation.

We plumbed in a heated towel rail next to the shower, added some coving, and some light strips.

Automatically scanning for malicious user-uploaded files

Sat, 06 Nov 2021 15:38:00 +0000

If you run a service that accepts file uploads from users, and then subsequent re-download by other users (such as images), then your service is potentially at risk of becoming a system for distributing malware. Without safeguards in place, bad actors could potentially use your service to upload harmful files with the intention of them being downloaded by other users.

Services like Google Drive and some email providers will automatically scan files for malicious payloads, but if you - like many people - rely on more basic object storage for storing files for your apps, then there may be less default protection available.

Luckily there are a number of methods available for addressing this.

Overview

Whilst the concepts are mostly generic and framework/infrastructure agnostic, in this post I’ll focus on a process that leverages Amazon S3, Lambda, and ClamAV. ClamAV is open-source antimalware software that can be executed as a binary without requiring a GUI.

In this post I won’t include code (for brevity), but I will walk through the key stages, which are:

Periodic refresh of malware/virus definitions
Running the antimalware check upon new file upload
Denying uploads to “infected” files.

Managing and refreshing virus definitions

This stage allows ClamAV to keep up-to-date and recognise the types of files that might be infected. It involves three steps:

Obtain a binary capable of downloading new definitions
Write a function to run the binary
Schedule the function to be run periodically

ClamAV provides FreshClam - a tool for updating and managing a local database of virus signatures.

In order to obtain a freshclam binary for use in Lambda, I recommend installing ClamAV on an EC2 instance running Amazon Linux 2, and then extracting the tool from the filesystem (e.g. by running which freshclam). Note that you may also need some other library files for the binary to work (you’ll notice errors which will be pretty self-explanatory).

Once you have the freshclam binary, create and upload a Lambda Layer containing the binary. Depending on the runtime your Lambda function will use, you will need to store the binary at a specific path in your Layer. Give your Layer a suitable name, such as freshclamLayer.

Next, we need to create a Lambda function. It doesn’t matter what runtime you use, so long as you can use it to execute the freshclam binary from the filesystem as some form of subprocess. Once ready, upload it to Lambda and reference the freshclamLayer Layer so that the binary can be made available to the function.

The function should outut the generated signature database to an S3 bucket. As such, your function will need to have an IAM role that enables write access to your chosen bucket.

Finally, use CloudWatch Events to schedule your function to be run periodically. For example, you could update the signatures once or twice per day, depending on your needs.

Running the antimalware check on new files

This stage uses the virus definitions to decide whether a newly-uploaded file might be infected. It involves two key steps:

Create a Lambda function that runs the virus scanner
Create an S3 trigger that runs the function when new files are uploaded

In addition to FreshClam, ClamAV also provides a tool called ClamScan which can be invoked on specific files or directories in order to check them for malicious content.

Obtain the clamscan binary as described in the previous step, and bundle this into a Layer (again, as above), named something like clamscanLayer.

Next, create a new Lambda function (again, choose a runtime that can invoke binary subprocesses). The function should check the event passed to it in order to determine the path to the file that was uploaded, download the previously-uploaded virus signatures from S3, and then run clamscan against the target file. The output from the binary should be monitored to understand whether the file is suspected to be malicious.

If clamscan determines that the file is malicious, then use an S3 tag to mark the file as infected. For example, you could create a tag named Infected and pass a value of true or false depending on the scan’s output. For this, you’ll need to give your function relevant IAM permissions to enable object tagging in S3 and also read access to the bucket where the virus definitions were stored in the previous stage.

Also make sure that your function uses your clamscanLayer Layer so that it can access the relevant binary.

Finally, configure the S3 bucket responsible for holding user uploads in order to add a new trigger that invokes your scanning function each time a new object is put to the bucket.

Restrict downloads of infected files

The final stage involves telling S3 to forbid access to infected files. To do so, simply create (or modify) the bucket policy for the user uploads bucket such that it denies the GetObject action for any objects that have the condition of an Infected tag with the value of true.

Conclusion

In this post I’ve provided a rough overview of a process that allows for scanning user-uploaded files for malicious content. Hopefully this might help if you’re looking to make your services more secure for your users!

A little trip to A&E

Wed, 03 Nov 2021 18:45:00 +0000

On Sunday I slipped and fell in the pouring rain. I landed hard on my side and ended up dislocating my shoulder.

I didn’t really want to try risking it and putting it back in myself (sensibly, as it turns out!), so I got a taxi to A&E (an emergency ward in UK NHS hospitals). I was lucky enough to be seen within a few minutes; I then had examinations, had X-rays, and was given gas. When the doctor got round to me, he managed to push it back into position after a few minutes of moving it around - accompanied by a very loud click.

I was sent home in a sling and now have a couple of weeks of physio ahead. I’m lucky that this was the first time I’ve ever actually needed to visit A&E - for myself, at least.

It’s a little ironic that I had just written about exercising and keeping active, and now I’m going to be out of full action for a little while!

Below you can see a picture (taken a few minutes after the fall) showing my left arm a little lower than where it’s meant to be!

Thinking positive

Sat, 30 Oct 2021 18:45:00 +0000

It’s been a weird 18 months. Before pandemic-initiated changes, our daily lives might have involved getting up and travelling on some form of commute (either by walking, public transport, car, or something else) to a place of work each morning, before reversing the process every evening.

From my perspective, the change to working remotely from home has a number of benefits. Although I do miss seeing people on a daily basis in the office, the flexibility of remote work certainly outweighs any downsides. This is a feeling also echoed by many across my team - especially those with children or other family life that they need to work around.

As the months passed, however, it was all too easy to begin to slip into the routine of getting up just before morning meetings began, slide into a t-shirt and joggers, and head over to a desk to begin work for the day.

Without the work-life separation that the commute and workplace provided, I - like many people - started living, what felt like, a Groundhog Day-style life. I began to care a little less about my appearance, my health, and routine, since every day felt the same.

I quickly realised that this was not good for my mental wellbeing and general happiness. I usually have quite a positive outlook but I felt myself slipping a little into a mild depression.

I needed to take action, and I did. These days I feel much more adjusted to mixing home and work life. I’m generally happier and feel positive and pro-active again. Below are some of the things I do to build routine and positivity into my life.

Get up earlier

Without needing to spend time commuting, it’s easy to get up later in the morning. This can make you feel sluggish and give you a more negative view of your work day ahead.

I recommend setting your alarm clock earlier. Give your mind time in the morning to wake up. Spend time journaling, reading, talking with family, or anything else. This helps to break the sleep-work-sleep routine.

Make your bed!

This sounds like (and is) a small thing, but taking the time to make your bed each morning, and tidy up a little, helps to restore your own pride in your appearance, as well as to give you a more positive outlook.

Get a morning routine

Treat your day as if you were going to go to a workplace. Have a shower, shave (if that way inclined!), style your hair, choose clothes you like, and so on. You’ll feel better for it than just pulling on some easy pyjamas.

You’ll also feel more alert during video meetings and will be able to contribute more positively.

If you need it, build some separation

Some people find it hard to get their minds into gear to begin work in the morning, especially if they work in the same place in which they live.

If you are one of these people, then build in some separation. After your shower, go for a quick ten minute walk round the block or a local park, get a coffee, listen to a podcast.

Exercise

Exercise had one of the biggest positve impacts on my daily life. I’ve already posted about running before, and I think this can be hugely beneficial for one’s mental wellbeing, as well as physical fitness.

You don’t need to join a gym (though, of course, that can help); just find something you enjoy doing - swimming, walking, or anything else - and that you can do a few times a week in and around your routine.

Personally I try and run three times a week and do some form of resistance training every day. I find it quite meditative and it’s always a good thing to be a bit fitter.

Organise your day

There’s nothing worse than sitting down to start your work day, and feeling lost with too much (or too little!) to do.

One way around this is to use to-do lists, and to write down things you hope to achieve during your day. You could write these the day or evening before, or during the extra time you have before work from getting up earlier in the morning.

There are lots of great tools for this. On my iPhone and Mac I use Sorted, which I find useful for combining tasks and events into one place. Other people use different tools with great success, or even pen and paper. Choose something that works for you.

Break up your day

If you find it tedious without the natural distractions a normal office and other people povide, then build these in too.

Go and work from a local coffee shop in the afternoons, have a lunch-time walk, do chores or exercise between meetings - all of these can help introduce variation into your day.

In general

Home (or remote) working is certainly not for everyone. Whilst I prefer it, I can fully appreciate why some may prefer spending more time in the office with other people. If and when (if not already) your team are back in an office then certainly take advantage of it.

If you will continue to be working from home, and want to mix things up a bit and feel more invigorated - perhaps by following some of the ideas in this post - then I advise finding something that sticks. Routines are a bit like a diet; they only really work if you can continue to live with them and can actually enjoy them.

If you’re in this latter boat, I’d love to hear from you, particularly if you have any of your own tips for keeping positive in mixing your home and work life!

Taking FreeBSD for a spin

Fri, 29 Oct 2021 08:44:00 +0000

Background

I’ve recently noticed (and read) more and more posts discussing *BSD systems. Creations like the new (and excellent) OpenBSD Webzine and blogs (such as Rubenerd’s and Solene’s) do a great job in raising awareness of the family of operating systems.

I’m pretty familiar and comfortable with Linux, having spent many years using it as a daily driver (I am back on macOS full-time right now). Whilst UNIX systems share a lot of similarities, I’ve never properly used a BSD system before.

Research

My interest was piqued, and I wanted to try one out for a few days. I did some research for good jumping-off points.

From what I read, FreeBSD seemed the best for someone coming from the outside world for the first time, due to its (generally) wider compatibility with known software. NetBSD is designed to work well on a wide range of architectures and platforms, and OpenBSD has a strong focus on “correctness” and security. It’s also worth checking out the other BSD operating systems for a wider understanding too.

Installation

I thought I’d get going by installing FreeBSD and trying it out for a few days. Unfortunately I did not have any suitable hardware to hand for the installation, as I currently only have a MacBook Pro.

In the end I opted to just use VirtualBox - it wouldn’t give the “baremetal” experience, but it’d be close enough to get the feel! I downloaded an image from the download page and it ran without problems.

During initial experimentation I ran out of storage space on the primary volume quite quickly, and so I recommend extending the virtual disk’s size (which can be done from within VirtualBox).

Impressions

Once installed, my initial impressions were great. The system was super easy to set-up and configure. Everything is logically organised and the documentation is fantastic.

I was able to easily setup a graphical envivonment by installing X, display managers and desktop environments. These all worked without a hitch and with simple edits to /etc/rc.conf to get things starting properly at boot-time.

I could install Firefox without problems, and sound (e.g. from YouTube videos) seemed to work out of the box. Development tools, such as Node, were also a simple pkg install away.

Hats off to the FreeBSD community for creating such a rich set of available packages and for all of the hard work in porting these from other platforms. The overall set-up and usage was simpler, better documented, and more logical than comparative Linux systems, such as Arch (which is still my go-to Linux distro).

The entire system almost feels like a pleasure to use and administer, and I look forward to learning and trying more.

Pitfalls

This may be more related to VirtualBox limitations than FreeBSD itself, but I was unable to route USB devices (such as microphone and webcam) through to applications on FreeBSD. For example, in order to use Google Meet or the Zoom web client. I’d like to try these again on a non-virtualised system when I get the chance.

Docker is a known missing-entity on FreeBSD, and so I would not be able to build images and run containers directly. However, the concept of jails seem super interesting and I would like to investigate these further.

Aside from these minor things, there is no reason why FreeBSD couldn’t become an excellent daily driver for my work in particular.

Worth noting

A few other bits worth mentioning are below.

Documentation

As I’ve already alluded to, the documentation for FreeBSD (and the equivalent for other BSD operating systems too) is fantastic.

The handbook contained all of the answers I needed when spending my week with the system, and it also makes generally interesting reading too!

There are also very well-attended forums, on which your answer is probably already available without needing to ask the community an additional question.

Distribution

Unlike Linux distributions, which typically ship with a separate kernel and a number of independent packages, FreeBSD (and other BSD systems) is installed as a base system containing kernel, build tools and related software all together. This makes updates and maintenance less fragile and more predictable.

Ports system

FreeBSD offers two (primary) ways to get new software installed on your system - ports and packages.

The FreeBSD Ports Collection provides the files and resources required to build the software from source (thus allowing pre-compilation tweaks, if this is of interest) and packages allow the user to directly install pre-built binaries.

Since larger pieces of softare can take a long time to build, many new users - myself included - opt for installing packages directly. Advanced users and organisations may even set up their own server and ports collection to automatically pre-compile packages to their own specification.

General

Seasoned FreeBSD users often describe their systems as “unsurprising” - in a good way. Things are just logical, they don’t unexpectedly break, the package manager doesn’t introduce conflicts, and everything is stored where you expect it to be.

One of the principles followed by FreeBSD developers is the Principle of Least Astonishment (POLA), defined as the following:

As FreeBSD evolves, changes visible to the user should be kept as unsurprising as possible. For example, arbitrarily rearranging system startup variables in /etc/defaults/rc.conf violates POLA. Developers consider POLA when contemplating user-visible system changes.

FreeBSD Handbook Glossary

Going forward

My next steps are to get hold of some older/cheaper (but well-supported) hardware to properly try out FreeBSD on a more permanent basis. I may also spin up a VPS in the cloud running FreeBSD (Digital Ocean offers these) to try running a couple of my lower-impact services through jails.

I also noticed that FreeBSD offers PINE64 images on their downloads page - I’m intrugued to see what my Pinephone thinks about the system too.

Beyond this, I’d like to also try OpenBSD, and to more fully understand the differences. The Why OpenBSD Rocks website has some interesting concepts I’d like to read through.

A mini loft conversion

Sun, 24 Oct 2021 20:43:00 +0000

Following on from my previous post about renovating our garden, I wanted to write an entry about another project we’ve recently completed.

Our home is a Victorian townhouse over three storeys, but the top floor has only one bedroom and a bathroom. The rest of that floor is essentially attic (or loft) space, accessible through a hatch in the ceiling of the middle floor.

As such, the loft is quite long, and far too big for us to use as-is. In fact, I’m a big believer in that if you need to put something in the loft then you probably don’t actually need it at all. I’m pretty sure we only have Christmas decorations stored up there.

This felt like a big waste of space! The loft was a decent height and one could (mostly) stand up in some of it. It also had old fashioned windows in the pitched roof - I have no idea why.

Based on this, we thought we’d try to convert half of the loft into actually usable house space.

The rest of this post briefly describes what we did to accomplish this.

Accessing the new “room”

The stairway up to the top floor has a switchback in it. The mini landing at the switch point happens to be at the same level as the loft floor, and we were able to knock through the wall on that landing in order to reveal the loft behind.

Knocking through the loft wall

As we knocked through, we revealed the loft space. You can see the windows, roof beams and insulation (along with promised bag of Christmas decorations) in the image below. Sorry it’s a bit dark.

Adding a door

Since the aim was to create a bedroom, we needed to add a door (for obvious privacy reasons). Luckily we had a spare one lying around from a bathroom project (I may write about this soon!) that we were able to cut to size and shape, and then install within a door frame.

In the image above you can see the original “sticks” (I’m sure there’s a more technical word for these) that form the wall structure. Afterwards we were able to plaster and paint over this, as shown below.

Back to the room

Next we added a floor to the room, insulated the (new) roof space and added plasterboard to form a celiling and wall.

In the image below you can see the new floor and the plastered ceiling and far wall. We added hatches to the wall to provide access to the rest of the loft space now behind this room

We also hired an electrician to add some nice lighting, switches, and electrical outlets.

Finishing touches

To finish the room off we washed and coated the brick wall in PVA. This allows us to keep it as an exposed brick wall, but without having dust or dirt coming off from it.

We also painted the walls, ceiling, and beams, and added some wallpaper. We added some flooring, skirting and some shelves. Finally we added some furniture, and the result is below.

The little window lets through a surprising amount of light. It’s a small bedroom, but it’s cosy and it’s great to have the extra space for when we have visitors!

Extraterrestrial by Avi Loeb

Sat, 23 Oct 2021 16:42:00 +0000

I often enjoy books that try to take a different view on known events. I don’t mean consipiracy theory - more around thinking laterally or “out of the box”. Such ways of thinking often inspire ideas that drive innovative change, and it’s important in order to counter “group think” or simply accepting what’s easiest.

One such book that tries to do this is Extraterrestrial by Avi Loeb.

I finished this book a couple of weeks ago, and have been thinking about it since. Whilst I am not an astronomer - and know very little about the science of astrophysics and the probabilities of celestial events occurring (simply understanding how the distance to the moon is calculated still baffles me) - the themes around the automatic rejection of theory by scientific communities based on pre-conception makes me consider all of the other possible scientific theories we may have thrown away, almost by default, as we strive to learn more as a human community.

The book’s subtitle is “The First Sign of Intelligent Life Beyond Earth”, and the focus is on the event of an intersetellar object travelling through our solar system back in 2017. Whilst it is not uncommon for objects to be seen moving around our solar system (think planets, asteroids, comets, etc.), the author posits that the particular attributes of this object are such that it is unlikely that it can be something that was created naturally.

The author discusses the size and shape of the object, the way it moved on a non-natural trajectory through the system, along with other factors, and essentially asks the question to the scientific community, “why could this not be the creation of an extraterrestrial intelligent community?”.

The book is about pooling theories and scientific approaches to proving or rejecting them based on available evidence. The author talks about his peers and how they often simply reject extraterrestrial intelligence as the basis for candidate theories simply due to the fact that “because, aliens” is such an unlikely option. Loeb argues that, in these cases, one shouldn’t just reject such theories based on pre-conceived notions of likelihood, but should instead consider all viable theories until the evidence grows sufficiently enough to disprove them.

Avi Loeb, the author, is a well-published professor of science at Harvard University, and makes robust and compelling arguments whilst trying to remain neutral throughout the book. I enjoyed it; the book is non-technical and is written in a way that makes it easy to consume if you are interested in this space.

This Week in Tech (TWiT)

Sun, 17 Oct 2021 20:15:00 +0000

Another podcast I frequently listen to likely needs no introduction of its own. The This Week in Tech (or just “TWiT”) network’s flagship podcast - also called TWiT - must be one of the longest-running tech podcasts.

The podcast series started back in 2005. It runs weekly, with episodes recorded live each Sunday evening and made available via podcast clients shortly afterwards.

It is hosted by Leo Laporte, who is joined by interesting and varied panelists from across the tech sector. Episodes feature light-hearted discussion of recent news and insights from the technology world.

Episodes are pretty lengthy, and they keep me going for a couple of days each week as I walk the dog. However, by listening I feel that I get a really good overview of the relevant news in the space. Leo and the guests are always great to listen to, it’s often funny, and I always learn something new. It’s not too technical and would be easily consumable by anyone interested.

If you’re looking to add something new to your podcast library, then I can certainly recommend subscribing for general weekly tech updates. I personally listen to it as an audio podcast, but episodes are also available via video.

Giving a talk at the BSV Wales meet-up

Wed, 13 Oct 2021 18:41:00 +0000

Last week I gave a talk at the Bitcoin Association BSV Meet-up for Wales, hosted by Tramshed Tech in Cardiff.

Before learning about this meetup, I had not heard of BSV - either from a technology or currency perspective. However, as well as promoting an interesting project, the event welcomes showcases from technologists working across the blockchain space.

At Simply Do we have recently completed a project that aimed to leverage blockchain distributed ledger technology to help protect and manage IP assets in complex international and cross-domain supply chains. The project was a success and this was what I - along with my colleague, John - presented about.

The other talk at the event was around the circular economy approaches undertaken at Celsa Steel, which was a mostly-new concept to me, and certainly highly interesting and topical at the moment.

It was great to be back at in-person events after 18 months of remote-only meetups. There were some excellent questions and comments, I met some interesting people from around the blockchain space, and I look forward to attending more in-person as events continue to open-up.

Dotty

Mon, 11 Oct 2021 17:19:00 +0000

A few years ago I was in the position of needing a solution to backup and sync dotfiles (configuration files for various pieces of software) across my machines.

Specifically, I had Mac computers and Linux servers, and needed a way to nicely keep these files up-to-date between them. For example, I may have spent some time crafting and tweaking files - such as my .vimrc and .tmux.conf - and needed a way of ensuring all of my devices could access the latest version of these files.

A few other tools exist for this - such as Mackup and homesick - but I wanted something quick and easy to get off the ground and distribute to other machines. Preferably a single binary.

This gave me the idea for Dotty - a hosted manager for dotfile-syncing across devices. It allows for downloading a single binary (pre-compiled for Linux and MacOS devices), from which one can login and pull/push configuration files as needed.

The service also exposes a RESTful HTTP API for integrating into other workflows. Both the client and the backend are written in Go, and backed by a MongoDB database.

Whilst I try to maintain Dotty when I can, and keep the service running for those who still use it, I personally don’t use it anymore and so do not plan to add additional features. These days, I think tools like Syncthing and Nextcloud offer more robustness, security, and flexibility, and I prefer using these for ease of use.

This is Going to Hurt by Adam Kay

Wed, 06 Oct 2021 20:27:00 +0000

Having recently read The Secret Barrister, which I loved, I was recommended to also check out This is Going to Hurt by Adam Kay.

The book is similar to the Secret Barrister in that it’s a collection of insights and stories from a working professional - this time a hospital doctor. The book is subtitled Secret Diaries of a Junior Doctor.

The author tells the story of his experiences in completing medical school and beginning work in the UK National Health Service (NHS) system. NHS doctors generally follow a pre-defined pathway from “F1” through to consultant (or slightly different if a GP), and this book describes experiences of the author as he works his way through this process.

Whilst he spends most of his time in obstetrics and gynaecology healthcare, it is clear that his experiences reflect the entire profession as a whole. The book is well-written and funny, yet the author effectively conveys the every-day stresses, the sleepless weeks and non-sensical shift patterns, and the (literal) life-and-death responsibilities thrown at junior doctors every day.

The fact that junior doctors can’t just “clock off” when their shift ends (what if they’re half way through an emergency 4 hour operation?), that any over-time is not paid, that the majority of social events are just not feasible for them to join - these all mount up in unimaginable ways for young medics in their mid-twenties.

Of course there are very rewarding moments too, and the author makes it very clear that this is what most doctors stay in their jobs for: the responsibility to actually help people and make positive differences in other people’s lives. It’s definitely not for the money.

I come from a family of doctors myself, but this book really throws things into perspective as an honest reflection of the publicly-unseen time, effort, and energy that goes into the NHS from its own people.

This is another book I can certainly recommend.

Pinephone: Manjaro and Phosh

Sat, 02 Oct 2021 15:04:00 +0000

It’s been a few weeks since my last post about the Pinephone. Since then I have been playing further with a different graphical shell and have been trying out new applications.

In that previous post, I noted a few points that made the phone tricky to use as a daily-driver. However, it should be noted that this was (intentionally) based purely on the phone’s out-of-the-box configuration. I fully meant to continue exploring and to discover ways in which the device could become more of a useful daily use phone for me. This post forms part of that journey.

Installing Manjaro with Phosh

Over the past few weeks I have been playing more with the Phosh graphical shell, still running on Manjaro. Phosh is developed by Purism, primarily with the aim of creating a useful mobile shell for their Librem 5 phone.

To install this system on the Pinephone, I downloaded the appropriate image, dd’d the image to an SD card (as described in the Pine64 Wiki), and booted the phone from the SD card.

The boot priority of the device dictates that the phone will initially try and boot from the SD card by default, which is handy if you have a few SD cards lying around with different configurations (like me!).

Once booted, the default user (manjaro) has the password 123456 - this allows for unlocking the device from the lock screen.

Initial impressions

The first thing that struck me is the speed and responsiveness of the new shell. It is significantly faster than the default Plasma Mobile interface - both in terms of application launch time and in general touch responsiveness throughout the UI.

This enhancement alone makes the device much more of a pleasure to use; following my intentions in real-time.

Application usability

Applications that were previously unusable, due to scaling, now seem to work much better. For example, Bitwarden (installed through the “Software” package) now scales much more nicely to the phone’s screen, allowing me to properly login (as shown in the image below).

Another key improvement is the ability to copy and paste text between applications. I just could not get this to work under Plasma Mobile (maybe this was just me?), but this change now lets me copy and paste passwords from Bitwarden, and also allows me to copy login tokens from a browser.

This has enabled me to fully login to Tootle - a great application for Mastodon. I can view my home, local, and federated timelines, and I can easily post toots (including images with text descriptions).

I can also now change the playback speed within the Podcasts app - another great quality of life improvement. Before, the UI would not let me select the appropriate radio button.

Other improvements

Beyond these, I can now use the Geary application to log into my Google Mail accounts, which works well.

I also installed the “Discover” application (from “Software Manager”), which allows easy discovery of apps from Flathub. Alternatively the Flatpak command can be used to install these from the terminal (or via SSH).

Other quality of life features allow for quick media control actions from the lock-screen (a feature which also worked under Plasma Mobile), and an easy-to-use app-switcher.

Moving forward

I’ve really enjoyed using the phone under Phosh, and I am looking forward to continue to experiment with new configurations, software, and processes. For example, setting up Syncthing to automatically make backups of photos and files, taking advantage of the Nextcloud integration, and - just generally - building out processes to improve daily life with the phone.

As always, any recommendations are very welcome! Please just get in touch.

The Secret Barrister: Stories of the Law and How It's Broken

Thu, 30 Sep 2021 16:58:00 +0000

The Secret Barrister: Stories of the Law and How It’s Broken is an oustanding book. In my opinion it is easily the best book I have read in the past year - certainly the most interesting.

The book is written by an anonymous barrister working in the criminal justice system for England and Wales. They explain the current inadequacies of criminal justice through a mix of interesting real-life and often first-hand stories.

I found the history of legal practice super interesting. The author also explains various terms and processes - many of which I had heard of before but never really fully understood the meaning.

Mostly, I found the book shocking. Many people are unlikely aware (myself included, previously) of the current state of the legal system. The levels to which it is under-funded, under-resourced, and - as a result - the volume of mismanaged cases that can easily result in innocent people getting remanded and guilty people walking free.

The author’s self-coined “stack-’em-high, sell-’em-cheap” model sums it up perfectly, in which magistrates and the crown court system - including the CPS - are just not given the time and resources needed to deliver accurate and honest justice.

The book is extremely well-written (and narrated - my version by Jack Hawkins), clever, and witty. It’s a book I can recommend to anyone. I just hope I never need to be involved in the UK criminal justice system!

Two months with Duolingo

Sat, 25 Sep 2021 19:10:00 +0000

English speakers’ privilege

I’ve always been crap at learning languages. From an early age my parents would encourage me to learn French, and I picked up Spanish and German at around the GCSE level too (exams we take around the age of 16 in the UK). But things just didn’t really ever sink in.

Part of this would have definitely been down to a childhood unappreciated privilege of understanding English as a first language. As I moved more into the technology domain to follow my interests (and then later for education and work), I was lucky that everything I needed was also in English (from programming languages, technologies, documentation, and more) - of course driven primarily by the US big-tech sector.

When I used to play online games as a teenager I’d chat to people who’d never even been to an English-speaking country, yet their grasp of the language far exceeded even my own as a native speaker. When I went to university, and more specifically in my post-grad studies, I met people from around the world and everyone would easily be able to speak to each other in English. Even if everyone in my student group spoke another language more fluently, they’d use English for my benefit.

Whilst these were, of course, acts of kindness, I would always feel embarrassed, uncultured, and often just plain ignorant. Here were people who had made an active effort to learn (at least) one additional language on top of everything else they had going on in their lives as children, teenagers, and young adults. Personally, I could just about tell someone the location of the local library in Spanish (and perhaps order a few drinks), but that was essentially my limit.

Over the past few years I’ve been spending a bit more time in Spain - encouraged mainly by an increased family presence there. This summer I made the decision to actively try and increase my Spanish language ability so I could do more than simply “get by”.

Duolingo

Shortly before Duolingo went public earlier this year, I re-downloaded the app (having used it briefly for Welsh a few years back) and committed myself to it by purchasing the Plus membership. I know there are other players in the space (such as Babbel), but some of my family were already using Duolingo and I thought the social aspect might keep me more motivated.

Since July I’ve used the app every day to learn Spanish, and whilst some of it seems familiar from my school days, I’ve learned a huge amount in the relatively short time I’ve been using it.

Things just seem to sink in much more. Sure, it gets a little repetitive, but the logic and the syntax of the language has become much more clear to me in a way that it never did before. The tips that go alongside each set of lessons explain things succinctly and with good examples.

The app has quickly become pretty addictive (though in a way that I don’t mind), and is probably my most-used application (certainly by “battery time”, as shown in the image above!). The “lightning” rounds (where you have to complete a lesson in a fixed amount of time) are super-fun and the “legendary” lessons (where you get three “lives” with which to prove you understand the latest learnings) add lots of extra tension.

It’s a way of learning that I didn’t expect to be so engaging and effective - for me, at least. It seems as though this model of learning is flexible enough to be used to teach other fact-based subjects, such as history or geography (or even science). It’s also elastic enough to support personalised learning, adapting well to the user’s own pace and ability.

Whilst it tries to encourage listening and speaking the language too, and also includes “audio lessons”, it’s very much focused on literacy and understanding. I think I would still struggle to hold anything more than a very basic conversation, but having the foundation in place (and a bit of confidence to try it out more when out and about!) will I hope help with this.

I look forward to continuing my learning!

Accessibility is for everyone

Wed, 22 Sep 2021 19:30:00 +0000

For many developers, the notion of adding accessibility features - such as image alt text attributes to web page images and integrations with host usability enhancements, such as screen-zoom and text-to-speech - might feel like a chore. Especially for those still in the startup or “do things that don’t scale” phase.

It should no longer be about “adding” accessibility features any more than one these days “adds” a mobile-friendly version of their site (long-surpassed by responsive design and mobile-first pricinples) or even “adds” a button to perform a specific task. Accessibility features are a core part of any product, and should factor into the software development process right through from requirement engineering through to planning, design, implementation, and testing.

Avoiding accessibility requirements is not only exclusionary and selfish; it results in sub-standard software. If you’re developing an MVP solution or software purely to be tested, by ignoring accessibility needs you automatically exclude whole segments of your audience, whose valuable insights and inputs at the important early or late QA phases will never make it into production before it’s too late.

It’s been well-known for a while that diversity in the software development process enhances the end product and creates a better and more well-rounded solution for everyone.

On this point, the other thing about accessibility features is that they improve usability for all, and gives everyone more freedom. Some people rely on the features and supporting technology more than others, but we can all take advantage of it. There is no downside to its inclusion in software development. As well as improving the experience for disabled people, accessibility factors can enhance usability for those using smaller screens, text-based browsers, or people on slow connections.

Going back to the example near the start of this post, one advantage of alt text attributes, and semantic markup, are that they empower screen readers to perform their job more accurately and helpfully to their user. Technology exists to help us, and companies like Apple and Amazon take advantage of this now to announce messages to us via AirPods whilst out on a run and to tell us about our day ahead perhaps whilst driving to work - amongst a growing array of usability and quality-of-life features.

It’s not just big tech that are expected to fulfil these needs (though we can learn from their examples). Many modern and popular frameworks today - such as the Create React App project - include code linting rules that help to enforce various accessibility traits for content like images and emoji. Additionally, UI component libraries, such as Bootstrap, automatically include various aria attributes in their components. This makes it easy for early-stage startups - or one-person teams - to enhance usability.

You yourself may not need to use accessibility features right now, but it’s likely you will at some stage in your life - especially as interfaces continue to develop in complexity through to enhanced gestures, spatial environments, and AR and VR. It’s up to all of us to encourage and practise accessibility-focused development so that it remains a growing priority in the decades to come.

My website is by no means perfect and should not (yet!) be an example to follow. However I continue to learn more about the space and endeavour to maximise usability and accessibility through factors like alt text, simple interfaces, using plain and semantic markup, improved reader mode optimisation, and by making all blog content available via RSS.

Pacman: signature is unknown trust

Sun, 19 Sep 2021 12:15:00 +0000

I was performing a standard system upgrade on an Arch server this morning and received the following messages (maintainer details redacted):

$ sudo pacman -Syyu

... # Download of packages

(159/159) checking keys in keyring                 [######################] 100%
(159/159) checking package integrity               [######################] 100%
error: fail2ban: signature from "... <...>" is unknown trust
:: File /var/cache/pacman/pkg/fail2ban-0.11.2-2-any.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] Y
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.

I followed advice in the forums and tried refreshing and repopulating the keys, clearing the Pacman cache, and a combination of these things. I still kept getting the same problem each time I tried to upgrade.

Eventually I just removed the package, ran the upgrade, and then re-installed it:

$ sudo pacman -R fail2ban
$ sudo pacman -Syyu
$ sudo pacman -S fail2ban

That fixed the problem, but I’m still not really sure why re-fetching the keys manually didn’t help!

Twitter Learnings

Sat, 18 Sep 2021 10:04:00 +0000

I recently wrote about reviewing my Twitter usage, with the aim of discovering any constructive takeways I get from the platform that warrants me keeping it installed as an app on my phone.

The up-shot is that I have now removed it. I didn’t delete my account as there is still enough value in visiting it less regularly (such as on my computer’s web-browser), but by removing the easy shortcut from my phone I have noticeably helped reduce the amount of time I spend doomscrolling.

To reach this conclusion, I kept a sort of takeway diary, in which I recorded useful or constructive things I learned or discovered whilst using the platform over the past week. The diary was nothing special - just a few ad-hoc bullet points in a note in Bear. Some of the takeaways I’ll mention below.

Motivation - both personal and professional. Twitter is full of inspirational stories about tech, business, and life. It’s mostly a bit showboat-y but can be good for motivation, but probably not useful (or healthy?) to read about every day since it (probably) wrongly implies these people have “perfect” lives.

Tech news - most technologies I use (frameworks, tools, databases, etc.) have some sort of of Twitter presence, which can be useful for keeping up-to-date with advancements, updates, fixes, and more. This can be solved by adding these accounts to a Twitter list and checking-up on it manually on a periodic basis.

Humour - probably the most addictive and enjoyable part about Twitter is the phatic content, that exists only for entertainment. Although this is generally good for the soul, it’s the thing that keeps me scrolling.

Otherwise, general news is easy to find on Twitter (which can instead be replicated by following the right accounts on Mastodon or RSS feeds) and events coverage, such as the recent Apple event, which one can learn about by simply watching the event themselves.

To me, the above doesn’t offset the negatives enough to keep me visiting on such a frequent basis, and it was time to break the addiction. The drama, complaining and - more and more - the witch-hunting that some people turn to Twitter for just always left a sour taste.

@pswilde on Fosstodon helpfully pointed out the existence of BirdsiteLive for bridging Twitter feeds into Mastodon, which I may use to keep up-to-date with a small number of Twitter accounts. But otherwise I am glad to be ready to control my usage a little better.

Columbus Day by Craig Alanson

Sun, 12 Sep 2021 11:25:00 +0000

Having recently read Project Hail Mary - and rated it highly - Goodreads suggested I try Columbus Day by Craig Alanson.

This is the first book in the Expeditionary Force series - one that I hadn’t yet heard of at the time.

Although it’s sci-fi, the book is set in the modern-day. Earth gets invaded by far more technologically advanced aliens and humanity suddenly finds itself playing along as the lowest echelon in a war involving many different levels of alien capability. The mostly-powerless humans need to work out which side they should be fighting on.

The book is humorous and full of comments from pop culture and classic “humanisms”, which often confuse the various non-human characters encountered. The story is told from the perspective of Joe Bishop - a former US Army soldier - who inadvertently finds himself pretty much at the centre of all that goes on. His character is a little rough around the edges but the reader quickly relates and warms to him.

As far as the story goes, I found the first third or so to be a bit of a drag whilst the scene is set. However, it soon ramps up and it became very hard to put down once the main story began to kick-in.

There’s a great mix of excitement, action, comedy, drama, and tragedy. It’s a book I can definitely recommend to those that enjoy action or more light-hearted sci-fi.

The book ends on a note that paves a clear pathway to sequels and the rest of the series, and I look forward to reading them further!

Using Telegram bots to receive updates from your automated systems

Thu, 09 Sep 2021 10:17:00 +0000

The need for notifications

I self-host several services on various servers - for both some professional and personal uses.

I use automated backup scripts to periodically sync data to Backblaze (which I recently posted about). However, once they were setup I would often worry about whether they were working properly. To verify, I’d have to log into Backblaze and check when the latest backups came through.

Although I trusted the process, this became a bit of a pain and more and more of a constant worry. The script might crash, run out of storage space, or anything else, and I wouldn’t know about it unless I actually checked.

For peace of mind I wanted to be able to receive a notification each time the backups completed successfully.

Note: the same is true also for any automated system. For example, service crashes, key events, visitors, new registrations, etc. Such notifications are great for keeping in the loop without needing to actively check.

Choosing how to receive notifications

To start with, I needed to find a mechanism for receiving the notifications. I looked around for suitable methods and noticed a number of commercial offerings (many with free tiers), such as Pushover and Healthchecks. Some of these required installation of new apps or weren’t quite right for my needs.

Another option was to rely on Matrix, but although I have the Element app on my phone already I don’t use it a whole lot and I ideally just want to set something up for the long-term without needing to change it later.

So I finally settled on Telegram. It’s an app I use every day on all of my devices anyway, and it would fit nicely into my existing workflows.

Using Telegram to receive event notifications

Luckily, Telegram offers a great API around the notion of bots. I created a simple bot that would send me a short message each time the backups completed. I’ll explain the process below, which should take less five minutes to complete.

Creating the bot

Telegram bots are managed through an existing bot, called the BotFather. Begin a new chat with this bot by searching for it or by clicking this link directly.

Once you’ve started the chat, type /newbot to kickoff the bot-creation flow. BotFather will ask a few questions about your bot (e.g. a name and username), and will then create it for you. Given Telegram bots can be used by anyone, you’ll need a unique username for your bot.

After the bot has been created, BotFather will give you an API token you can use to interact with your bot (i.e. read and send messages).

Creating a chat with your new bot

The next stage is to begin a chat with your new bot. In Telegram, each chat has a unique ID, and we need to use this ID in order for your bot to send messages to the chat.

First of all, open a chat with your new bot by either following the link to it from BotFather or by searching for its username. When the chat opens, tap /start to begin the chat.

Once you have initiated the conversation, the chat will be created. The next step is to retrieve the chat ID from the Telegram API.

To do so, use cURL (or just your web browser) to make a GET request to https://api.telegram.org/bot/getUpdates, replacing with the token provided earlier by BotFather.

Some JSON will be displayed, describing the chats your bot is involved in. Given that no one knows about your bot yet, the results should contain just one entry, and will display your own username and details. Copy or make a note of the chat ID for your chat, as we’ll need that next.

Sending notifications

The final part is now to use these details to send notifications from your bot. This is done by making a request to the following URL: https://api.telegram.org/bot/sendMessage?chat_id=&text= (again, replacing the bits in < > with your own values).

Your chat with the bot should receive the message and you’ll get a notification.

You can now add a simple curl command to the end of your important scripts, or embed a call to the URL in your applications to get updates about key events.

Customising your bot

You can also customise your bot with an avatar image and other details. To do so, head back over to your chat with BotFather and use the /mybots command to get started.

Conclusion

In this post I’ve briefly covered how to create a simple bot for providing event notifications.

I hope you find it useful!

Twitter is still too addictive

Tue, 07 Sep 2021 14:25:00 +0000

TL,DR; I’m starting a Twitter diary to log interesting findings, and to measure its value to me.

Twitter is pretty much the last bastion of mainstream centralised social media that I use (aside from messaging services like Whatsapp and Telegram).

Although I primarily use Mastodon for my every-day social networking, which is more focused on the things I am actually interested in, I always kept Twitter around too as an app on my phone. This is because every time I try to remove it, I quickly feel as though I must be missing out on something. It always felt as though some news or useful announcement would go unnoticed.

However, over the past few weeks and months I’ve found myself once again often getting into endless and mindless Twitter scrolling sessions. I feel as though I rarely come out of these with any positive gain, and then get frustrated with myself for wasting time.

One might ask what is is that I get out of Mastodon, and why this is time more usefully spent than on Twitter. This equally then may lead to thoughts along the lines of “you’re just not following the right people”. This is a fair comment, but curating a following list should be an interesting and pleasurable experience. This feels to be no longer the case on Twitter.

Beyond this, there’s also the notion of the algorithmic timeline (tweets you may have missed, or which Twitter thinks you should see again), as well as the pseudo timelines you’re force-fed around the “likes” of those you follow.

This is seemingly at random, but I imagine it is massively under the control of “the Twitter algorithm” to the extent where what was once an innocent and useful bookmarking feature now becomes a game of retweet roulette.

More and more I even see cases of tweets injected into my timeline from users that are followed by the people I follow - this is two hops through the social graph but without explicit retweets or likes from those that I directly follow. Given Twitter’s own 3.4 degrees of separation (and that paper is from 10 years ago already), what is the point of choosing who to follow?

On top of these “features” are the seemingly-newer “topics you may like” feature and, of course, the adverts.

The image below shows my Twitter home feed containing three tweets (the first and last cannot be fully shown due to the size of my phone screen). None of these tweets are from people I follow; the first is an ad, the second is advertising a “topic”, and the last is a “like” from someone I do follow.

All of this boils down to the fact that choosing who you follow becomes less and less impactful on the information you actually receive on Twitter. The only reason for this is that it makes the platform more addictive (as I’ve experienced) so that you spend more time on it so that Twitter makes more money from you viewing the ads. Although there are other apps available that try to bypass some of these “features”, it doesn’t feel right to give the service more of my time without getting anything useful out of it.

Anyway, /rant over. What am I going to do about it?

It’s time to qualify what Twitter actually does offer. I’ve started a sort of Twitter “diary”, in which I plan to log takeaways from my time on the platform. If I find myself on Twitter (as I often subconciously do, even having removed it from my homescreen), I will note useful or meaningful learnings or findings in the diary. I’ll also include humorous and phatic content, since a laugh always feels constructive (to me, anyway).

After a few weeks, I’ll assess the diary and measure any learnings against whether I could have learned the same from other sources (e.g. my RSS feeds or Mastodon).

SSO Tools

Thu, 02 Sep 2021 19:58:00 +0000

Another project I try to maintain (when I can!) is SSO Tools.

This is a simple web service that aims to help developers test their own services’ single sign-on (SSO) functionality. The motivation behind the project was that many commmercial offerings were too expensive for solo developers, or just far too complex for simple testing.

SSO Tools aims to provide a simple interface, with functionality that allows for registering identity providers (IdPs), test IdP users, and service providers (SPs). It is targeted at developers looking to quickly, yet robustly, test and iterate on their SSO setup in their applications.

The IdP users can be configured with custom attributes to allow for observing how a service provider application behaves when receiving different attributes.

Currently SSO Tools allows for testing SAML2 configurations, and supports both SP-initiated and IdP-initiated login for SSO. It also supports single-logout functionality, in which the user is automatically logged-out of the IdP when the user logs out of a connected service.

SSO Tools is offered as a free service. I needed something myself when building out a SAML2 service provider, and thought the features could be useful for others too.

If you’d like to help or get involved in the project then I’d love to hear from you.

Darknet Diaries: True stories from the dark side of the Internet

Sun, 29 Aug 2021 12:11:00 +0000

Another of my favourite podcasts is Darknet Diaries.

Created and presented by the excellent Jack Rhysider, Darknet Diaries releases new episodes fornightly. Each episode contains a true story from the “dark side of the internet” and includes content related to cybercrime, hacking (in the information security sense), dodgy government activity, and much more.

Typically, most episodes involve guests that Jack interviews in order to tell a story. Although it does occasionally stray into some technical detail, the vast majority of episodes are totally accessible to everyone.

If you have an interest in tech and love stories about crime or beneath-the-radar operations, then this podcast is perfect for you. I always look forward to new episodes.

Whilst some stories may be shocking (or even occasionally a little disturbing), they are all certainly exciting and super interesting. I’ve learned a lot and I can also definitely recommend listening if you’re involved in security or operations in your own work.

If you’re interested in giving it a go, you can subscribe to new episodes. If you end up enjoying it, then you can also donate to the show.

I hope you enjoy!

Stripping sensitive EXIF data from uploaded images

Sat, 28 Aug 2021 16:46:00 +0000

The problem with image uploads

Many services - including web and mobile apps - allow for their users to upload imagery. This could be to enable users to upload an avatar image or perhaps create a gallery of image files.

Either way, many photos contain some degree of sensitive metadata information as part of their EXIF data. For example, if you take photos using your phone, it is likely that the camera application will embed metadata into the image file it creates. This could include the geocoordinates of the position from where the photo was taken, the make and model of the camera device, as well as lots of other data (exposure time, focus, balances, etc).

This data is useful, since it allows photo-viewing applications to display your images on a map or to allow for examining files in more detail.

However, if a user chooses to upload such an image file to a service, unless the metadata is correctly stripped out the exact same file can be downloaded by other users - which could give them access to sensitive information. For example, if someone created an image post along with some text describing themselves as being at home, other users could quickly (and accurately) discover where the user lives.

In this post I will cover:

A brief introduction to the Pillow image processing library
How to use Python to process images in order to remove EXIF data
How the approach can be automated when using S3 and AWS Lambda

A basic approach to removing sensitive information

Luckily, removing sensitive EXIF data is not too difficult to do. This post leverages Python for demonstrating the approach, however the process would be very similar in other languages too. As such, the discussed method could easily be included as part of a Flask or Django service, or a standalone Python application.

Specifically, I will make use of the Pillow library for Python - it is a very useful tool for manipulating images.

This post does not cover the process or code required to actually upload a file from a client, since the following approach is agnostic as to whether the service is part of a mobile app, a web app, a simple API, a local application, or anything else.

Another thing to remember is that orientation data (i.e. the positioning of the camera when the photo was taken) is also included in the EXIF data. This information tells image viewers with which rotation to display the image, and so we will need to handle this too.

The following provides a basic approach to accomplishing what we are trying to achieve.

First, install the needed dependency:

poetry add pillow

Or use pip if preferred.

You’re then ready to write your code, assuming the Python application has a reference to the image data (e.g. by reading from a request stream, a file, or elsewhere) called image_data.

# At the top of the file, include PIL in your imports:
from PIL import Image, ExifTags
import io
...

# Calculate orientation key in ExifTags
orientation = None
for orientation in ExifTags.TAGS.keys():
  if ExifTags.TAGS[orientation] == 'Orientation':
    break

# Create a new PIL image from the image_data:
image = Image.open(image_data)

# We need to use the orientation EXIF info to rotate the image:
image_exif = image.getexif()
if image_exif:
  exif = dict(image_exif.items())
  if exif.get(orientation) == 3:
    image = image.rotate(180, expand=True)
  elif exif.get(orientation) == 6:
    image = image.rotate(270, expand=True)
  elif exif.get(orientation) == 8:
    image = image.rotate(90, expand=True)

# Retrieve the image contents as a list representing a sequence of pixel values:
image_contents = list(image.getdata())

# Create a new image based on the original, but without the full EXIF data:
new_image = Image.new(image.mode, image.size)
new_image.putdata(image_contents)

# Finally, create a new buffer object and put the new image file data into it:
new_image_data = io.BytesIO()
new_image.save(new_image_data, 'PNG') # Or the file format needed
new_image_data.seek(0)

And that’s it. The new_image_data object can now be sent to wherever you need it to be - the filesystem, another server somewhere, or to a storage solution like S3. For example, you may want to give the new image the same filename such that it overwrites the original.

During this process you can also leverage Pillow to do some additional image processing, such as resizing, cropping, colour transformations, or even statistical analyses. I recommend checking out the handbook for more information on this.

Working with Amazon S3

If you use S3 as a storage backend, you can easily add an S3 trigger - along with a quick Lambda function - to automatically handle this process for you without the need for additional servers or changes to your existing application.

This process will cause S3 to automatically trigger your Lambda function each time a new file is uploaded to your bucket.

To start with, bundle the above code into a function called handler that accepts an event object as an argument. You’ll also need to import the boto3 library:

# Include the previous imports from above...

# Also import boto3 and os:
import boto3, os

# Create a new function
def handler(event):
  # Create a new S3 client:
  s3 = boto3.client('s3')

  # Get the pathname to the uploaded object from the event:
  file_path = event['Records'][0]['s3']['object']['key']

  # Extract the file name and extension
  file_name, file_extension = os.path.splitext(file_path)

  # Check the file is an image (and not another type of file, like a PDF):
  if file_extension.lower() not in ['.jpg', '.png', '.jpeg']:
    # If not an image, bail out early:
    return

  # Retrieve the object's tags. Here we can check if the file
  # has already been processed in order to prevent an infinite loop:
  tagging = s3.get_object_tagging(Bucket='mybucket', Key=file_path)
  object_tags = tagging['TagSet']

  # Go through the tags and exit if EXIF data has already been stripped:
  for tag in object_tags:
    if tag['Key'] == 'ExifStripped' and tag['Value'] == 'True':
      return

  # Retrieve the image from S3 into the image_data object:
  image_data = s3.get_object(Bucket='mybucket', Key=file_path)

  # Now process the file by including the earlier code
  ...
  ...

  # Finally overwrite the existing S3 object with the new image file data and tag:
  s3.put_object(Bucket='mybucket', Key=file_path, Body=new_image_data, Tagging='ExifStripped=True')

Replace mybucket with the name of your existing S3 bucket. Depending on your setup you may need to add additional configuration to the S3 client (such as the region of your bucket).

We use S3 object tagging to tag files we’ve already processed. Otherwise the final put_object call would result in S3 recursively calling the function again and again.

Next, deploy the above function to AWS Lambda - either through the command line or by copying the code into the AWS web interface. You’ll need to name your function appropriately (e.g. image_handler) and remember to register the handler function as the Lambda handler.

You should assign a suitable IAM role that allows the function to read and write from and to your S3 bucket, and also to read and write tags. I won’t cover this here, since if you already use S3 I’ll assume you’re also familiar with IAM.

To create the trigger in the S3 console, navigate to your bucket’s “Properties” tab and click the “Create event notification” button. Go through the form and configure the trigger based on your needs, including your new Lambda function in the “Destination” section.

Configure the basic details as above, replacing the information as needed.

Finally, choose your Lambda function from the list as a destination, as shown above.

Conclusion

In this post I have talked about one method for handling uploaded images in order to remove sensitive EXIF data and also explained how the same process can be used to incorporate other types of image processing.

Whilst the approach won’t be suitable for everyone or for all use cases, it should provide additional means for protecting the security of users of your services and applications.

Postscript

Update (February 2023): Thanks to Ercan for pointing out an omission in this blog post relating to calculating the orientation key.

Adding 'dark mode' and dynamic theming to your React websites

Sat, 21 Aug 2021 13:05:00 +0000

Adding theming and the choice between “light” and “dark” modes to your website can enhance your site’s accessibility and make it feel more consistent with the host operating system’s own theme.

With JavaScript (and React in particular) this is easy to do, as I’ll explain in this post. We’ll use a combination of our own JavaScript code, the Zustand state management library, and the styled components package to achieve the following:

Create a theme library
Dynamically update the site’s styles based on the theme
Allow the user to change the theme
Remember the theme for the next time the user visits
Detect the operating system dark/light mode to auto-set an appropriate theme

This post is written from the perspective of a GatsbyJS website, however the same concepts should apply to any React single-page application.

TL;DR: If you just want the code, scroll to the ‘Complete code’ section at the bottom of the post.

Setting up the theming

To start, add these new packages to the project:

yarn add zustand styled-components

Next, declare some themes! For this you can create a themes.js file (e.g. in your top level src/ directory):

const themes = {
  light: {
    name: '☀️ Light Theme',
    background: '#FFFFFF',
    text: '#000000',
    headers: '#000000'
    links: '#01BAEF',
  },
  dark: {
    name: '🌒 Dark Theme',
    background: '#0F0E17',
    text: '#A7A9BE',
    headers: '#FFFFFE',
    links: '#FF8906',
  },
};
export default themes;

You can add as many attributes to your themes as you like in order to increase their flexibility. I use Happy Hues for inspiration. You can also create as many different themes as you like!

Now we need a way to manage the themes and to maintain the currently-selected theme. This is where the Zustand library will come in useful for managing the app’s global theme state.

For this, create a store.js file (e.g. again, this could be in your site’s top-level src/ directory) with these contents:

import create from 'zustand';

export default create(set => ({
  theme: 'light',
  setTheme: theme => set({ theme }),
}));

This tells Zustand to create a new store with two attributes: a reference to the current theme key (defaulting to light) and a function for changing the theme.

Next, use the styled-components library to create some dynamic components for use within the app that change their style based on the currently selected theme.

The package’s createGlobalStyle function is great for injecting styles that affect the entire app. Import the function at the top of your primary layout file:

import { createGlobalStyle } from 'styled-components';

And then use it to create a global style component for your website based on the currently-selected theme (in the same top-level layout file):

const GlobalStyle = createGlobalStyle`
  body{
    background-color: ${({ theme }) => theme.background};
    color: ${({ theme }) => theme.text};
    transition: background-color 0.25s, color 0.25s;
  }
  h1,h2,h3,h4 {
    color: ${({ theme }) => theme.headers};
  }
  a {
    color: ${({ theme }) => theme.links};
  }
`;

We also added a transition to fade nicely between styles when the user later switches theme.

Finally, add some code (again, to the same layout file) to import the current theme from the store and pass this to the global style component we created.

// First import the store you created earlier near the top of the file:
import useStore from '../../store';

// Import the themes from the theme file you created earlier:
import themes from '../../themes';

...

// In your main layout component load the theme from the store:
const LayoutComponent = ({ children }) => {
  const theme = useStore(s => s.theme);

  ...
  
  // Then render the global styles and the layout's children:
  return (
    <div>
      <GlobalStyle theme={themes[theme]} />
      <div>{children}</div>
    </div>
  );
}

Your website is now using the theme to set the colours you defined in your theme object. If you change the default theme (in the store.js file) to dark and refresh the page, you should see the styles for the dark theme rendered.

Changing the theme

We now need to provide a way for the user to change the website’s theme. Typically this might be done in a dedicated component elsewhere in your app, which is where using the Zustand store comes in handy.

Currently, on this website, I have a change theme select box in the site’s Header component (as shown in the image below):

This can be achieved through something like the following:

// Import the styled-components library:
import styled from 'styled-components';

// Import the store we created earlier:
import useStore from '../store';

// Import the theme list we created earlier:
import themes from '../themes';

// Use styled-components to create a nice select box:
const StyledThemeSelector = styled.select`
  padding: 4px;
  background: white;
  border: 1px solid gray;
  border-radius: 5px;
  cursor: pointer;
`;

// In the component load the store:
const Header = () => {
  const store = useStore();

  // When rendering the component, include your nicely styled theme selector:
  return ( <>
    ...
    <StyledThemeSelector value={store.theme} onChange={e => store.setTheme(e.target.value)}>
      {Object.keys(themes).map(themeName =>
        <option key={themeName} value={themeName}>{themes[themeName].name}</option>
      )}
    </StyledThemeSelector>
    ...
  </> );
}

Now, when you switch theme using the select box, the entire app should change to reflect the new styles. Also, because you added the transition to the styles earlier, the new colours should fade in nicely rather than suddenly changing.

Of course, the theme-selctor can be any type of component - a button, a list of radio buttons, or anything else. If you have lots of themes you could even create a more visual theme gallery.

Remembering the selected theme

You can configure your website to remember the user’s theme choice by leveraging localStorage.

To do so, first update your theme-selection component (as above) to implement a function which updates the site’s local storage each time the user changes the theme:

...
const Header = () => {
  const store = useStore();

  // Add this function:
  const switchTheme = themeName => {
    store.setTheme(themeName);
    localStorage.setItem('theme', themeName);
  };

  // And update the onChange for your theme selector:
  return ( <>
    ...
    <StyledThemeSelector value={store.theme} onChange={e => switchTheme(e.target.value)}>
      ...
    </StyledThemeSelector>
    ...
  </> );
}

Then, back in your top-level layout file, add a useEffect hook to update the theme on app-load:

// Add the useEffect import:
import React, { useEffect } from 'react';
...

const LayoutComponent = ({ children }) => {
  const setTheme = useStore(s => s.setTheme);
  ...

  useEffect(() => {
    // Load and set the stored theme
    const rememberedTheme = localStorage.getItem('theme');
    if (rememberedTheme && themes[rememberedTheme]) {
      setTheme(rememberedTheme);
    } 
  }, [setTheme]);
   ...
}

Now the website will load the user’s preferred theme each time they come back to your site.

Automatically matching the operating system theme

Many operating systems allow the user to choose between a light and dark theme, or even to dynamically change the theme based on the time of day.

It can be seen as good practice for websites to display a theme which more closely matches the background OS style to provide a more consistent user experience.

Luckily, this is very easy to do. Just update the useEffect block described above to check for the OS ‘dark mode’ as the app loads.

...
  useEffect(() => {
    const rememberedTheme = localStorage.getItem('theme');
    if (rememberedTheme && themes[rememberedTheme]) {
      setTheme(rememberedTheme);
    } else {
      const isDarkMode = window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches;
      if (isDarkMode) setTheme('dark');
    }
  }, [setTheme]);
...

The nice thing about this approach is that if the user has already chosen another theme preference, this will be used instead of the default.

Complete code

I hope this post helps you to create your own theme libraries and nice style-switchers. As mentioned earlier, you could create as many themes as you like and make them as complex as necessary.

You can also import the store into any component if you want more fine-grained control (e.g. themed message boxes), and even include fonts and other types of styles.

For the complete code discussed in this post, see below.

/src/themes.js

const themes = {
  light: {
    name: '☀️ Light Theme',
    background: '#FFFFFF',
    text: '#000000',
    headers: '#000000'
    links: '#01BAEF',
  },
  dark: {
    name: '🌒 Dark Theme',
    background: '#0F0E17',
    text: '#A7A9BE',
    headers: '#FFFFFE',
    links: '#FF8906',
  },
};
export default themes;

/src/store.js

import create from 'zustand';

export default create(set => ({
  theme: 'light',
  setTheme: theme => set({ theme }),
}));

Top-level layout/app file

import React, { useEffect } from 'react';
import { createGlobalStyle } from 'styled-components';
import useStore from '../../store';
import themes from '../../themes';

const GlobalStyle = createGlobalStyle`
  body{
    background-color: ${({ theme }) => theme.background};
    color: ${({ theme }) => theme.text};
    transition: background-color 0.25s, color 0.25s;
  }
  h1,h2,h3,h4 {
    color: ${({ theme }) => theme.headers};
  }
  a {
    color: ${({ theme }) => theme.links};
  }
`;

const LayoutComponent = ({ children }) => {
  const theme = useStore(s => s.theme);
  const setTheme = useStore(s => s.setTheme);
  ...
  useEffect(() => {
    const rememberedTheme = localStorage.getItem('theme');
    if (rememberedTheme && themes[rememberedTheme]) {
      setTheme(rememberedTheme);
    } else {
      const isDarkMode = window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches;
      if (isDarkMode) setTheme('dark');
    }
  }, [setTheme]);
  ...
  return (
    <div>
      <GlobalStyle theme={themes[theme]} />
      <div>{children}</div>
    </div>
  );
}
export default LayoutComponent;

Another component (or wherever you want your theme-switcher to be)

import React from 'react';
import styled from 'styled-components';
import useStore from '../store';
import themes from '../themes';

const StyledThemeSelector = styled.select`
  padding: 4px;
  background: white;
  border: 1px solid gray;
  border-radius: 5px;
  cursor: pointer;
`;

const Header = () => {
  const store = useStore();

  const switchTheme = themeName => {
    store.setTheme(themeName);
    localStorage.setItem('theme', themeName);
  };

  return ( <>
    ...
    <StyledThemeSelector value={store.theme} onChange={e => switchTheme(e.target.value)}>
      {Object.keys(themes).map(themeName =>
        <option key={themeName} value={themeName}>{themes[themeName].name}</option>
      )}
    </StyledThemeSelector>
    ...
  </> );
}
export default Header;

Starting out with BookWyrm

Fri, 20 Aug 2021 16:18:00 +0000

For several years I’ve been a user of Goodreads. It’s a very popular platform, and I primarily use it for keeping track of the books I’ve read, for receiving suggestions about new books, and for keeping up with what some of my friends are reading.

It’s a good service (though sometimes a little slow) - the website and mobile app are nice to use. However, as with any closed system, it’s always a worry of mine to think about what might happen if the service were to disappear or if I were to get locked out for some reason.

More recently I’ve been watching the progress and development of BookWyrm - a decentralised and federated (through ActivityPub) social reading platform. As such it can co-operate with other “fediverse” services, such as Mastodon. The platform aims to allow users to record their reading activity, reviews, and follow others.

The Join BookWyrm website advertises some known instances on the web. Although I understand the underlying technology project powering these instances is still in its relatively early stages, I was excited to try one out for myself.

I signed-up to the bookwyrm.social instance and set up my profile.

In the short time I’ve been using it, I’ve found the interface to be clear and focused (without all the extra bloat Goodreads comes with). I was even able to import my Goodreads data (after exporting it first as a CSV). The website scales nicely so that it’s also highly usable on my phone’s browser.

It feels safer to be using open-source software for managing another piece of my online data footprint, and it’s great to become more involved in the decentralised movement and community.

I’ve already discovered some ineteresting “to-read” books. I look forward to exploring more over the coming months, and to continue seeing how the project develops.

Fixing up our outside space

Sat, 14 Aug 2021 15:02:00 +0000

About 18 months ago we bought a new home. The house is an 1880s (ish) Victorian building, and many of its original features - such as tile floors, cornice, and fireplaces - had been retained, which is great.

It had previously been a student-style (HMO - house of multiple occupation) house, in which nearly all available space had been converted into bedrooms. As you might imagine, the fixtures and fittings weren’t in great conditions, with old and worn carpets and wallpaper throughout.

Our plan was to try and restore the house back into a more family-style home. We knew this wouldn’t be a small undertaking (every room needed work to some degree). And this included the outside space.

Our garden (or yard) isn’t massive (typical for a house in the city), measuring about 5x5 metres along with a side-return. In the decades past, the garden would have been used for access to the house for staff (cooks and cleaners, etc.) and for deliveries. As such, it featured raised beds and a mix of pathways throughout.

When we first moved in, there was no flat or actual usable space, the beds had become overgrown, and it was full of junk left by the previous occupiers. Our plan was to flatten it into a practical, urban and low-maintenance space. We’d floor it with slabs and use nice pot plants to bring in some greenery.

The rest of this post shows our progress!

What we had to work with: the day we moved in

When we first moved in, the garden was full of overgrown and dead plants, and lots of junk.

Flattening the garden

We cleared lots of the junk and were able to flatten the beds. We used sledgehammers to knock out the low walls and raked the rubble and soil to level most of the area.

Slabs foundation

We began to lay some gravel as an eventual foundation for the slabs.

Restoring the walls

The walls looked grotty, as they were covered in peeling paint and plants. We cleared the plants and took a multi-tool to the walls to scrape off the old paint.

Fencing!

We knocked down (with our neighbour’s permission) the dividing fence between our gardens, and rebuilt a newer and sturdier one over a very rainy weekend.

You can also see the more restored walls in the background and a new back gate that we built.

We also fenced in front of the wall on the other side.

Completing the floor

We laid the remainder of the slabs - almost two tons’ worth that we needed to carry through the house from the street. We pointed them and cleaned them up.

You can see the other completed fence in the background.

Finishing touches

And that’s it, really. We put some new pots in, added some planks to create a sort of pergola, and some solar lights for the evening.

We also cladded the little out-building (a utility room) with wood and repainted some of the brickwork.

Treadl

Fri, 13 Aug 2021 17:13:00 +0000

I don’t tend to talk much about the projects I’m working on, but thought this would be a good opportunity to write a post about one such project - Treadl.

Treadl is a web app (and more recently and less popularly a mobile app too). It enables weavers to create and store their weaving patterns and projects online. This could be simply for personal use, or for sharing projects with others as a portfolio.

The platform also allows for groups to be created, in which weavers can come together to discuss and share ideas (e.g. as part of a class or weaving group). Weavers can set up their individual profiles in order to tell others a little more about themselves. Users can also comment on other people’s projects and creations.

Treadl is compatible with the Weaving Information File (WIF) file format standard. This means that patterns created in other weaving software (such as Fiberworks) can be imported into Treadl. Likewise, Treadl also allows for exporting patterns in WIF format to be imported into other compatible software.

I myself am not a weaver. However, my mother is, and having a space where she and her different groups of weaving colleagues can design and store their patterns and drafts - along with engaging with each other socially - is useful. She initially had the idea for the platform and most of the feature ideas came from her!

Treadl is free to use as a means of giving back to the community. I had initially planned to create some paid-for features to help cover some of the ongoing hosting costs, but I have parked those ideas for now.

The project is written in React on the front-end, and Python on the back-end. The mobile app (which is mainly useful for the groups functionality) is written in Flutter.

Pinephone update: the first few weeks

Mon, 09 Aug 2021 17:51:00 +0000

Back in April, I bought a Pinephone. I used the phone quite consistently for the first few weeks and I meant to write an update here a couple of months back, but work (and other things) got in the way a bit.

So, here is my delayed “first few weeks with a Pinephone” update.

As mentioned, I initially aimed to simply use the phone in its out-of-the-box state (i.e. Manjaro Linux with KDE Plasma Mobile) - not as a daily driver, but more as a means of measuring the phone’s base case usability. However, hopefully with an aim to eventually being able to use such a device more full-time.

Overall impression

Generally, I’ve really enjoyed using the Pinephone. I mean, it is slow (compared to other modern phones), apps take a good few seconds to open, and the UI freezes every now and again. However, things are moving quickly - there are other OS builds and desktop environments available to try.

It is encouraging to see successes from the base shipped version of the device. For example, below is a picture of my dog that I took today using the Pinephone.

What did I try?

In my previous post, I outlined a few areas and day-to-day tasks I wanted to try and accomplish. I’ll evaluate against each of them (so far!) below.

Basic calls and texts: ☑️ worked out of the box. The Pinephone accepted my SIM card and connected without issue.
4G cellular connectivity: ☑️ worked out of the box. Drops out every now and again, requiring a reconnect.
WiFi connectivity: ☑️ worked out of the box, automatically connects.
Bluetooth connectivity (including headphones): ☑️ worked without issue, was able to connect and play through my Airpods Pro. Airpods-initiated pause/play also worked!
Photo- and video-taking using both front- and rear-facing cameras: ☑️ ⚠️ taking photos works fine using the Megapixels app (though does require an app restart for the camera to activate properly), videos not yet present.
Web browsing: ☑️ works great, using the shipped Angelfish browser.
Podcast subscribing, listing, and listening: ☑️ Works great using the Podcasts app. Could subscribe to all my podcasts. Only thing I couldn’t get working is changing the playback speed (though this I think is just a UI bug as the setting is there).
Audiobook downloading and listening: ☑️ I use Audible still as I am not aware of a good DRM-free audiobook retailer, however there are tools to convert the Audible AAX files you own to MP3 for DRM-free playback. AAXtoMP3 looks like it does the trick.
Music-playing (preferably through Spotify): ☑️ Works fine using the Spot app. Can access playlists and play music (ironically the Spot app is more responsive than the Spotify app on my iPhone!).
Mastodon (tooting and reading my timelines): ⚠️ Tootle looks great, but I have been unable to fully use it. (I can’t for the life of me get copy/paste to work properly, and so I cannot copy the authorization code to login). But it works via the web.
RSS (viewing my feeds from my FreshRSS server): ☑️ Working fine using the NewsFlash app (connected via the Fever API to my existing FreshRSS server).
Email reading and sending: ☑️ worked OK using Geary. Unable to login to my Google Mail accounts, but others worked.
Telegram messaging: ☑️ Worked with no problems.
Password-management: ⚠️ Could not get working: the available Bitwarden app does not scale small enough to enable me to login. But promising it exists in the store.

Above: a screenshot from the Podcasts app listing some of my subscriptions.

Moving forward

Clearly the Pinephone is capable of fulfilling most of my every-day mobile device needs - right out of the box. As I say, the device is a little slow and crash-prone (currently) to use fully as a daily driver. Additionally, the issue I’m having with Bitwarden and copy/paste makes accessing some services tricky.

However, this certainly feels like it’s just the start. I am excited to give Phosh a go and to get involved further in the Flathub ecosystem to see if I can improve on the functionality and usability. I’ll update again soon!

My development stack

Wed, 04 Aug 2021 18:56:00 +0000

Some people have complex development processes and flows - making use of tools such as heavy editors and IDEs, Docker for running and building locally in development, or even develop entirely remotely over SSH connections. Other people use simpler combinations of tools.

I thought I’d write briefly about what I use on a daily basis. I have a relatively simple development tech stack:

Terminal application (I use the Terminal.app application that ships with my Mac, since this works best for me)
tmux - for handling multiple windows and panes
tmuxinator - for managing complex tmux sessions (I recently wrote more about this)
vim - a simple yet powerful text editor
git - for source control

I also use a small number of Vim plugins - installed via Vundle - to add nice quality-of-life features to my editor:

pangloss/vim-javascript - better JavaScript syntax highlighting
mxw/vim-jsx - JSX syntax highlighting
dart-lang/dart-vim-plugin - Dart language syntax highlighting
rust-lang/rust.vim - Rust language syntax highlighting
morhetz/gruvbox - Attractive Vim colour scheme
airblade/vim-gitgutter - In-editor Git indicators
ctrlpvim/ctrlp.vim - Awesome and quick file searching
scrooloose/nerdtree - Easy file/directory tree navigation

I backup and sync my dotfiles (e.g. my .vimrc and .tmux.conf) by symlinking from my Nextcloud sync directory.

And that’s it, really. In terms of per-project management, I make use of native tooling depending on the languages and frameworks being used. For example, for Node/JavaScript projects I use yarn and for Python projects I use virtualenvs and Poetry for handling dependencies.

It’d be nice to have an elegant cross-platform approach to provisioning new machines with this setup, but given its relative simplicity it isn’t too much of a headache to get things back up and running again when I switch machines!

Our first few months with a dog

Tue, 03 Aug 2021 18:59:00 +0000

About nine months ago - at the end of November last year - we adopted a dog. Although I’ve always grown up with and around dogs owned by parents and siblings, I’ve never been a huge “dog person” myself. However, it is very easy to get attached very quickly!

The dog we adopted is an English working cocker spaniel. As you might imagine, he is extremely energetic. He’s a nightmare on (and sometimes off) his lead, and his recall isn’t fantastic yet due to us needing to re-train him after his previous owner.

However, he is lovely to have around the house. He’s great company for me when working from home and still always follows me wherever I go. He sprints around the park for hours on end with his dog friends but is super calm and relaxed when at home, which is great. He’s always extra cuddly in the evenings, but hates having to go to his own bed when it’s time to go to sleep.

He gets me out the house even on rainy days, and I’ve met and made friends with lots of other dog owners in the parks near where we live.

It’s only been a few months, but he is definitely already a big part of our family. It’s hard to now imagine life without him!

Code syntax highlighting in Gatsby

Wed, 28 Jul 2021 17:59:00 +0000

Providing code snippets on your website or blog can be a great way to convey meaning for technical concepts.

Using the HTML pre tag can help provide structure to your listings, in terms of spacing and indentation, but highlighting keywords - as most people do in their code text editors - also vastly helps improve readability.

For example, consider the below JavaScript snippet.

class Greeter {
  greet(name) {
    console.log(`Hello, ${name}!`);
  }
}

const greeter = new Greeter();
greeter.greet('Will');

The exact same listing is displayed below with some simple syntax highlighting. The structure and meaning of the code becomes much easier to understand.

class Greeter {
  greet(name) {
    console.log(`Hello, ${name}!`);
  }
}

const greeter = new Greeter();
greeter.greet('Will');

If you build your website with GatsbyJS and write your blog posts or pages in markdown format, then adding syntax highlighting is super easy.

PrismJS is a well-established library for achieving syntax highlighting on any website. Luckily, there is also a Gatsby plugin for Prism which makes it much easier to get started in Gatsby websites.

First of all, add the required dependencies to your project:

yarn add gatsby-transformer-remark gatsby-remark-prismjs prismjs

Next, add an entry to the plugins array in your gatsby-config.js file to use the plugin. For example, the below is what I use.

  ...
  plugins: [
    ...
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: 'gatsby-remark-prismjs',
            options: {
              classPrefix: 'language-',
              showLineNumbers: true,
              noInlineHighlight: false,
            }
          }
        ]
      },
    },
  ],
  ...

Finally, you’ll need to choose a theme. There are a number of official ones available from Prism. To enable one, simply add an entry to your gatsby-browser.js file. For example:

require('prismjs/themes/prism-solarizedlight.css');

That’s it - now in your markdown you can add a block of code to syntax-highlight as follows:

You can change the javascript part to a supported language you are using, and Prism will syntax-highlight accordingly.

Extra configuration

If you want to add extra features - such as line numbers - or if you run into issues, I recommend checking out the documentation for the plugin.

Capsule.Town

Tue, 27 Jul 2021 17:55:00 +0000

The Gemini protocol has gathered even more momentum in the few months since I last posted about it.

Its popularity is largely driven by its privacy-focused and content-oriented design. It doesn’t allow for bloated sites or resource-hungry client-side scripting. It’s a means for simply and easily accessing content that is useful to you - either by hosting a capsule yourself or by joining an existing community.

In this post I am introducing Capsule.Town - a way in which I can try and give back to the FOSS community.

Capsule.Town (accessible using your Gem client by visiting gemini://capsule.town or through your web browser via a web proxy service) is a means for easily and safely hosting your Gemini content.

It’s a free service for publishing your own Gemini capsule. Currently this gives you a subdomain at the capsule.town domain, but I hope to later add support for bringing your own domain. No registration is required, meaning you can get started straight away without needing an email address or username.

All of the code is open-source and distributed under the BSD 2-clause license. This is aside from the Gem server itself, which is built on top of the Agate project.

The recommended approach is to download the binary for your system, and use this to publish your capsule. To get started, visit gemini://capsule.town/start.gmi (or the equivalent via the web) and follow the instructions.

Whilst other free services exist that also allow you to host Gemini pages, the aim with Capsule.Town is to create a Netlify-like process for quick and easy publication of content.

Feel free to give it a try. If you have any questions or problems, just get in touch.

Accidental Tech Podcast

Mon, 26 Jul 2021 19:27:00 +0000

I listen to a number of podcasts each week. One of these is ATP (Accidental Tech Podcast).

This is one of my favourite weekly podcasts. It’s humurous and full of cutting-edge discussion from the tech world, and I always look forward to new episodes.

The episodes are primarily Apple focused, which is fine for me since I’m a big user of Apple products. Some episodes are more technical than others - discussing programming and development approaches - whilst others are focused more on user-facing items.

Episodes also include lots of informal chit-chat about general tech (electric vehicles, drones, networks, and more). It’s easy listening, and perfect for when out walking or exercising.

I can definitely recommend trying it out. It should come up in your podcast app if you search for “ATP”. I use Overcast on iOS (which was created by one of the ATP hosts), and if you do also you can subscribe at this link.

There is also an option to become an ATP member, which gives you unedited and ad-free versions of the show (along with other benefits).

Generating video previews in webapps

Mon, 19 Jul 2021 16:04:00 +0000

Many web apps have support for uploading video files. Whether it’s a media-focused platform (such as a video sharing service) or just offering users a chance to add vlogs to their profile - videos are a powerful mechanism for distributing ideas.

For services providing image upload functionality, it is relatively simple to build in processes that extract smaller versions of the files (e.g. thumbnails) to be used as image previews. This allows other users to see roughly what an image is about before opening a larger version. It also enables more interesting, responsive, and attractive interfaces - since the smaller images can be loaded more quickly.

For videos, however, the process is less obvious. Particularly when using the native video tag, on some browsers often just an empty black rectangle is displayed before the video begins playing. This can result in uglier interfaces in which users cannot get a preview before the video plays.

However, there is a way in which video preview thumbnails can be extracted from your videos. These can then be stored alongside your video and displayed in-place of the video on a webpage to provide context about the video before it plays.

This process can be completed as part of the video selection process on the user’s device, so no extra server-side processing is required. It essentially involves “playing” the video in the background in a hidden player for a second or two, and then extracting the current frame as an image to be stored.

The code below should help explain the process in more detail.

const getVideoPreview = (file, time) => {
    return new Promise((resolve, reject) => {
        // create a video player
        const player = document.createElement('video');

        // when the metadata loads…
        player.addEventListener('loadedmetadata', () => {
            // when the video has seeked…
            player.addEventListener('seeked', () => {
                // create canvas and draw current frame
                const canvas = document.createElement('canvas');
                canvas.width = player.videoWidth;
                canvas.height = player.videoHeight;
                const ctx = canvas.getContext('2d');
                ctx.drawImage(player, 0, 0, canvas.width, canvas.height);

                // resolve with image file
                ctx.canvas.toBlob(resolve, 'image/jpeg', 0.75);
            });
            player.currentTime = time;
        });
        player.setAttribute('src', URL.createObjectURL(file));
        player.load();
    });
  }

Elsewhere in your code, you could now call const videoPreview = await getVideoPreview(file, 1); in order to get an image file representing the video after playing for one second. Then both file (the video selected by the user from the filesystem) and videoPreview (the preview image) can be separately uploaded.

When you then want to show the video preview, you can simply load the image file and then begin to play the video once it is selected by the user.

The Night Circus by Erin Morgenstern

Wed, 14 Jul 2021 17:30:00 +0000

Every now and again it’s nice to dive back into a young adult book. I recently read The Night Circus by Erin Morgenstern.

The book is a sort of dark romantic/fantasy mashup. It’s about a travelling circus, those who perform in it, and those who run it.

Le Cirque des Rêves is not just any circus, however. It’s only open at night, and always closes before dawn. What appears to be clever trickery may actually be much more behind the scenes, and the garish colours found in other circuses have been replaced by a simple black and white colour scheme.

The story focuses on two young people - Celia and Marco - who are both participants in an unknown competition. As the story develops the characters discover more and more about themselves, each other, and the purpose and reason behind the circus itself.

What I enjoyed most about the book is its characters. They are a mix of interesting people from different backgrounds, and each is refreshing in their own way. They are all intriguing, and usually likeable, despite some having perhaps darker and more selfish motives.

The story itself also has interesting concepts. Although it jumps back and forth a bit between points and places in time and location, it’s great to observe how the characters and the various relationships between them grow. The “magic” and more technical elements are never really fully explained, which leaves significant parts of the story’s background to the reader to fill-in.

I can recommend this book if you’re looking for an easy-going story and if you don’t mind finishing with a few unanswered questions on your mind!

Syncing RSS feeds with FreshRSS

Mon, 12 Jul 2021 11:00:00 +0000

I enjoy reading my RSS feeds across my devices - whether it’s on my phone when out and about, my Mac in between bouts of work, or my iPad when in downtime.

Being able to sync feeds across these devices is important to me, both so I can maintain a single collection of feeds and to ensure that I can keep track of read/unread articles as I switch devices.

There are lots of web-based clients available, but using Reeder - a native app - gives a far nicer reading experience. There are lots of other clients for other types of devices too.

By default, Reeder syncs using iCloud (I guess using CloudKit?). This means that the information is not easy to extract and is potentially subject to lock-in.

Reeder also supports FreshRSS as a backend. FreshRSS is a self-hosted feed aggregator with a web UI. Importantly, the project also includes a Google Reader compatible API, which allows it to act as a backend for Reeder (along with many other compatible clients).

This reduces the lock-in, and there is a clear flow of data and source of truth. A much more open arrangement than syncing via iCloud.

FreshRSS can easily be run using Docker. I recommend setting up a (sub-)domain for it and running it behind a reverse proxy that will handle the TLS side of things.

Once setup, you should be able to navigate to your instance using the domain you chose in order to register a new account. Once logged-in you can begin to subscribe to feeds, import settings, and change some of the administrative options (e.g. to disallow additional registrations).

One setting in particular (under Administration -> Authentication) allows you to enable API access for use by app clients.

Once that’s enabled you should be able to configure your client to use your FreshRSS instance as a backend. For example, in Reeder I go to Accounts -> Add Account -> FreshRSS and then enter the server details:

For the Server field, enter your domain plus the /api/greader.php path at the end to allow the client to integrate with the API.

Once done, you can manage your feeds as usual by subscribing and organising into folders from your client. Then, if you login to your FreshRSS instance within clients on your other devices, everything should be kept in-sync.

The '5AM Club'

Wed, 07 Jul 2021 19:53:00 +0000

Recently my colleague was talking to me about the concept of the “5AM Club”, as defined in the book by Robin Sharma.

The “Club” is focused around starting your day early, with defined time slots for exercising and thinking.

There’s a great video here that summarises it all in about eight minutes.

The rough idea is to get up strictly at 5AM, and then spend 20 minutes exercising, 20 minutes reflecting, and then a final 20 minutes growing. By 6AM you are then energised and invigorated to start your day more effectively and successfully.

The motivators for this are that many successful people - including Tim Cook and Oprah Winfrey - begin their days early, and the concept of “if you want uncommon results [e.g. success] you need uncommon habits”.

Although I find it difficult, I am a big believer in getting up early. I find that the days I get up for a run at 6AM are the ones in which I do my best work, and I have a continual buzz that lasts all day.

Conversely, when I stay in bed until 8AM I feel a little sluggish, and that feeling can last until the evening. This must definitely have an effect on my work and life outlook.

Having said that, the ability to get up at 5AM for such a routine every day - whilst great on paper - may not work for all people. For example, those with young children awake all hours of the night (where every minute of extra sleep counts), or those with a social life in the evenings beyond 10PM.

To get up at 5AM, and still get enough sleep, it would mean going to bed at 9PM each evening. I usually go to bed quite a bit later than this, and whilst I can (mostly!) get up for early runs when I need to, I think this additional commitment may be too much for me at this point in my life. But maybe these are sacrifices I should try and make?

Do you have any “uncommon” habits that you think make you more effective? If so, I’d love to hear what worked for you!

How to resize images client-side in your webapps

Tue, 06 Jul 2021 21:08:00 +0000

The problem

Image processing and resizing is a common task in many types of applications. This is made even more important by modern phones that take photos several megabytes in size.

For example, if you offer an application that allows people to choose an avatar image, you won’t want to render the full multi-MB size image each time it’s shown. This is extra data for your users to download each time (which costs them both time and money, and can give a poor sluggish experience) and also means you need to fork out more cash for the additional bandwidth usage. If you have lots of users, then this time/money saving can be amplified significantly.

Lots of tools exist for server-side processing. However this may mean you need to invest in infrastructure to handle asynchronous or scheduled server-side processing, or additional compute capacity to process image uploads on-the-fly.

I’m a big fan of letting the users’ clients do more of the work, since this helps to distribute the compute power required. Allowing your user clients to upload files directly to a service such as S3 is often quicker and means you don’t need to provision and pay for the bigger server capacity yourself to deal with centralised processing of image files.

So, to return to the avatar example above, what if your web front-end can instead resize the image before it is uploaded? This means you don’t need to do the additional server-side processing, you store less data, and the process is a little more predictable.

Client-side JavaScript resizing

Modern browsers - both on desktops and mobile devices - are more than capable of doing a bit of additional work. For image-resizing, as you might have guessed, this involves making use of the HTML canvas element.

Essentially, the process involves the following steps:

Read in the user-chosen file (e.g. from a file chooser) as a data URL.
Load the image from the file.
Determine the correct dimensions for the image.
Draw the image to the canvas.
Build a new file based on the canvas.

The code below should help illustrate this further.

// Allow the caller to specify the file and a max width/height for the file
const resizeImage = (file, maxWidth, maxHeight) => {
  return new Promise((resolve, reject) => {
    // Create a new FileReader
    const reader = new FileReader();

    // Once the FileReader is ready...
    reader.onload = e => {
      // Create a new image
      const img = new Image();

      // Once the image is ready...
      img.onload = () => {
        // Create a new canvas
        const canvas = document.createElement('canvas');
        const ctx = canvas.getContext('2d');

        // Determine the new image dimensions based on maxWidth and maxHeight
        let width = img.width;
        let height = img.height;
        if (width > height) {
          if (width > maxWidth) {
            height *= maxWidth / width;
            width = maxWidth;
          }
        } else {
          if (height > maxHeight) {
            width *= maxHeight / height;
            height = maxHeight;
          }
        }
        canvas.width = width;
        canvas.height = height;

        // Draw the image to the canvas with the new sizes
        ctx.drawImage(img, 0, 0, width, height);

        // Build and return the resized image as an image file
        canvas.toBlob(blob => {
          resolve(new File([blob], file.name));
        }, 'image/jpeg', 1);
      }

      // Begin to load the image
      img.src = e.target.result;
    }

    // Begin to load the file to the FileReader
    reader.readAsDataURL(file);
  });
}

// Elsewhere in your code (e.g. after the user has selected an image file)
const resizedFile = await resizeImage(file, 400, 400);

// Now you can upload the resized image, etc....

At this point you can upload the resizedFile directly to a storage provider, such as Amazon S3. In case it helps, there is more information on the direct-to-S3 file upload process in this article.

If you care less about storage cost and would prefer to store multiple image sizes anyway (in case you want to offer users the chance to view the full-sized avatar too), then you can upload both file and resizedFile. You can then choose to serve one or the other in different scenarios.

Either way, the key thing is that you don’t need any extra image-processing on the server. I’m sure there are more efficient methods than this in practice, but if you’re looking for a simple, no-dependency approach then I hope this might help!

Project Hail Mary by Andy Weir

Mon, 05 Jul 2021 19:45:00 +0000

Andy Weir has become renowned over the past decade for his science fiction novels. The Martian (and its movie) was hugely enjoyable and successful. I wasn’t so keen on Artemis, but still did enjoy the excitement of the story.

I thought his latest book - Project Hail Mary - was fantastic.

The story opens with a lone astronaut waking up in a spaceship that he has no memory of. He doesn’t know where he is, who he is, or how he got there. Although he works out that he is of pivotal importance to the survival of the human race, the story cleverly keeps you guessing about what might come next right to the end.

Whilst perhaps the story is a little more far-fetched than his other novels - particularly The Martian - it still sits very much within the realm of possibility when compared to most other science fiction stories. It’s set in the - more or less - present day, and, as always, everything is explainable by the author using maths and science.

This helps to make everything still feel “real” and relatable. The characters are all great, and each with their own quirks. I enjoyed how the story switches back and forth between the past and present in order to explain current events as the story continues to unfold. You build a bond with the astronaut as you learn things about his own past and the present developing situation together as the story progresses.

My only wish was for the story to be longer! I felt that the ending was perhaps a little rushed and wasn’t as satisfying as I’d hoped it would be.

Interestingly I also recently read We Are Legion (We Are Bob) by Dennis E. Taylor. Published about five years ago, it follows a vaguely similar storyline with the main character leaving Earth on a mission to save humanity. The Audible versions of both books are narrated by the same person - Ray Porter - who is fantastic at injecting even more energy and excitement into the stories.

Project Hail Mary and We Are Legion are relatively short reads (or listens), and so I can definitely recommend them both if you’re in the market for new fiction books.

Using Blurhash to create placeholders for your images

Wed, 30 Jun 2021 21:14:00 +0000

Loading indicators

In user-facing software, loading indicators are extremely important to let your users know that something is happening. This is the same no matter whether your software is a CLI program, a GUI application, a web app - or anything else.

Without such indicators, users of your software may become frustrated, or assume the program has crashed, and try to close it or leave.

Generally speaking, developers should try to keep long-running tasks to a minimum (or even offload them to a cron-job or an asynchronous server-side worker). However, in some cases this is not possible. For example, in a cloud-based file storage solution, in which uploads and downloads are a core and direct user-facing feature, the user must wait until a bunch of files finish uploading - though any post-processing can of course still occur in the background afterwards.

A loading indicator can be as simple as a Loading... string (appropriate for CLI apps) or perhaps a spinner (appropriate for mobile and web apps). However there are lots of other interesting approaches to this too.

For graphical user interfaces - particularly on the web - using placeholder layouts (or “skeleton screens”) is a great way to give the user an idea of what to expect in terms of layout before the actual data finishes loading and rendering.

Loading text vs images

In a GUI or web app, text data is quick to load due to its smaller payload size. For example, when making a request to a webpage containing a mix of text and larger images, the text will load first and render and the user must then wait until the images finish downloading before they can be displayed.

This may cause the page to shift about once the images do load and get rendered on-screen - this is particularly annoying if the user has already begun reading and lose their point in the document as elements get re-positioned.

One method to avoid this is to show an image preview, which gets rendered at the same time as the text. The preview can then be “filled in” with the real image once it loads.

To get an image preview on-screen at the same time as the surrounding text and other components one can deliver a smaller version of the image alongside the rest of the data. For example, by generating and delivering an inline data URL of an image directly within the HTML returned from the server.

Modern web browsers are remarkably efficient at rendering this type of thing. However, it does require that you do some (probably) server-side image processing in order to derive a compressed version of the image before returning it in your HTTP response.

Blurhash

A perhaps nicer way of accomplishing this is to use Blurhash. This tool enables the derivation of a compact string representation of an image that can easily be stored alongside your other data - right in your database - and can easily be returned in API payloads.

Essentially, the library takes a “hash” of the image, which results in a short string. This string can then be decoded into a smaller image that can be rendered.

As an example, we can use this picture of my dog:

Using Blurhash to encode this image, the following string is returned: URH_SPDl_HxZItM|Iqt7EQxrIpNI9uj?jboM. When decoding this string back into an image we get something like the below:

It is clear that the second image is an approximation of the first. Given that the hash string is so short, downloading and rendering this as a placeholder before the full image itself downloads is quick and easy. It gives a nice preview of the image to the user for the short time it takes for the full image to load.

Blurhash offers additional ways to tweak the hashing algorithm (e.g. to make it more or less “detailed”), and has implementations for many different languages and frameworks - across desktop, mobile, and web.

I recommend checking out the documentation for more information on this.

Using Blurhash in a JavaScript app

If you are writing a webapp, this section may help you get off the ground with Blurhash.

Use of this library involves two steps; deriving the string “hash”, and then rendering this string as an image placeholder. To get started add blurhash to your project (e.g. with yarn add blurhash).

First of all we’ll look at getting the Blurhash string from an image. For this, we can do the processing browser-side to save on server resources. Assuming the user has just selected an image file for upload, the approach relies on drawing the specified image file to a hidden canvas that can be used to take the hash. The JavaScript code below illustrates this process.

import { encode } from 'blurhash';

const getHash = file => { 
  const canvas = document.createElement('canvas'); 
  const context = canvas.getContext('2d'); 
  const image = new Image(); 
  return new Promise((resolve, reject) => { 
    image.onload = () => { 
      canvas.width = image.width; 
      canvas.height = image.height; 
      context.drawImage(image, 0, 0); 
      const imageData = context.getImageData(0, 0, image.width, image.height);
      resolve(encode(imageData.data, imageData.width, imageData.height, 4, 4)); 
    } 
    image.src = URL.createObjectURL(file);
  }); 
}

// Elsewhere in your code (e.g. once the user selects a file)
const hash = await getHash(file);

The hash variable can then be sent up to your API and stored safely in your database.

To later render the Blurhash string, it can be returned directly from your API along with other relevant information and then decoded into an image.

Below is a React component - BlurrableImage - I use to render an image’s Blurhash whilst it loads in the background. Once loaded, the image gets rendered in place of the Blurhash.

import React, { useState, useEffect, useRef } from 'react'; 
import { decode } from 'blurhash'; 

function BlurrableImage({ src, blurHash }) { 
  const [loaded, setLoaded] = useState(false); 
  const canvasRef = useRef();  

  useEffect(() => { 
    const canvas = canvasRef.current; 
    const context = canvas.getContext('2d'); 
    const imageData = context.createImageData(200, 200); 
    const decodedHash = decode(blurHash, 200, 200); 
    imageData.data.set(decodedHash); 
    context.putImageData(imageData, 0, 0); 
  }, [blurHash]); 

  return (<> 
    {/* This image will never display. We just use it so we know when the browser has downloaded it. */}
    <img 
      src={src} 
      onLoad={e => setLoaded(true)}
      style={{ display: 'none' }} 
    /> 

    {/* When the image has been downloaded, we can render it. E.g. here we use it as a background image. */}
    {loaded &&  
      <div style={{
        width: 200, height: 200, 
        backgoundSize: 'cover', 
        backgroundPosition: 'center',
        backgroundImage: `url(${src})`
      }} /> 
    } 

    {/* We only show this canvas while loaded == false */}
    <canvas 
      width={200} height={200} 
      ref={canvasRef} 
      style={{ display: !loaded ? 'block' : 'none' }}
    />
  </>); 
} 

export default BlurrableImage;

If you display images in your application, Blurhash might offer a good solution for keeping your interfaces speedy and intuitive.

Self-host your web searches with Whoogle

Thu, 24 Jun 2021 21:48:00 +0000

Google and DuckDuckGo

It’s common knowledge that part of Google’s business model is to use the data it knows about you, your searches, and browsing patterns in order to more effectively serve ads.

Many people feel uncomfortable with this and so there is a strong movement to adopt more privacy-focused options, such as DuckDuckGo. This was my position, too. For a few years I’ve been a solid DuckDuckGo user, and it was my default on Mac and mobile devices.

However, I do find that for more technical queries - e.g. for specific parts of an API’s documentation - it doesn’t perform as well as Google. DuckDuckGo supports using bangs for automatically forwarding searches to another service. For example, prepending a search with !g will forward the query to Google instead.

As time went by, I found myself using !g more and more - for both technical and non-technical searches. It got to the point where I was just !g-ing everything. And so I wondered what the point was in proxying through DuckDuckGo at all.

Self-hosting Whoogle

Not long ago I saw a link to Whoogle Search - a self-hosted open-source Google search replacement.

Whoogle does not display ads, doesn’t rely on lots of JavaScript, and still returns great results. As long as it’s hosted somewhere in the cloud then there is no reason for Google to be able to track you, either.

Getting it up and running on one of my servers was super easy. I use Docker to deploy services and so after pointing a subdomain to the server, setting up the needed certificates, and adding a virtual host to my nginx container, all I needed to do to get Whoogle running was to pull and run the container:

docker run -d -p 5000:5000 benbusby/whoogle-search:latest

I could then browse to the server and begin searching.

There are lots of ways the deployment can be tweaked, including the interface and security settings, so be sure to take a look at the documentation for more options.

Setting Whoogle as Firefox’s default search engine

The main way I search the web is to simply type my query into the address bar in Firefox. To make the best use of Whoogle I needed to configure Firefox to use Whoogle as my default search provider.

There is no direct option for this in the Firefox settings, however Whoogle complies with the OpenSearch standard. This means that if you visit your self-hosted search-engine in Firefox you should be able to add it to Firefox by right-clicking the address bar and selecting Add “Whoogle Search”.

Once Whoogle is added to Firefox it can be set as the default search engine through the standard Firefox settings.

On mobile Firefox - which is even more great now that iOS allows for selecting a default browser! - the process is slightly different. Simply tap Add search engine in the app’s settings, and in the URL box enter https://server.address/search?q=%s (replacing with your server’s address) to add your Whoogle server as your search engine.

If you’re someone that enjoys self-hosting things, then I recommend giving Whoogle a try.

Joining a panel at Wales Tech Week

Wed, 23 Jun 2021 22:21:00 +0000

Wales Tech Week is an annual event held by Technology Connected. The 2021 event is running this week, aiming to bring technologists together from a wide range of businesses and organisations across Wales.

Today, I was a member of a panel discussing blockchain - “Welsh Businesses Bringing Blockchain to Life”. I was speaking alongside experts from other companies working in the blockchain and crypto space, and an academic focused on applying the technology to government functions.

It was a great opportunity to talk about how we are using blockchain at Simply Do to power some of our complex supply-chain processes, and to also hear about the exciting work from the other panellists.

The panel was excellently chaired by David Blake from the Development Bank of Wales. They are one of our investors and have been of great help and support to us over the past few years.

I look forward to continue watching the events over the next couple of days.

Anxious People by Fredrik Backman

Sat, 19 Jun 2021 18:13:00 +0000

Anxious People is a book about an attempted bank robbery in a Swedish town (not Stockholm!). It is written by Fredrik Backman.

The story involves a would-be bank robber arriving unexpectedly at an open apartment viewing whilst trying to run away, and taking the prospective buyers hostage in the process. It is mostly split between being set at the apartment itself and the police station in which the hostages are separately interviewed after the event. It is told primarily from the perspectives of the bank robber, the hostages, and the police officers.

I think that the first few chapters set this book off in the wrong light - they seem a little childish and appear to be filled with annoying and unconvincing characters. However, once past the first few scene-setting parts the story comes into its own.

Very quickly I got the impression of a deeply intertwined collection of lives that span across all of the characters, from the hostages to the bank robber and the police officers. The interconnection also spans across time, with past events that affected one or more of the characters having impact on their present lives, and the particular situation at hand.

The author cleverly introduces concepts and events earlier on in the novel, which then solidify and take on further meaning and importance as each character’s story progresses further.

Characters I didn’t really like at the start of the novel soon become more relatable as I understood them more clearly towards the end. It’s definitely an interesting book and one I can recommend as a light, but thought-provoking, read.

Tmuxinator: simple terminal workspaces for your projects

Fri, 18 Jun 2021 19:55:00 +0000

Living without workspaces

IDEs and richly-featured text editors - such as VS Code and Sublime Text - support many great features. One of these is the notion of projects or workspaces.

Such workspaces let you save your project’s development configuration to disk - things like the project directory, open files, editor layout, integrated terminal commands, and more. Often, each project can have its own workspace, too.

If you use workspaces then you don’t need to go through the tedious process of setting everything back up again each time you switch project, re-open your editor, or reboot your computer.

However, if, like me, you use the terminal as a primary development environment, things don’t work quite so nicely out of the box. For example, I use tmux as my primary development environment, and make use of multiple windows and panes for things like Vim, source control, logs, and running commands.

At any given time, I might have a handful of tmux sessions running (one for each project). A single small session project might consist of a web API service and a separate front-end - each comprising Vim editor panes, and a mix of other things. Context switching is super easy, as I can just detach from a session, and then re-attach to another one that tmux has kept running for me in the background.

However, the pain point comes when rebooting. Once the tmux server process terminates, all of the running sessions are lost. This means setting each session up again individually each time you want to begin working on a different project after rebooting.

It certainly feels like a blocker to performing system upgrades that require reboots, and is also extra friction that may prevent one from working on specific projects if the set-up is too painstaking. Both of these are clearly not good.

However, there is a solution: tmuxinator.

Tmuxinator

Tmuxinator is a program that partly aims to try and fix the workspace problem for tmux-based workflows, and my life is so much easier because of it.

The program does not interfere with the tmux server directly, and neither does it maintain individual explicit tmux session data - tmux sessions are still lost after reboot.

However, what it does do is make workspace session management so much easier by storing your project window and pane layout in a simple YAML file on disk.

For example, a simple API and separate web front-end project (as mentioned above) could be described as the following tmuxinator project:

name: cool-project
root: ~/project/my-cool-web-project

windows:
  - api:
      root: ~/project/my-cool-web-project/api
      layout: main-vertical
      panes:
        - vim
        - app:
          - source .venv/bin/activate
          - source .envfile
          - flask run
        - zsh
  - frontend:
      root: ~/project/my-cool-web-project/web
      layout: main-vertical
      panes:
        - vim
        - yarn start
        - zsh

This project represents two tmux windows, each with three panes: an editor, a server (or watcher), and an empty shell pane that can be used for issuing commands (like git). The inbuilt main-vertical layout automatically provides a nice, big Vim window (in these cases) for editing, and then a vertically-split extra pair of panes.

Each window has a separate root directory, and the project as a whole has its own root directory too, to provide better automatic working directories in case new windows are later created when inside the session. Each session and window also gets its own name (e.g. api and frontend above) to make identification easier later on.

If this file is stored in ~/.config/tmuxinator/cool-project.yml, one can simply run tmuxinator start cool-project to get started. If the project is already running it will attach you to it as-is. If the project is not currently up and running, tmuxinator will go ahead and create all your windows and panes, run the commands you specify, and connect you to the new session.

Once inside the session, it’s just controlled by plain old tmux. To detach from the session, just use the usual tmux sequence (by default ctrl-B-D). You can then connect back to the same session or another one.

Creating more project configurations

Tmuxinator comes with lots of other commands to make set-up easier. For example, running tmuxinator new will open up your editor on a template YAML file for you to edit and then save.

If you have lots of similar types of projects with the same layouts then the copy command is useful for duplicating projects as a convenient start-point: tmuxinator cp . You can also list and delete projects in a similar way: tmuxinator ls and tmuxinator rm .

Definitely take a look through the documentation to learn more about these and other commands.

Installing tmuxinator

Many distributions include tmuxinator in their repos. On macOS it’s a simple brew install tmuxinator.

Take a look at the installation instructions for more information.

Backing-up and syncing projects

Being able to recover tmux sessions is great, but what if you want to sync projects between devices, or back them up?

There are various approaches for this. I store my configuration files in my personal Nextcloud, which means I can hydrate any new devices with a simple link:

ln -s ~/Nextcloud/dotfiles/.config/tmuxinator ~/.config/tmuxinator

That way, if I use tmuxinator new to create a new project configuration it will automatically get synced to my Nextcloud. This approach also works if you use software like Syncthing.

If you work in a team and want to share your setup through version control, you could also commit the project-specific YAML file to your repo. The tmuxinator start command will look in ./.tmuxinator.yml before anywhere else, and so this offers a nice way to get your whole team using a consistent setup. However, in my experience the workspace setup can be quite a personal thing!

If you have any other thoughts for maintaining terminal-based workspace sessions, then I’d love to hear them. Please let me know.

RSS: include your entire posts in your feeds!

Sat, 12 Jun 2021 11:05:00 +0000

Recently I’ve noticed that some of the RSS feeds I subscribe to have become more and more restrictive. A post might contain just a title, or perhaps a short snippet or introductory paragraph, with the expectation that I then proceed to follow the link to visit the website itself in order to view the post in full.

I suppose in many ways that this is similar to distributing podcasts via RSS: the feed contains the podcast title, description, and other metadata, and then a link to download the podcast episode file itself. But this is because podcasts are in audio or video format and cannot be reasonably embedded directly into an XML file.

But blog posts and articles are primarily text, for which XML is perfect for transferring.

Most of the examples I’ve seen are commercial news outlets that probably still make (at least some) income through selling adverts. Although I still disagree with this (we all use ad-blockers anyway), in these cases they have a business objective to get you to their website for their own analytics and to drive their revenues.

However I have seen some personal text blogs and non-commercial outlets doing the same thing, and if this is intentional I just wonder what the motivation is. Maybe it’s for site analytics? Or maybe the author is worried about the size of the feed’s XML file getting too large?

If you need analytics of subscribers the author can simply log basic request info to their feed’s XML file download. To prevent the XML file from getting too big authors can simply limit the feed to the most recent n posts.

Either way, there are a number of good reasons for allowing your subscribers to retrieve entire post contents in their feeds.

Client familiarity

Many people use a familiar client to consume blog posts and articles. For example, I use Reeder, which has a fantastic interface that makes reading posts and content very enjoyable. It uses a great readable font, displays images beautifully, respects my system’s dark/light theme, and much more.

Other people might enjoy different clients, but the point is that this makes the experience consistent across feeds. Therefore, consuming the content is much quicker and feels more natural. If I have to visit your website to view the article then I have to find where the content starts on your particular site, deal with whatever font and colours you choose and have inconsistent layouts across my different subscriptions.

I often read posts on my phone, and if your website is non-responsive to smaller screen sizes then this is a massive pain. In general, reading your posts via a client is much less invasive on my time and I can concentrate on actually enjoying the content.

Clients can also make use of accessibility features (like screen readers) in order to make your post available to a wider audience.

Caching

When my client refreshes the feed it downloads all the latest unread posts (as well as storing previously-read ones). This means that if I am about to take a flight or get on the tube I know I will have lots of interesting content to read whilst my phone is out of the network’s reach.

However, if I have to visit your website to view the post then it simply can’t be read. By the time I’ve landed the title has probably been forgotten and I won’t remember to go back through and load it.

Protocol agnosticism

RSS is protocol-agnostic in the sense of accessing the content within. For podcasts this may be a link (usually using HTTP) to access the episode file.

For text feeds, it doesn’t matter what the “source” is: it could be a website, an FTP server, a gemini capsule, or anything else. Maybe, in some cases, there isn’t an explicit source and RSS is the primary means of distribution?

Either way, one shouldn’t assume that people always want to access via HTTP, and so including the text content directly in the feed helps to keep it pure and simple.

Trust

If I can read your content directly within my own client, it helps build trust. I know you aren’t trying to track my every move and that you care about my ability to read the content.

I know that you are sharing your content for the sake of the writing or piece itself (perhaps because you enjoy writing or want to share your thoughts), and not in order to drive sales or use patterns to manipulate me into carrying out an action that you want. It also shows you respect user privacy.

Lightweight

Distributing text-only versions of your posts is much lighter than having to transfer entire webpage files, CSS, JavaScript, and much more. In fact, if you pay for server space with per-MB billed traffic egress then this could help save you money.

More and more people in the tech community browse the web without JavaScript enabled anyway, so if your site relies on JS to load then they won’t be able to view your content. Think about the people in the intended audience of your post.

Conclusion

One can easily argue that “this is my site, and if I write the content then I want people to view it my way”. This is perfectly fine, and is entirely up to you. This post is more about explaining why these practices might convey quality-of-life enhancements to your readers, and is just my opinion and not a set of rules.

The web (and internet in general) is great in that it gives everyone a platform to distribute their content in the way they choose. However, since it’s up to others if they choose to read what you post, by making this easier and more accessible to them you can make sure you reach a wider audience.

City-centre beekeeping

Fri, 11 Jun 2021 19:00:00 +0000

Getting and “installing” the nuc

For his birthday a few years back, I bought my (now-)husband a beehive and a honeybee nucleus.

Some might see this as a strange gift, especially given that we live close to the city centre. It was certainly a surprise for him, but given his love for animals and science I knew he would like it.

We were lucky enough to have a relatively large garden, given our location, of around 20 metres in length. Since we didn’t really use the end of the garden much, it was a good location for hive - though other people have successfully kept bees in much smaller areas and on rooftops.

The hive is an Omlet Beehaus, which features two entrances (one on either end). This means that - using the included divider board - one can keep two separate colonies, and the distance between the entrances is sufficient to ensure the bees know which one is theirs.

The image above shows the Beehaus hive and the nuc box on the floor. The bees are being added to the right-hand side colony and there is a divider board in place.

I collected the nuc from a farm in Oxfordshire, England - a few hours’ drive from our home. The nuc box had to be kept cool on the way home and the bees also needed water. It was a little disconcerting travelling with a box full of bees!

After we got them home, we moved the bees to the hive by transferring the nuc frames (which are shorter than normal brood frames) from the nuc box to the brood box of the Beehaus.

The queen

Luckily, the queen had already been marked and so she was relatively easy to identify. Each colony has one queen, which is marked by some harmless ink on the back of her body to make her more visible. There is a recognised standard for marking queens based on the year in which they are born - ours was white for “2016”.

The image above shows a frame of capped brood in our nuc. The white marked queen is visible near the centre bottom of the frame.

Queen bees are extremely important for the correct function of a honeybee colony. They lay the eggs that become new bees, they secrete hormones and other scents that control the colony in various ways, they dictate the mood of the colony, they choose when it’s time to swarm, and much more.

Without a queen present, the bees will try and create a new queen out of an existing larvae at the right stage of development. Without an egg-laying queen, a colony would eventually die out.

Our first queen was lovely. The colony was super-calm (we often wouldn’t need a suit for inspections), there was always a healthy brood, and the honey collection was strong even from the start.

The first few weeks

We got the bees at the start of summer. Since a new colony from a nuc takes some time to get up to full-size, it was unlikely that they’d swarm the first year. Swarms take place in the summer months, but only if certain conditions are met.

This was good, as it gave us a year to get the hang of things before having to deal with a swarm!

Over the first couple of months, the colony grew in size and we gradually swapped out the nuc frames for full-size brood frames in order to give the bees space to grow (i.e. create the cells for storing eggs, larvae, pollen, and honey).

The workers (female bees) were out in full-force collecting pollen, and the drones would do their thing (usually nothing) back at the hive.

Workers can travel a good few miles on a single trip, and gradually travel further out as they learn the local area. Due to this range, it isn’t recommended to move bees within a few miles of their established location, since when they leave the hive there is a good chance they will recognise the area and travel back to where the hive used to be.

Bees can be quite sensitive to changes in the hive location, with the general rule being to move the hive only less than one metre OR more than six miles.

We worried that the neighbours might be a little nervous about a colony of bees so nearby, but actually they seemed to really enjoy it. They often asked us about it and planted flowers especially!

This felt really positive given current worldwide efforts in conserving bee populations.

One of our neighbours also had a pond, which was great as it allowed the bees to drink (they don’t like to drink water from sources too close to their hive).

Generally, bees aren’t interested in people at all. They would only become aggressive if they feel their colony is under threat, and so finding a bee out in the wild or a few metres away from the hive is pretty safe. Bees usually travel quite quickly about 10 metres above the ground, and would usually only descend when collecting pollen or when returning to the hive.

Adding the “supers”

We were able to add the shorter “super” frames after a couple of months. It was a long summer, and the bees continued being quite active long into September and beyond.

Super boxes and frames sit on top of the brood box (you can see two in the photo above), and there is a “queen excluder” board that sits between the supers and the rest of the hive. The excluder features holes that allow workers through to deposit honey into cells in the super frames, but are too small for the queen to get through.

This is useful for harvesting honey, as the eggs and larvae and pollen stay below in the main brood box. The workers take the excess honey into the super frames, which can be easily withdrawn without disturbing the main hive.

Bees collect more than enough honey and place most of it in their main brood box, which we do not remove. It is only the excess honey which is taken.

First harvest

We got our first harvest late in the summer. Cities are full of lots of different types of flowers and plants, since every garden is different, and the vibrant gardens were clearly very attractive to the bees!

The image above shows a super frame full of capped honey ready for harvest.

We checked the supers every week as part of our normal inspections, and eventually decided that the supers were full enough to harvest. The supers, when full of honey, are heavy, which is very satisfying! The honeycomb structure is effective: a single frame could easily hold several jars’ worth of honey.

We extracted the honey by carefully scraping the comb away from the foundation sheet. This gave us a bowl full of honey and wax from the comb, which we strained through cloth into jars. We kept the wax and later purified it - we still haven’t done anything with this growing collection of wax!

Once harvested, the empty frames were re-added to the supers to allow the workers to rebuild the comb and continue honey production.

Winter

As the days got colder, the colony began to naturally shrink in size. We removed the supers and gradually reduced the number of frames within the brood box. This helps the bees maintain their own internal temperature during the winter.

During the winter there is no way for the bees to collect additional food. They can feed off the honey they produced during the warmer months, but we helped them along by feeding them. This can be done by making a sugar solution and placing this in a “contact feeder” upside-down in the hive.

We checked the bees less during the winter. This helps prevent the cold air getting in when opening the lid.

Spring and swarm watch

In spring, the days began to get warmer again, which was marked by more bee activity around the hive entrance. We gradually re-added frames as the colony began to increase in size again.

As the months progressed we kept an eye out for signs of bee swarming. This can be fiddly business, but there are a number of indicators you can use. The most obvious indication of a colony preparing to swarm is the presence of queen cells in the brood frames.

Queen cells are created by workers when it is time to create a new queen. It is done by elongating a normal worker (female) brood cell outward and feeding the larva royal jelly (a special kind of honey) as it develops. This jelly makes the bee larva grow bigger than a normal bee and with the characteristics of a queen.

Queen cells are obvious to see, and usually show near the bottom of the frame. I don’t have any photos myself but there are lots if you search the web.

Once a queen cell has been capped (i.e. sealed off for its final development), it is likely that the colony will swarm within the next couple of days.

Swarm control

Swarms are part of the natrual honeybee colony lifecycle, and usually occur every year in healthy colonies. A swarm signifies the division of a colony: the old queen flies away with most of the colony, leaving some bees behind (workers and nurse bees, as well as drones) along with the soon-to-hatch queen(s).

A swarm is quite an intense process, and involves lots of loud flying bees as the hive gradually empties. However, a bee swarm is actually pretty safe: the bees are often at their most docile when in a swarm.

There are many ways one can prevent an all-out swarm. Swarm control is definitely recommended in a city to avoid concerning neighbours.

Some people clip the wings of the queen to prevent her from being able to fly (the colony won’t fly away without her). Others carry out an “artificial swarm” - this is what we opted for.

To artificially swarm the bees, we took some of the brood frames of honey and pollen from one half of the Beehaus and placed them in the other half - making sure the queen was also transferred.

The image above shows us selecting frames for transfer to the new side of the hive to artifically swarm the colony.

This left us with two separate colonies - the existing one containing the “new” bees with a few frames of brood, the queen cells, and some pollen and honey, along with some remaining workers, drones, and nurse bees. Importantly, the new colony does not have a current queen.

The “swarmed” colony on the other side of the Beehaus contained the existing queen, plus many of her workers, drones, and frames of pollen and honey and - importantly - no queen cells! This will now trick the queen into thinking she has already swarmed: it is a “new” hive, no queen cells around or brood. She can begin the process again of egg laying as before.

The two colonies

The “swarmed” colony continued to grow as before. The existing queen gradually continued laying to get the colony back up to size. We were able to place supers back on that side not long after.

In the “new” queenless colony we removed all but two of the queen cells. After a few days she emerged and went on her mating flight. During this flight she mated with a drone and returned to the hive. Shortly after she began laying and started her other queenly duties.

We were lucky in that the process worked quite smoothly. Sometimes, the first queen to emerge will explore the hive and sting any still-to-emerge queens through their cells to ensure she is the only queen. If two queens hatch at similar times, they can fight. Either way, there can only be at most one queen. If the queening is unsuccessful for any reason (e.g. they die when fighting or if they fail to emerge from their cells) the colony can be requeened by purchasing a new queen from a farm.

Uniting the colonies

Having two colonies is fine if you are a commercial beekeeper with lots of space. However, colony division would mean that we would eventually have four, eight, sixteen colonies, and so on as they continued to swarm and divide year on year.

We only really wanted one colony, and having the spare half of the Beehaus is useful for ongoing bee maintenance. As such, we opted to unite the colonies.

Uniting a colony involves getting rid of one queen and then gradually bringing the colonies together. Given the success of the first queen we had (both in terms of bee temperament and honey production), we chose to keep our old faithful.

To unite the colonies we first of all removed the new queen (I won’t go into the details here). We then modified the divider board between the two halves of the hive to add a sheet of paper. The bees belonging to the new queen would recognise an entirely different set of scents and hormones, and require a slow introduction to our older queen.

Adding the paper allows the various chemicals to gradually filter through as the bees on either side eat through. This gradual process allows bees on both sides to become accustomed to each other and to allow the now-queenless bees to get used to their new queen.

Eventually the paper is gone, and we were able to move the frames all back to one side of the Beehaus as before.

The second year

After the swarming and re-uniting, the bees carried on much the same as the first year. We had several great harvests from some very productive bees!

Swarm un-control

In our third year, we again looked out for the signs of swarming but we must have missed a trick. One day I noticed a huge amount of activity around the hive and realised what must have happened. There could have been less obvious queen cells present, or perhaps the existing queen just decided to leave early.

Either way, the bees were definitely properly swarming. Luckily, it was a weekday and so nearly everyone around was at work or at school (we worked from home at the time). We could see the bees congregating around a tree in a nearby garden - this must have been where the queen was.

The image above shows bees swarming and a mass of bees gathering around a small tree trunk.

The swarm was very noisy. We knew we didn’t really have long to try and get the queen and the rest of the bees back (they would return home once they knew the queen was no longer there). We didn’t want to just go into someone else’s garden without permission, and so I went round to introduce myself. Luckily they were home, and were very understanding.

They let us into their house and garden and we were able to scoop the mass of bees into a shoebox. There was too much activity to identify the queen, but we just hoped we had her.

We transported the bees over the wall to avoid carrying them through the house.

Success!

We were able to get the box back to the hive and emptied them back inside. We removed any of the queen cells we could find. We closed the lid and hoped the ordeal was over.

Luckily, the activity died down over the rest of the day, and the bees resumed normal activity over the next few days. All of the drama was clearly enough to convince the queen that she had already swarmed, and she was happy to resume business as usual.

Continuing on

The rest of our beekeeping adventure was much less exciting. We got stung a few times (mainly through the gloves), but nothing serious or anything to write home about. We now have a new queen from a more recent swarm.

Other than the swarm mistake we made, the bees did not bother our neighbours. They enjoyed having the bees around and actively engaged with us and them.

We definitely learned a lot (and still continue to). It feels great to help support the bee preservation movement, and they are probably some of the most interesting animals on the planet.

If you are interested in bees, have some space and some easy-going neighbours, I can definitely recommend giving it a try. You can also join local beekeeping societies to join a wider community, and everyone is very supportive!

Married

Thu, 10 Jun 2021 08:04:00 +0000

Just a quick post to say that I recently got married! By coincidence the event was three years to the day after our engagement.

It was a lovely day - great weather and really nice to see those that could attend. Hopefully we’ll get a chance to go away later in the year if/when things start opening up again 😁

I can't play games anymore

Wed, 02 Jun 2021 21:00:00 +0000

Growing up and the “Warcraft years”

In my earlier years I was fairly into gaming. I was definitely only ever a “casual gamer” in the scheme of things today, but I would play at least a small amount of something most days.

When I was young it was mainly those games based on Nintendo platforms - Super Mario, Mariokart, Super Smash Bros, etc. These were great with friends and were the kind of games (along with their various sequels) that we could play over again and for many years to come. Pokemon was also a big hit for me, which would continue on through the consoles.

As I moved into my early teens, strategy games became more my thing. My elder brother introduced me to Warcraft and I would play and re-play the various campaigns of Warcraft I and Warcraft II.

When Warcraft III became more relevant to me I would sink hours into playing LAN games with friends and siblings. This was also my first proper foray into online gaming (through Battle.net).

I didn’t really touch the other Blizzard games (Starcraft and Diablo) too much, but did get involved with a few other RPG- and simulation-type ones (The Sims, Rollercoaster Tycoon, and others).

When World of Warcraft was released in my early-mid teens, this was a bit of a game-changer. To me, it was the perfect combination of success/reward, social factors, depth of story, and (at the time) a huge world to explore.

It quickly became my only game and a sort of addiction. My friends and siblings all played it, and it would end up replacing other online hang-out spaces of the era (MSN Messenger). As such, it was much more than just a game to me - and I think many would feel the same. The nostalgia for those early WoW years has always been on my mind - though unfortunately was not really rekindled even when WoW Classic was much later released.

Moving to university

I was still playing WoW for several hours each day, even when I hit my late teens and my A-Level exams. I was lucky enough to scrape the grades needed (despite a distinct lack of study and revision!) and headed to university.

This is when things began to change a litte. I suddenly had new responsibilities, new friends, and new experiences. Gaming took a rather sudden back-seat to everything else that was going on in my life - new people, exploring, partying (and learning, I guess).

As I progressed through university, I would still play sporadically. But this would only be in my “home” life - the life I had when I visited my family and younger siblings. They were still at an age of no responsibility, and so gaming would be a natural pastime. We would play Minecraft, Garry’s Mod, and other similar games.

If we went on family holidays together I might buy the latest Pokemon RPG to play together, but this would quickly be forgotten afterwards as I got back to my main life.

When I play games with family I really enjoy it, and since in these cases I am on holiday anyway it feels like real leisure time.

Now

When I left university to begin full-time work my life changed again. I now had even less time, and I felt no real motivation to game even in my spare time. The last real game I played was Animal Crossing: New Horizons when that hit the Switch during last year’s lockdown, but that’s it really aside from occasional bouts with my siblings.

This post sounds pretty downbeat, but I don’t mean for it to at all. I actually love games as a concept - they are a work of art and often reflect years of creative input across gameplay, story, graphics, music, and more.

I love that small indie (and solo) developers can be given a platform to sell from that enables them to compete with the larger studios. I am fascinated by all the different genres, and the new twists we see on these year on year.

I still follow lots of gaming news and am interested in keeping up to date with developments.

It’s weird - I just don’t have the motivation to play games myself anymore.

I know games should be used as an opportunity to wind down and relax, but every time I do my mind protests, “you should be doing something more productive”. I don’t know why this is, and why I can’t seem to shut down any more, but there it is. I can’t really sit and watch TV either.

I would be more comfortable spending my “downtime” continuing working, learning, and improving myself. This all sounds very noble, but it’s actually pretty frustrating. I enjoy learning and working, but understand the importance of being able to shut down and relax once in a while.

Does anyone have any experience with this or have any tips?

The H.G. Wells Classic Collection

Wed, 26 May 2021 21:50:00 +0000

The Classic Collection of H.G. Wells novels contains five well-known stories: The War of the Worlds, The First Men in the Moon, The Time Machine, The Invisible Man, and The Island of Doctor Moreau.

Despite the fame of these novels, I had never read any of them until I recently listened to them via the audiobook version, which was excellently narrated by the likes of David Tennant, Hugh Bonneville, and others.

Wells is famous for being an early sci-fi writer (indeed, he is known as the ‘Father of Science Fiction’), with his first such book - The Time Machine - being published in 1895.

Books in this collection have formed the basis of both classic and modern films - perhaps most notably The War of the Worlds (published as a novel in 1898) and The Invisible Man (1897).

I’m a big lover of science fiction and it seemed sensible to go back to what can be seen as its birthplace. I really enjoyed the collection - each novel is thought provoking and engaging. The novels aren’t long and the stories progress quickly, which helps to add to the excitement.

Although quite light-hearted seeming (in a classic Wellsian and British fashion), and often humorously written, the stories do go into interesting detail around the science and methods behind the phenomena that form the plots. From discoveries of new materials in The First Men in the Moon, descriptions of interesting physics in The War of the Worlds, and the effects of chemicals on human physiology in The Invisible Man, Wells certainly explores the scientific depths whilst also leaving some to the readers’ imaginations.

In many of the books there is a significant level of human self-reflection as part of the story, which I think definitely helps place the stories ahead of their time. Whether talking about alien life, or life at a different point in time, there is definitely a sense of Wells projecting the position of humans in their new contexts, and the characters spend time either thinking or talking about themselves, their achievements (or failures), which gives the notion of humility with regard to human ability.

To me, The Island of Doctor Moreau stands out as being a little different from the others. Although it is certainly science fiction and is a well-known and acclaimed novel, I found the story’s concepts a little strange and did not enjoy it as much as the rest of the collection.

Either way, if you are interested in science fiction I would strongly recommend this collection of novels and enjoy the basis of - at the time - what would become countless further books and films often based on the same ideas and similar concepts.

The networking mall

Sun, 23 May 2021 15:04:00 +0000

Someone non-technical recently asked me the question, “what actually is a server?”. They knew it was just a type of computer that runs somewhere that can be accessible over the internet, but they were interested in how they differ from “normal” computers.

The conversation moved on to how these computers can make several different functions available at the same time over the network, which brought us on to the topic of services and network ports.

I was considering a few analogies to best describe the concept of services and ports, and then began talking about shopping malls.

Shopping malls and servers

A single shopping mall allows visitors to interact with a large range of different shops and services - such as stores, restaurants, post offices, vending machines, car parks, and more. A single shopping mall is a bit like a computer (or server).

Each unit (that hosts a service) within the mall is usually numbered, like houses on a street. For example, a specific restaurant might be given the number 2500 within the mall. This allows for each service to be addressed uniquely for easier discovery (e.g. for delivering mail or packages). Although each service can be complex and provide a range of functionality, there can only be one service available at each service number.

If, for example, I wanted to visit the post office in the mall I might visit unit number 110. Here I can prove my identity in order to receive mail that they may be holding for me. Bringing this back to severs, this concept is similar to that of using POP (Post Office Protocol) for retrieving email from a mail server; I connect to (typically) port 110 on the mail server, authenticate, and then I can download the messages.

If I wanted to know the time, I might choose to visit an exhibition of old fashioned watches that happens to be on display in unit 37. Here, I can’t interact with the service in a way other than viewing the time (and appreciating the watches), and each person can only stay for a short while. Similarly, in computing, if I connected to port 37 via TCP of a server running the appropriate Time Protocol service I should simply receive back the current time.

If I happened to work in managing the mall, I might visit unit 22 - the manager’s office (equivalent to connecting to a server via SSH on port 22) and remain there all day until I finish work.

The analogies can go much further. The essential thing is that - in computing - I can send traffic over a supported protocol to a specific port in able to interact with the type of service at that port. Some services (like the time protocol one above) might just send a response and then close the connection, whereas others (such as SSH) allow for an ongoing connection to be maintained in order to support a rich and feature-ful experience.

Although many malls have their manager’s office at unit 22, this is just convention and is not a requirement. The SSH daemon (the service that handles the SSH connection) can run on a different port if so desired. Similarly, libraries are often available at unit 80 in many malls - however in some malls there may be multiple libraries available at a range of different unit numbers (and maybe an extra secure library in unit 443).

Some malls may have a watch exhibition, but it has been closed by the managers (still sitting in unit 22). Since I can’t get in, I am unable to view the current time even if the exhibition itself still exists (I may not even know there is a watch exhibition).

Other malls may not have a watch exhibition on at all. If I visited unit 37 of these types of malls it would probably be closed. If the unit happens to be open for some reason, the unit would just be empty and I would not be able to receive the service or interact with it in any way.

Lots of malls recruit security guards to protect the entrance (and exit) of each unit. These guards (the “firewall”) ensure that visitors are allowed into the unit in order to receive the service (even if the unit is open) - perhaps by verifying their proof of address (source IP) - and turn people away if they don’t fulfil the requirements. The firewall guards may also prevent people from leaving the unit.

If someone keeps trying to repeatedly enter a guarded unit without the appropriate information, they might get banned (either temporarily or permanently).

Additionally, some units may only admit staff that work in other units of the same mall - this could be done by issuing new rules for the firewall guards or perhaps there is a non-public back corridor connecting the units that only mall staff can use (the loopback interface).

If the mall is closed completely, then I can’t reach any of the ports or receive any service. For example, if the server is currently turned off or disconnected from the network.

Some differences

Of course, the mall vs. server anaology isn’t perfect. Most servers only have a small handful of ports open at a given time, and these would be heavily restricted with firewalls and other network protections.

Equally, when someone does visit a server, they usually do so with one goal in mind (e.g. to download mail OR retrieve web content). In reality, visitors may spend a few hours in a mall and visit a large number of different shops and services.

However, I find this analogy an interesting and useful way to describe some of the basic networking principles.

How I back-up my personal server

Tue, 18 May 2021 22:13:00 +0000

For a couple of years now I have been using a self-hosted Nextcloud as a replacement for iCloud and Google Drive. I won’t go into the details as to why (especially given the additional upkeep and other overheads required), as this has been covered before - but mainly it’s about maintaining control over my data.

I use a cloud VPS to host my Nextcloud instance - rented from Linode, whom I can certainly recommend if you’re looking for a good VPS provider - and since starting my Nextcloud journey I have begun hosting a number of additional services on the same server. For example, FreshRSS (which I consume using Reeder), Monica, Gitea, a Matrix server, and more.

Considering the pervasiveness this one machine has with respect to my data and day-to-day life, and the impact it would have if I were to lose access to it, having backups for it is crucial.

3, 2, 1, Backup!

Linode offers a backup service for servers, which takes periodic snapshots in order to enable easy recovery of a service, or lost data. That’s one layer, but what happens if Linode itself experiences problems or if I lose access to my account for any reason? Having all of the live data and backups tied to a single provider was definitely a worry for me.

Many people follow the “3-2-1” rule for backups. This strategy is concerned with - for any piece of data - having at least three copies of that data, two of which stored locally but on different media, and another copy somewhere else (geographically separate).

Enabling Linode backups allows me to comply with the “3-2” bit of the rule. However, by stopping at this point there is no additional off-site backup in case of catastrophic failure.

Finding my “1”

In order to fully meet the needs of the 3-2-1 strategy, I needed to find a solution for maintaining off-site backups. Additionally, this wouldn’t be a one-time backup; ideally I’d need something that could at least back things up on a daily basis (if not more frequently).

I began researching solutions, but it wasn’t long until I settled on Backblaze B2 - an S3-compatible object storage solution that has great GB/$ pricing. Side note: Linode also offers S3-compatible object storage, but that wouldn’t help me in this scenario as it’d still be managed by the same provider.

B2 is cheaper than S3 itself, and also has the benefit of not having to maintain a complex AWS account for simple personal projects.

Setting up backups to B2

Setting up the backups involved a few simple steps:

Creating a new Backblaze B2 account
Setting-up a bucket on B2
Writing a script to automate the backup to the B2 bucket

1. Create a Backblaze account

You get 10GB free on Backblaze. Head over to the sign-up page in order to create your account.

2. Set-up your bucket

Once you’ve got your account and have verified everything, go to the “Buckets” tab of your Backblaze account’s UI, and click “Create a bucket”. This will open up a dialog.

Enter a unique name for your bucket and ensure you mark files as “private”. I also turned on default encryption for an extra level of security. When ready, click “Create bucket”.

Since we will be periodically backing-up data to this bucket, the bucket will quickly take up more and more space (and cost you more too). As such, I recommend adding a lifecycle rule to tell B2 to automatically delete “old” backup files. To do so, click the “Lifecycle Settings” option on your new bucket, and configure how long you want to keep old file versions around for (I used 10 days):

Finally, we need to create some credentials that will enable the backup system to write files to the bucket. Go to the “App Keys” tab of your B2 dashboard, and click “Add a New Application Key”. On this dialog, name your key and ensure this key can only write files to the bucket. You may also want to restrict this key to only work with your specified bucket.

Make a note of the keyID and applicationKey that will be displayed (as well as your bucket’s name and the “endpoint” shown on your bucket), as you’ll need these later.

3. Write a backup script

Backblaze does have an API for managing buckets and files, but using this (especially for larger files) felt overly complex. Since B2 is S3-compatible, we can just make use of standard S3 tools, such as awscli.

As such, my script for backups simply creates a tarball containing all of the directories and files I want to backup, and then sends it to B2. This can be as simple as the following:

#!/bin/bash

tar --warning=no-file-changed -czf /tmp/backup.tar.gz /first/path /second/path
aws s3 cp /tmp/backup.tar.gz "s3://$BUCKET/backup.tar.gz" --endpoint-url "$ENDPOINT"

Before running the script, ensure that the following environment variables are set:

AWS_ACCESS_KEY_ID (set to the keyID of the B2 key you created)
AWS_SECRET_ACCESS_KEY (set to the applicationKey of the B2 key)
BUCKET (the unique name of the bucket you created on B2)
ENDPOINT (the endpoint shown on your bucket on the B2 UI: similar to https://s3.eu-central-003.backblazeb2.com)

If these are correctly set (and dependencies like awscli are installed), you should be able to mark the script as executable and then run it to backup the directories /first/path and /second/path (obviously change these to real paths on your server, and you can always add more).

You can verify the upload was successful by browsing the bucket on the B2 interface. Please note it can sometimes take a few minutes for files to show up!

Note: I use the --warning=no-file-changed flag to prevent tar from warning about files that change during the tarball creation process (this happens to me because I backup my Matrix server files too, which change quite frequently as new messages arrive).

Automatic backups

The above setup is useful for one-off backups, but I wanted to automate the process. This could be as simple as a cron job, but I like Dockerizing things (this makes the environment variables easier to manage too).

To see my approach to automating the backup, feel free to clone and use the Docker image at this repository.

Running

Wed, 12 May 2021 19:20:00 +0000

The effects of working from home

The UK went into its first proper COVID-induced lockdown back around March time last year. At this time, our company locked its office doors and we all began working from home. We’re now all still working remotely about 14 months later and will continue to do so for the forseeable future.

Before we closed the office, I used to walk across my city - Cardiff - to get to work. It’s about a 3km walk, which would take me about 30 minutes to walk each way. I enjoyed the walk - I could stop for coffee on the way through, and the distance meant I could take different routes on different days if I wanted a change of scene.

Now, and since last March, my daily commute simply involves me walking down the stairs to the corner of my living room that is my home “office”. Whilst it is definitely convenient (and I would certainly prefer this to full-time back in an office), it had its downsides.

For the first few weeks, I just felt lazy. I was working hard (we all were, and were performing great as a remote team), but my body almost craved that morning walk. The walk was time that enabled my mind to sort itself out ready for the day of work, meetings, decisions, and everything else.

Without that walk time I felt my work starts were slower, and I was more easily distracted in the mornings. To try and alleviate this a little, I began walking around a park near me each evening after work - this definitely helped me wind down and the effects lasted until the following day.

🏋️‍♂️ Workouts

Around the same time I began working from home full-time, my brother told me about an app - Fitbod - that aims to be like a mini personal trainer. It’s not the only app of its type around, but it caught me at the right time.

I thought that having an additional exercise goal each day - as well as the evening walk - would help in making me feel more invigorated. I began using it in the afternoons after I had finished my main work for the day (before my walk).

Daily workouts, just simple ones at home following the app’s instructions, definitely had a positive effect on my mental wellbeing - it felt almost like personal meditation time for me.

It wasn’t long before I switched the routine to morning workouts (before work or after my first meetings of the day). This definitely helped my work too. I’ve been doing the same thing ever since (I think I’ve only missed 10 or so days of workouts in total for the whole of the last year).

🐶 Adopting a dog

In early December we adopted a dog, and this flipped things on their head a bit. Suddenly control over my own life changed slightly, as I now had someone to be responsible for and think of - at many times before myself. I’ll write more about my dog in a later post, but will move on back to exercise for now.

Since getting the dog, I no longer had times for nice leisurely walks after work or workouts in the morning. I now had a new member of the family that needed to be walked once or twice a day, and entertained during the times at home.

People who tell you that you do more exercise when you have a dog are lying. When “walking” him, my time is spent mostly standing around whilst he runs and plays with his friends in the park. It’s the only way he can get real exercise - walking him on a lead on my usual walk (especially a dog with as much energy as mine!) just does not give him the exercise he needs.

I wanted to find a way to maintain my level of exercise whilst also giving me the time to go to the parks for 1-2 hours each day to allow my dog to run around properly off the lead. (Note: we live in a city, and it’s not very convenient to have to drive out to countryside trails every day).

🏃‍♂️ (Re-)starting to run

Some of the people I met in my local dog-walking friend group are quite heavily into running. I used to love running in my mid-20s, and would jog 15km or so three times a week. I was put off as had been told by some people that it can have long-lasting damage on knees and other joints, and so I stopped for several years.

Coincidentally I had recently been doing research about the long-term effects of running, and the results are mixed; some studies indicate what I had heard from others (about joint issues), but many talked about the benefits of building leg muscles and how this might even protect the joints. It also turns out that running properly and with good equipment (i.e. trainers) also makes a big positive difference.

I thought that running could be a good replacement for my walk and some of my workout time - it burns the calories, helps maintain fitness, and has many positive psychological effects too. Especially if I could do it a few times a week.

The dog-walking friends mentioned a shop nearby that could run some gait analysis with me and suggest running trainers most appropriate for me. I booked an appointment, ran the analysis, ordered the trainers, and within a week had them collected and at home.

The first few weeks

I’m now about three weeks back into running, so I thought I’d report on how it’s going.

I thought I’d be much more of a mess than I actually am. I’m by no means quick (I do about 5:30 minutes per km on a good day), but I’m getting faster and definitely feel more fit. There’s certainly some muscle memory there still after all of these years.

I run an average of three times per week, and go about 6km each time. I run first thing in the mornings before doing my workout, and then work. This then gives me the time I need to give the dog a chance to run around after work.

On the days I don’t run in the morning, I instead go for a 30 minute walk with the dog.

The routine is good (I ❤️ routine), I get the same (if not more) exercise than before, and my dog gets more running time too. It means I need to get up earlier in the morning (more about that in a future post), but I actually quite enjoy that.

The main thing is that I no longer feel the laziness I felt before. I start work with a good hour’s worth of solid exercise done every day, a nice cup of coffee, and much more focus.

Self-hosted notes and to-do lists

Sun, 09 May 2021 21:31:00 +0000

In this post I will talk a little about how I handle my digital notes and to-do lists. In the spirit of my last post on data sovereignty, the focus will be on self-hosted approaches.

To-do list management

It feels odd that the first task many new technical frameworks guide users through, by way of a tutorial, is a simple to-do list; yet finding great production-ready examples of such software can be challenging.

It’s a pretty personal space. Although there are awesome time management processes out there (such as the pomodoro technique), at the end of the day everyone is unique and what works for one person doesn’t necessarily work for others.

A few years ago I got quite heavily into Todoist. It’s a very feature-rich platform with great apps across web, desktop, and mobile. It supports tagging, projects, deadlines, sub-tasks, and much more.

However, it’s almost too feature rich, and I find this can distract from the intended simplicity of to-do lists. Whilst it’s important to set up a process that allows you to work effectively, spending too long configuring and reconfiguring things is counter-productive.

It also means that your data is held elsewhere and out of your control. A better solution might be one that you can keep local and sync or self-host.

There are a few examples of open-source to-do list alternatives that you can self-host. The one I use is Taskwarrior.

Taskwarrior is Free and Open Source Software that manages your TODO list from the command line. It is flexible, fast, and unobtrusive. It does its job then gets out of your way. - taskwarrior.org

I like Taskwarrior for many reasons. But mainly it’s the speed and clarity of use - it really does just “get out of your way”. Tags and projects are created automatically for you as you go, and querying feels as fast and sensible as Ledger is for accounts.

I have my terminal open all of the time anyway, and so I can quickly and at any time view my current list (by running task), and see my currently in-play tasks listed right at the top.

I can also query by tag (task ls +tagname), or to-dos for a specific project (task ls project:projectname).

Adding todos is also just as easy, and arguably quicker than commercial offerings like Todoist. E.g. if I wanted to add a new task to buy that gift for my friend (and tag it as “life”), I can just run task add +life Buy gift for Sam and then forget about the task for now. I can then check out my “life” todos (task ls +life) at a time when I’m out of work and have time to actually complete such tasks.

I’m on a Mac, and so I just used brew install task to install it. There is likely a package for your own distribution too.

In terms of self-hosting for multi-device setups, there is the Taskserver project from the same developers, and this is the recommended approach. For me, however, I only use Taskwarrior on one device and so I backup my tasks by simply syncing them to my Nextcloud. To do so, I just edited the relevant line in ~/.taskrc:

...
data.location=~/Nextcloud/tasks
...

There is much more you can do with Taskwarrior, should you wish (including things like theming and task prioritising). I can certainly recommend taking a look through the documentation for more information.

Notes (and notebooks)

Sometimes you just can’t beat an old fashioned pen-and-paper notebook. The process of physically writing things down definitely seems to have a psychological effect on my ability to remember things. However, this approach isn’t really compatible with my other information-oriented habits (particularly backup paranoia and minimalism).

The same concepts around organising notes into notebooks, and keeping things logically organised, can still be applied to digital note-taking too.

There are a number of free and commercial offerings that exist. Simplenote is great (though perhaps a little too simple). For Apple users, Bear is also good, but potentially locks you into the Apple ecosystem.

For some time I’ve used Obsidian. I like Obsidian as it just uses your filesystem as a way of organising notes (directories are “notebooks” and each note is a simple markdown file). This approach also makes syncing over Nextcloud super easy (just set your Obsidian vault to a directory in your local Nextcloud sync folder, and away you go). There is also a mobile app that’s currently in closed beta.

Recently I’ve been trying to get more into Joplin. I like this software because it is open-source, has a terminal interface as well as GUI ones, and has mobile apps available for note-taking on-the-go.

Joplin also has native sync-ability with Nextcloud, which is useful for backup and cross-device access. I find searching quick and intuitive, and the note editor uses Vim (by default, at least), which is great for easy editing.

All in all, I still teeter on the edge between Obsidian and Joplin - both are great options and are worth exploring for your own use.

Open to ideas

I’m definitely open to other ideas for both note-taking and to-do list management. If you have any good examples of software to help with either of these then please get in touch!

Data Sovereignty

Wed, 05 May 2021 19:50:00 +0000

The term ‘data sovereignty’ is something we hear much more about these days. Increasingly I’ve also heard it being mentioned in different contexts.

We’ve seen it more in the world of enterprise SaaS; particularly in the case of UK-based public sector organisations amid post-Brexit data flow policies. More and more organisations are getting stricter in the geographic location of their users’ data. Whereas before most organisations would be happy as long as the data is stored somewhere within the EU, they would now require it to be stored onshore within the UK.

They call this data sovereignty. At our company we’re lucky to be agile enough to adapt and change our service offering to enable UK-only data processing and storage. However I can imagine many larger organisations might experience more inertia. Interestingly though, finding a UK-only mail provider isn’t as easy as it sounds - most such services offer “EU” or “US” servers, but stop there (potential SaaS service offering there: UK-based mail provider).

The other place I’ve been hearing the term is in the indie tech and self-hosted community. In this case the data sovereignty concept relates more to data ownership, where the individual maintains control over their own data, where it is stored, how it is processed, and often goes as far as to keep their own data at home (for example, self-hosted setups using home servers).

I’m definitely in this camp too; whilst I don’t keep stuff stored at home, I do keep my own data - when possible - on private servers in a secure datacentre or on services I trust. Things just feel more in-control with this approach.

Without data sovereignty, people are at risk of losing data they think they “own”. For example, someone recently lost access to their iCloud data because of issues with an unrelated service.

There’s not much more to this post. I just think it’s interesting that we’re hearing more and more of the same phrase being used in different contexts by different groups of people and organisations.

Go Time

Tue, 04 May 2021 18:44:00 +0000

I listen to a number of podcasts each week. One of these is Go Time.

The Go Time podcast releases episodes every Thursday. Its format is mostly comprised of panel discussions and interviews with founders and specialists in the community about the Go programming language. Episodes are usually between 60 and 90 minutes long.

I don’t program in Go a lot myself these days, though do have one or two older projects written in the language. However, I feel that the content is often broadly relevant for non-full-time gophers - like myself - also.

The episodes include discussions around a diverse variety of topics - such as testing, networking, web-apps, tooling, startups, programming principles, and much more. Many of these concepts are interesting to gophers and non-gophers alike, as they touch on the broader problems as well as to dicuss how Go can specifically be used to solve these problems.

Recently I have started using the Rust language more and more, and particularly on this side project which I have used as a mechanism for learning the ins-and-outs. Although the two languages (Go and Rust) are by no means the same, they do share a small number of similar attributes and I have found that the Go Time podcast has often touched on topics relevant to both languages.

Episodes also feature interesting guests from a variety of backgrounds - from specialists in the community through to startup founders. Hearing their stories is always great. Additionally, the show hosts are engaging and add light-heartedness to what can be deep technical conversations.

If you’re a programmer, and even if not a gopher yourself, I recommend checking out a few of the episodes to see if you agree.

It should come up in your podcast app if you search for “Go Time”. I use Overcast on iOS, and if you do also you can subscribe at this link.

Starting out with the Pinephone

Tue, 27 Apr 2021 21:00:00 +0000

As you may know, I recently purchased the beta edition of the Pinephone. It arrived last week in the Pinephone Beta Edition box shown below.

As mentioned in my previous post on the subject, I bought the phone for purely experimental purposes, to get involved in the community, and to be a part of the freedom and Linux-on-phone movement.

I fully understand that the device is not yet really considered ready for every-day reliable production use (especially when compared to my current iPhone 11 Pro Max). However, the Pinephone is less than 20% the price of my iPhone, and comes with the freedom to do so much more - without the restrictions of Apple’s “walled garden”.

However, I am very excited to see what can be done with it. At the end of the day, it’s just an ARM-based computer with support for running mainline Linux and the added benefit of having cellular capabilities to make phone calls and handle data connections.

It’s also super easy to try out different operating systems by simply dd-ing to an SD card - much easier than the tedious root-recovery-flash song and dance often required in the Android ecosystem.

The next few weeks

Anyway, I’m going a little off-topic. My initial plans aren’t to try out new operating systems just yet (although I am excited to try). Instead, I’d like to spend the first few weeks tinkering with the out-of-the (beta) box underlying system and seeing how well it does handle my day-to-day tasks on an as-is (i.e. without needing to change SD card) basis.

The beta edition comes pre-installed with Manjaro Linux on the eMMC along with the KDE Plasma Mobile desktop environment, so this is what I’ll stick with for now. Upon initial boot-up I can already see that it comes pre-installed with some useful packages (e.g. Telegram messenger and the Megapixels camera application).

However, below is a list of day-to-day tasks I can do on my current phone and which I will try and accomplish using the device over the next few weeks.

Basic calls and texts.
4G cellular data connectivity.
WiFi connectivity.
Bluetooth connectivity (including headphones).
Photo- and video-taking using both front- and rear-facing cameras.
Web browsing.
Podcast subscribing, listing, and listening.
Audiobook downloading and listening.
Music-playing (preferably through Spotify).
Mastodon (tooting and reading my timelines).
Twitter.
RSS (viewing my feeds from my FreshRSS server).
Email reading and sending.
Telegram messaging.
Password-management.

I’ve purposefully kept a couple of things off this list - including Whatsapp, my bank’s app, and some enterprise apps I use for work - since these systems are proprietary in nature and so would not be fair to expect of the phone. One could argue that this impacts its viability as a daily-driver, however that is not my current goal. Presently I am just looking to see how well some basic tasks can be accomplished before trying to take it further to be fully useful for daily use.

I also want to document the journey for myself and others wanting to get involved in this project.

Projects like Anbox look like potential routes for getting additional things working when needed in a pinch. However I’ll save that for another time.

Check back in a few weeks to see how I get on. If you have any advice for starting out in this way then please let me know!

After this initial period I will look to try out other shells and underlying systems. The ARM Arch with Phosh project looks like a good start point for when I come to this.

My appearance in the Wales "35 Under 35"

Mon, 26 Apr 2021 10:48:00 +0000

This is a bit of a vanity post, but back in December I was lucky enough to be included in the 2020 WalesOnline “35 Under 35”.

This list aims to present the “best young businessmen in Wales” for the year. It was definitely an honour to be included and it’s great to see the efforts from the whole team at Simply Do reflected. We’re still only at the beginning of our journey and so we have an exciting few years ahead!

Steve Jobs by Walter Isaacson

Sun, 25 Apr 2021 16:41:00 +0000

I was recently asked whether Steve Jobs was someone that inspired me. It’s a difficult question, I find; he’s definitely an inspiring person in the sense of his work ethic, the products he envisages, and his way of understanding the needs of the target customer better than they know it themselves.

As a person, however, I find his personality and the way he treats others less inspiring. I try to be empathetic to others and take into account the emotional and psychological position of someone else when interacting with them. In a professional workplace this (hopefully) contributes towards creating a space that enables people to grow and develop whilst also emboldening colleagues to put forward their own thoughts and opinions in a more risk-free environment.

Jobs, on the other hand, has his own vision and - although these visions, if executed, are bound to be successful - you’ll need to be on his train in order to succeed in working with him.

The reason my colleague asked me this question was because I was reading the Steve Jobs biography by Walter Isaacson at the time. The biography’s subject is not a hero of mine in any way, but he is indisputably a legend in the consumer technology space and so his story definitely deserves knowing (whatever your particular stance is).

Although I knew the rough story of his life - his co-founding of Apple with Steve Wozniak, his time and successes at Pixar, his founding of NeXT before his subsequent return to Apple and eventual battle with cancer - understanding how individual products came to be imagined and created was fascinating.

His relationships with others - friends, colleagues, competitors, and romances - undoubtedly helped shape his life and his successes. His obsessions over food, art (and the appearance of products, both outside and within) and his focus on work right to the end were certainly areas I did not know about, but it’s clear that these all contribute towards what he managed to achieve.

I know that a lot of people don’t like Jobs, or don’t agree with the type of closed end-to-end technology he pioneered and obsessed over (myself included), however his achievements - even by the age of 30 - and his focus on the end goal should definitely be an inspiration to all technologists.

The Giver of Stars by Jojo Moyes

Sun, 18 Apr 2021 12:09:00 +0000

The Giver of Stars by Jojo Moyes tells the story of a young English woman - Alice - who marries an American man and moves to a small town in Kentucky in the late 1930s.

Not long after arriving in Kentucky Alice realises she may have made a mistake when it comes to her new husband. However, the real story focuses around a job Alice gets working with the local library.

The library begins offering a new service, in which the (female) librarians travel around the local area (often hard to traverse due to mountainous terrain) on horseback to deliver books to those unable to get to town or who wouldn’t usually engage with the library. The concept is based on a real project - the Pack Horse Library Project - and Alice and the other women are met with many different types of personalities on their rounds.

There are focuses on racism, sexism, misogyny, domestic abuse, murder, and much more in the story, and the librarians are faced with a number of hugely difficult situations both when at work and when at home.

The story was fantastic and engaging. I enjoyed the scene-setting, and could easily picture the local town and all the surrounding countryside. You feel an undeniable sense of unfairness in the world as the story progresses - in which rich white men nearly always get their own way in most situations - however the bond that builds between the characters, and their shared experiences, show that this can be overcome.

Reporting business accounts using Ledger

Sat, 17 Apr 2021 14:00:00 +0000

As is the case with many countries, all businesses in the UK must report the state of their financial accounts to the relevant inland revenue service at their year-end (in the UK, this is HMRC).

This is also the case if you are a freelancer or sole-trader (or if you’ve made other untaxed income - e.g. from investments). In these cases, this is called your Self Assessment. Self Assessments are pretty straight forward, and can usually be completed online by the indiviual themself - as long as they have kept good accounts and know their numbers.

However, the required year-end business accounts are different and are more complex in order to account for all the different types of operating models and variety of business types. There are also various rules for businesses of different sizes and if you don’t know what you’re doing you may end up paying too much or too little tax.

As such, it is generally advisable to appoint an accountant to help you at your year-end (even if you’re making a loss!). It gives you peace of mind and also saves you time.

My business year-end passed recently. Historically I’ve used Xero to track business finances - since it is a good one-stop shop for issuing invoices, getting paid, tracking bills and finances, and automatically reconciling against your bank. It’s a great tool for small businesses as it helps you make sure everything is correctly accounted for, and it allows your accountant to easily get the information they need in order to make their reports for your business to HMRC.

However, it is a paid-for service, and if you’ve paused trading at least temporarily - like me - or if you’re going through a financial dry patch, it feels a waste to pay for something that you’re not using.

About a year ago I got quite heavily into plain-text accounting - it feels logical and in-control. I was using it for some of my personal finances and so I thought I’d also switch business bookkeeping to the Ledger approach too.

I exported my Xero accounts into a new Ledger file and paused my Xero subscription. Every month I would run through my bank statement/invoices/bills, and update the ledger and reconcile against the business bank account. As such, when it came round to year-end, I had a full set of books for the relevant tax period.

This is where I worried a little. The lady who normally files my accounts had access to my Xero and can run everything from there (many small business accountants in the UK recommend and sometimes only work with Xero). I didn’t want to have to look for and begin working with a new accountant, and so I looked to see if I could get Ledger to output balance sheets and P&Ls in a similar way to Xero.

The Ledger tool offers a number of reporting mechanisms. The most useful are perhaps the balance and register commands, which respectively show the balance across your accounts and a transaction log.

After running a few of these simple Ledger commands, I had the files I needed: a balance sheet (covering all accounts), a profit & loss account (essentially a balance sheet covering income and expense accounts), and a transaction register. Examples describing how I generated these are shown below (in this case assuming a year-end of 31st December).

Balance sheet: To generate the balance sheet I used ledger balance -b 2020/01/01 -e 2021/01/01, which outputs something along the lines of:

          £-XXXX.XX   Assets
           £XXXX.XX    Bank 1
          £-XXXX.XX    Bank 2
           £XXXX.XX   Equity:Shareholder:Dividends
           £XXXX.XX   Expenses
             £XXX.XX    Advertising
              £XX.XX    Compliance
              £XX.XX    Domains
             £XXX.XX    Hosting
             £XXX.XX    Services
             £XXX.XX      Accounting
               £X.XX      Banking
              £XX.XX      Legal
             £XXX.XX    Software
            £XXXX.XX    Tax:Corporation
            £-XXX.XX  Income:Sales:Product
--------------------
                   0

Profit & loss account: The rough “P&L” was generated with ledger balance -b 2020/01/01 -e 2021/01/01 income expenses:

           £XXXX.XX   Expenses
             £XXX.XX    Advertising
              £XX.XX    Compliance
              £XX.XX    Domains
             £XXX.XX    Hosting
             £XXX.XX    Services
             £XXX.XX      Accounting
               £X.XX      Banking
              £XX.XX      Legal
             £XXX.XX    Software
            £XXXX.XX    Tax:Corporation
            £-XXX.XX  Income:Sales:Product
--------------------
            £XXXX.XX

(Where the final line indicates the overall balance between income and expenses).

Transaction log: The register was generated using ledger register -b 2020/01/01 -e 2021/01/01. I won’t include a sample below, as a transaction log is mostly obvious. I also generated it in CSV format in case this made it easier for the accountant at all: ledger csv -b 2020/01/01 -e 2021/01/01.

I placed the outputs from these commands into separate files and sent them to the accountant, who was then able to submit the company accounts without needing Xero. This was a great experience, as it gives me confidence in the end-to-end functionality of Ledger (and other similar command-line accounting tools). Writing and keeping books using plain-text files is quicker than Xero (which can be quite clunky), and now I can also easily get the information out the other end too. And it’s free!

Six months of Invisalign

Mon, 12 Apr 2021 21:38:00 +0000

Back in November I started an Invisalign course to help straighten my teeth. Invisalign works like traditional braces, but is instead formed from transparent teeth “trays” that others can only really notice up-close. Given my personal situation, this seemed like a better approach than the traditional metal braces.

In all honesty, my teeth weren’t that bad to begin with but - like many people - as I got older I was beginning to notice a little more “crowding” (where teeth bunch together and start to move out of place). Invisalign was something I had wanted to try for a while, and whilst the UK was in lockdown and I couldn’t see anyone anyway, it felt like a good time to go ahead with it.

The process

I had a couple of initial appointments with my dentist just to ensure I was dentally fit for orthodontic work and in order to take a scan of my teeth. The scan was cool - it showed exactly what my teeth looked like and the software then uses the result to design a series of aligners that would bring the teeth back into line. I also got access to a website on which I could see how my teeth would be moving over time.

After my scans, I went back to the dentist a couple of weeks later in order for some attachments to be added to my teeth and to collect my newly-manufactured aligners. In total I was given 22 sets of aligners, with the aim being to start with set number 1 and then proceed to the next one each week - every change in aligner gradually moving the teeth into line.

I was also given a scanbox, into which I could place my phone in order to submit photos of my teeth every week through an app to my dentist. This enabled him to track the progress each week and to ensure I moved onto the next aligner set at the right time.

For the next two-three months I wore my aligners for 22 hours each day. Every week, I scanned my teeth and was instructed to move onto the next set of aligners in order to progress the treatment. In February I had to go back to visit the dentist in order to have some additional filing between some of my teeth so they could move into position properly.

I then continued for another few months - until today. I completed my last set of aligners last week and had a check-up this afternoon to see how things went. I was pleased with the result, and we agreed that no more movement was needed. My dentist removed the attachments from my teeth and we ordered the retainers, which I will need to continue to wear full-time for a few months and then beyond that just at night - in order to ensure things stay in place.

My thoughts

In general, the process was super easy. For the first few days of the treatment I didn’t think I would be able to keep it up for six months - the aligners felt pretty uncomfortable and can be a little painful for a couple of days every time I switched to a new set. Also, the extra work needed when brushing my teeth and having to remove the aligners between meals seemed inconvenient.

However, after a few weeks it all became second nature. It now feels weird when I don’t have them in!

The treatment is also quite expensive. However, it is cheaper (I think?) than traditional braces, it’s a shorter treatment period, and I preferred to have the almost-invisible aligners rather than metal braces in front of my teeth.

In addition, given that the sets of post-treatment retainers included in the treatment plan last for years, it feels like the treatment is a “one off” (🤞) - as long as I keep wearing the retainers properly then the teeth should now stay in place.

All in all, it was (and is still) a good experience and I am glad to have done it.

Is Facebook scraping the Fediverse?

Wed, 07 Apr 2021 19:44:00 +0000

I don’t use Facebook often. In fact, I only have an account currently because our company uses the “Login with Facebook” functionality in order to offer an additional single sign-on option for some customers.

I logged-in today as we needed to update some of the app’s configuration on the Facebook Developer portal, and I went via the Facebook homepage feed to get there. A couple of “Suggested for you” posts that showed near the top of my feed were unusual and caught my eye.

There wasn’t just one. As I scrolled further, more and more showed up - all seemingly from the same user.

The page (“Nostalgia Vienna”) doesn’t seem to be selling anything in these posts, and I’ve never interacted with them before. I also don’t have any content on Facebook and use browser plugins such as Firefox Containers, Privacy Badger, and others to try and prevent inadvertent data sharing with the social platform.

I know Facebook potentially has other ways of gathering user information, but I just simply don’t have a big interest in Viennese trams (or trams in general). I don’t really know why it is so keen to show me a new picture of a tram every few posts down the home feed.

I then realised that I recently posted a picture to Pixelfed of a tram that I took on a trip to Basel a few years back.

My Pixelfed account is not explicitly tied to my own name or identity, but my bio there does contain a link to my website.

Interestingly, the styles and images of the Viennese trams suggested by Facebook are not a million miles away from my own post of the Swiss tram. The link feels tenuous but I can’t think of anything else that might cause Facebook’s algorithm to so strongly suggest this type of content to me.

I just wonder whether there is some clever scraping going on behind the scenes to further bolster Facebook’s knowledge of its users.

From Apple Mail to Spark to Thunderbird

Sun, 04 Apr 2021 11:10:00 +0000

Like many people, I own and manage multiple email accounts - for example, some are for work, for home, or for specific projects. I used to be a strong user of solely web-based email clients (such as Gmail or Fastmail’s web apps) for each of my accounts. However the number of tabs I needed to keep open for all of this grew to the point where things became unmanageable - both in terms of needing to check multiple tabs several times per day and also frustrations when the browser would restart, or if I’d lose my tab setup for some other reason.

I needed a proper client, and although I knew that web-based software like Roundcube and Rainloop existed - which I could self-host - they just never felt stable or feature-ful enough.

This post is a short round-up of three mail clients I’ve been trying over the past few months.

Apple Mail

For several years I’ve been an Apple Mail user on both my Mac and iPhone - mainly because it was the default on the devices I have but also because it’s generally quite smooth and works reliably.

It’s relatively painless to get setup, and the accounts sync well across the mail, contacts, and calendar apps. On Gmail (and other larger providers) there is an authentication “wizard” to help get the accounts setup. Fastmail allows you to install profiles that automatically configure everything for you.

However, over time I began to find the interface a bit unintuitive. On iOS the general “Accounts” setting - which was useful as a single source of truth - seemed to disappear, and for some mailboxes it wouldn’t let me add alias send-from addresses. I’m sure there was a reason for this, but I sometimes find that the iOS settings UIs overcomplicate things in their efforts for simplicity.

Whilst the Mac (currently) still has a dedicated Accounts setting in System Preferences, it also had other problems. Several times a day I’d frustratingly get my workflow interrupted by warnings of network problems.

I still think Apple Mail is a pretty decent app, but I thought that there must be something else out there that would work better, and be less frustrating, for me.

Spark Mail

Back in February some of my colleagues recommended the Spark mail app from Readdle. I’ve used some of Readdle’s other software in the past (see this post, for example), and generally find it quite useful and intuitive. Like Apple Mail, it’s also available for Mac and iOS.

Spark is free to get started (and I imagine most individuals would fit into their free plan long-term too). One of the features I immediately liked was that all of your mail accounts are tied to a single account. That means that if you get a new computer or phone, you don’t need to go through the tedious business of setting up all the mail accounts again - just login with your main email and everything else gets pulled-through.

Email management is easy, search is lightning-fast, and the settings are useful.

Spark also comes bundled with a calendar that syncs well and automatically with services like Google Calendar and Fastmail Calendar. Like Apple Mail, there are dedicated setup wizards for email and calendar with the larger providers, and an option for manual entry for others. The calendar’s event creator is nice, and also allows you to automatically schedule a video meeting.

One drawback is that there doesn’t seem to be any way to view or manage contacts, and neither does it seem to integrate with the system contacts. I imagine it works directly with the relevant provider’s contacts service.

Another frustration I had was in managing shared calendars. I think I’m a bit of a calendar power-user, however I imagine this must be affecting other people too. If someone else - who also shares their calendar with you - creates an event and invites you to it, there does not seem to be any way to select your own entry in order to interact with it (e.g. to accept or decline the invitation).

In the event below, if my calendar was the “green” one, for example, there is no way for me to select that in order to accept or decline. Again, I may be missing something but I’ve been trying to find a way for a while now without needing to “hide” my colleagues’ calendars first.

Then it comes to security. Whilst I “trust” Readdle - in that I imagine they have decent security practices in place - we know that even the most secure companies can become compromised. The account sync feature mentioned earlier is certainly useful, however this must mean that Readdle are storing the Gmail access keys or IMAP connection details on their own servers in a centralised location. Your email is the last thing you want to get compromised - since it is likely that this controls a number of your other online accounts - and so this risk is a bit of a concern.

Readdle claim that everything is encrypted at various levels but it still feels a little risky to me. Having the sync and push notifications is useful, and so it’s up to the individual to choose what works best for them.

Thunderbird

The last client I want to mention in this post is Mozilla’s Thunderbird. This is a bit of a re-visit for me, since this is the client I used consistently during my University years.

In honesty, the client doesn’t seem to have changed a huge amount over the last decade, but then again - neither have the underlying email technologies themselves. It’s an open-source client available on a number of operating systems - but not yet (or ever?) for mobile.

Despite the slower development, I find Thunderbird to be a very powerful client. It has great support for email, calendar, and contacts straight out of the box. Things seem clearly organised, and account-management is super easy. There are no dedicated setups for Gmail, Outlook, etc., but it was able to automatically detect the relevant IMAP/SMTP servers for all of my accounts.

It’s very unopinionated about ordering, views, threading, and much more - which allows you to set things up the way that works best for you. The interface doesn’t try to be flashy or too clean and I find I am very productive when using it.

The calendar is easy to use and works with open standards like CalDAV.

It also has built in support for chat through systems like IRC and XMPP (if you use these types of things), and there’s also a rich ecosystem of plugins to add extra functionality too. It’s certainly the most flexible and powerful of the desktop mail apps I’ve used.

A few areas where it frustrates are around its performance. When adding a new account it proceeds to automatically download all of the mail headers for that account to be stored locally in its databases. This allows it to support searching and other local tasks, but the process causes the app to run slowly whilst it’s in progress. If you change computer often, or have several machines to setup, then this could be a pain.

When opening large mail folders containing perhaps several hundreds of thousands of messages - for example my combined “Archive” folder - things get very slow to the point where it is unusable. However, I don’t really ever need to use these views so this isn’t too much of a problem for me, but for some people this could be a blocker.

When compared to Apple Mail and Spark, the search function seems very slow. The results returned are quite accurate though, and the fact that results get shown in their own “tab” means that your flow isn’t interruped elsewhere in the app. This is a nice feature.

Generally, I love Thunderbird. Mozilla is renowned for being privacy-centric and the fact that everything is stored locally gives me more confidence about its security. Of course, it has drawbacks which will put some people off, but it’s good to be supporting open-source software where possible.

Conclusion

Like a web browser, I think people should be free to continue to try alternative mail clients as their needs and the software features change. The software mentioned above comprise just a small sample, and are focused largely around the Apple ecosystems since these are the devices I happen to be using at the moment.

Some others I’d like to try are Airmail and Polymail. However, it’d be great to get some feedback on what other people are using. If you have any suggestions then please get in touch using Matrix (@wilw:matrix.wilw.dev) or on Mastodon (fosstodon.org/@wilw).

The simplicity and flexibility of HTTP for APIs

Wed, 31 Mar 2021 22:00:00 +0000

Simple and RESTful HTTP APIs

The HTTP standard is an expressive system for network-based computer-computer interaction. It’s a relatively old standard - it started life as HTTP/1.0 in 1996 and the HTTP/1.1 standard was formally specified in 1999. HTTP/2 (2015) introduced efficiencies around how the data is transmitted between computers, and the still in-draft HTTP/3 builds further on these concepts.

I won’t go into the nuts and bolts of it, but - essentially - for most applications and APIs, the developer-facing concepts haven’t really changed since HTTP/1.1. By this version, we had all the useful methods required to build powerful and flexible APIs.

When writing a web service (e.g. a website or a web-based REST API), actions are based around resources. These are the “things” or “concepts” we are concerned with. For example, if one was to write a to-do list app, two of the concepts might be “to-do list” and “to-do list item”. Generally, such an app might also maintain user accounts and so may have “user” and “session” resources, too, along with others if required.

In such a service, resources are usually indicated by a path. This is the bit that comes after the host name (e.g. example.com) in the URL and are usually mentioned in plural.

For example, in our to-do list example, a resource which indicates all available lists might be given simply by /lists, and a specific list with ID aaa-bbb-ccc would be available at /lists/aaa-bbb-ccc.

This system allows the engineer to indicate hierarchy or ownership in the data model. For example, to address all of the list items in a specific list one might use /lists/aaa-bbb-ccc/items. Then to access an item with ID xxx-yyy-zzz inside this list you’d use /lists/aaa-bbb-ccc/items/xxx-yyy-zzz. In many cases, for a simple web service of this type, this would be sufficient - it may not be appropriate to enable addressing a to-do list item directly without the context of its “parent” list.

Paths may sometimes include other metadata, such as the API version being called, but this can be simply included and described in the documentation.

In HTTP, methods describe what the API consumer wants to do with a resource. Some of the most widely used methods are GET, POST, PUT, DELETE, and OPTIONS. These methods are defined in the spec and some clients may handle requests differently based on the method being used. Unlike resources, you can’t define your own methods to use. However, the flexibility provided as-is allows for most services to be built without requiring this level of customisation.

Some of these have pretty obvious semantic meaning. POST is typically used to create a new resource (e.g. “post a new to-do list item”) and PUT is used to update an existing resource.

This means that, given the combination of our resource addressing and methods, we can express a powerful web service. Structuring your to-do list app using the system described here caters well to typical to-do list actions: creating lists and items (e.g. POST /lists/aaa-bbb-ccc/items), crossing-off items (probably PUT /lists/aaa-bbb-ccc/xxx-yyy-zzz), and retrieving, updating, and deleting things in a similar way using the appropriate methods.

HTTP request headers can be used to provide authentication information, describe how the client wants information to be returned in the response, along with other ways to further annotate the request being made and to customise the expected response. Of course, the effectiveness of supplying these request headers depends on the server’s own capability and configuration. However, the use of headers should certainly be considered by the engineer whilst planning and building out the service.

Using standards like these - resources, methods, and headers - in your APIs enables your users (consumers) to more easily learn and understand how to use your service. This saves them time, helps your service to grow, and means you’ll spend less time dealing with support requests (unless your documentation is really good).

Custom implementations

I think the above system is the most ideal, expressive, learnable and expected way of building web services.

However, HTTP is flexible, and your server-side code can - in theory - do whatever you want it to, no matter what the request path, method, and headers are. But I don’t really understand why one would want to.

I recently migrated my photos collection to pCloud, and wanted to explore their API to see if I could also use the service for programmatically backing-up other things, too.

Unfortunately I am unable to actually use their API, since I use two-factor authentication on pCloud and the API doesn’t seem to work if this extra layer of security is in-place. However, whilst researching I discovered that pCloud’s API is an example of a service that seems to defy the standards one is usually familiar with.

For example, it appears that it’s perfectly acceptable to use POST https://api.pcloud.com/deletefile?fileid=myfile to delete a file or GET https://api.pcloud.com/renamefolder?path=oldfolder&topath=newfolder to rename a folder.

There’s nothing technically wrong with this implementation, especially given the fact that I’m sure it works. It perhaps makes it easier to route requests through to the correct internal functions. However it just feels inelegant to me, and it seems to focus more on what’s easier for them rather than their users.

The page that lists file operations could instead show a couple of simple example paths and then rely on request methods and parameters to describe available options.

I don’t mean to pick on pCloud - the service itself is great and I’m sure the API works nicely. I plan to continue using the service via its web UI and official clients. I only bring it up because it seems odd to re-invent the wheel.

I’m completely on-board with the notion of discouraging system and process monopoly, but I don’t think this is the same thing. The web is formed from a set of open standards that anyone can comment on or help contribute to.

“Good” implementation examples

The web is full of services that expose sensible and learnable APIs.

An example I always love is the Stripe API - arguably a much more complex service than pCloud. However its simple “compliant” API makes credit card payments - and loads more - very easy to integrate with.

The Spotify web API also looks useful, though I haven’t used that before myself.

Beyond REST

REST has been a cornerstone of the web over the past couple of decades, and I think there is still very much a space for it - both for now and in the near future. Its flexibility has allowed it to remain useful across industries and settings - from small private IoT setups through to highly-secure enterprise-enterprise systems.

There are movements to begin using other technologies that may be better suited to the future of the web and communication - particularly as things continue to scale. Efforts such as GraphQL, Netflix’s Falcor project, and even RPC provide alternatives for when REST isn’t the most appropriate solution.

However, if building a web API that you want other people to use, and which would be well suited to REST, then I always think it’s worth sticking to these HTTP standards as much as possible.

PinePhone and PineTime

Sat, 27 Mar 2021 16:31:00 +0000

Pre-ordering the PinePhone Beta

Earlier this week I ordered a PinePhone, which recently became available as a Beta Edition.

I’ve been excitedly following the progress of the PinePhone for some time now. I’ve joined various Matrix rooms, subscribed to blogs, and started listening to the PineTalk podcast. The phone is a hackable device that runs plain old Linux - not an Android variant - and thus helps users escape from the grasp of the Google and Apple ecosystems.

Other similar devices exist - such as the Librem 5 from Purism - however the unopinionated nature of the PinePhone, and its cost ($150 compared to the Librem’s $800), make the Pine64 offering much more attractive to me.

I understand that the phone and software are still under very active development, and I fully expect that the phone is not yet ready to become a daily driver. However I am excited to try it out, support the project, and contribute where I can. The potential of this movement is huge.

Some thoughts on PineTime

Whilst researching the PinePhone, I stumbled across the PineTime smartwatch. This is a wearable device also from Pine64, which aims to offer an open-source and hackable system in a similar vein to the PinePhone.

Pine64 offers the device for purchase but fully acknowledges that it is not yet ready for daily use, and encourages interested people to instead purchase the Development Kit so that they can learn more or contribute to the project.

The device aims to offer health tracking solutions (since it includes a step counter and heart rate detector) and notifications, and so the intention is for it to offer a similar experience to other smart watches - except with much more freedom.

The open and community-driven nature of the device could take it any number of ways.

We envision the PineTime as a companion for not only your PinePhone but also for your favorite devices — any phone, tablet, or even PC - pine64.org/pinetime

This vision seems to embody the Pine64 philosophy that we see across all of their products. I’m not the right person to be able to contrubute much to the project in its current stage (I don’t have much experience with developing on embedded operating systems), but I look forward to seeing how it progresses and hopefully getting more involved slightly further down the line.

The Great Alone by Kristin Hannah

Tue, 23 Mar 2021 19:45:00 +0000

The Great Alone by Kristin Hannah is a book set out in the Alaskan wild. It tells the story of a young family that move in order to live off-the-grid after the father returns from being a prisoner of war in the Vietnam war.

The book mostly focuses on the viewpoint of the daughter, Leni, who is thirteen years old when she moves with her mother and father. The story tells how Leni adapts and grows into her new Alaskan life over the years, whilst at the same time trying to navigate some of the perils at home in her family cabin. Leni and her family meet and grow close to different members of the local community, in which there are a variety of views regarding the types of people that should be allowed to come to Alaska.

The book certainly has its dark moments, and there is an ongoing sense of violence and intensity. At the same time, the author wonderfully describes the peacefulness of the environment, and the wildness of the Alaskan landscape, the wildlife, the weather, the sky, and the sea. It is clearly a place where humans and nature meet, and a place where - if people are to live off the land - they must learn and respect it and all it has to offer.

After all, in Alaska you can only ever make one mistake. The second one will kill you.

I loved the book and its intertwining themes of love, family drama (and more), forgiveness, wilderness, comradeship, and escapism. The author makes you feel frustrated with some of the decisions made by the characters in one moment, and the next you are cheering them on from behind the pages.

With everything that goes on in the story - the town and its community of interesting characters - it isn’t always obvious where the title of the book comes from. However, as you progress further you realise that it’s not just the landscape and geography that can evoke loneliness; the feeling can be more the result of the actions of others and having to keep secrets about what goes on behind closed doors.

Running your own Matrix homeserver

Mon, 22 Mar 2021 11:50:00 +0000

Why use decentralised communication services

Centralised communication services, such as Telegram, Signal, and Whatsapp, offer convenient means to chat to friends and family using your personal devices. However these services also come with a number of pitfalls that are worth considering. For example;

Many of these services are linked to your phone number, which can affect your privacy.
They can be invasive with your contacts (“Jane Doe is now using Telegram!”).
They usually require you to use proprietary client software. If your OS/platform isn’t supported then you can’t use that service.
They typically require that everyone using the service has to use the same client software.
They can be unreliable (Whatsapp frequently has downtime).
They are invasive and collect data about you (particularly Whatsapp). If you don’t pay for the service, then you are the product.
Even though Signal is encrypted end-to-end, its servers are based in the US and are subject to the laws there. Also their open-source server-side software appears to not have been updated for some time.

There are, of course, other factors on both sides that you may want to consider. It can be hard to move away from these services - after all, there’s no point using a system that no-one else you need to talk to uses.

However, for some people, being able to avoid these issues can be important. One way to do so is to participate in a (preferably open-source) decentralised communication service in which the entire network is not owned by a single entity and where data colleciton is not the business model. This also helps prevent unstability and downtime, since there is not a single point of failure.

This is analogous to using services such as Mastodon and Pixelfed over Twitter and Instagram, respectively - the underlying software is open-source and anyone can host an “instance”. In these cases, each instance can communicate with others using the ActivityPub protocol. In this post I will talk about another protocol that offers decentralised and federated encrypted communication.

The Matrix protocol

The Matrix protocol is one example of a standard for real-time decentralised communication. Since the standard is open, anyone can build server and client software that enables end-to-end encrypted communication between two or more people. Another example of a similar protocol is XMPP, which is also very popular and has been around (in its earlier forms) since 1999.

When using Matrix, you belong to a “homeserver”. This is where your messages and some account details are stored. However, since Matrix is a federated protocol, you can use your account to communicate with others on your homeserver as well as people from other homeservers that federate with yours.

The standard was introduced back in 2014, and by now there is an established ecosystem of software available for use. In fact, you can use Element on your device and get started by joining an existing homeserver right now.

Additionally, if you don’t want the hassle of self-hosting yet another service, then Element also provides plans that allow you to run your own homeserver on managed hosting.

Self-hosting a Matrix homeserver

If you want more control over your data, you may opt to self-host your own homeserver that implements the Matrix standard. Even if you self-host you can still take advantage of the protocol’s federation features and communicate with people on other homeservers.

The resource requirement for Matrix servers is a bit on the heavier side (especially when compared to the lighter XMPP servers). However if you already run a small-ish VPS anyway (as I do for things like Nextcloud), and if you only expect one or two people to be enrolled directly on your homeserver, then you can certainly host Matrix on that same VPS without too much trouble. For reference, I have a single $10 server from Linode, which happily runs Matrix alongside a number of other services.

The Synapse project is probably one of the most robust and feature-complete homeserver implementations, and is the one I’ll talk about in this post. They also offer an officially supported Docker image, which is what I would recommend using to keep things in one place.

Homeserver name

Firstly, I’d recommend setting up a domain (either an existing one or a new one) and then updating your DNS such that the relevant entry points to your server.

It is important to think about the domain name you choose for your homeserver, since this cannot be changed later. Matrix recommends using your root domain name itself rather than a subdomain for your homeserver name. However if you already host a website using your full domain name you will need some extra configuration to make it work properly. I personally don’t, as I wanted an easier setup!

Exposing ports and preparing TLS certificates

In order to configure HTTPS, I’d recommend setting up an Nginx container or server as a reverse proxy and issuing certificates using Let’s Encrypt. The Matrix protocol uses standard port 443 for communication with clients (e.g. from an app) - known as the “client port” - and port 8448 for communication with other homeservers (the “federation port”).

You may wish to read some of the official documentation on setting up a reverse-proxy, but I’ll run through roughly what I do below.

Depending on your Nginx setup, you may need a couple of server blocks similar to the following to configure your reverse proxy (assuming your homeserver name is “example.com”):

server {
  listen      80;
  listen      [::]:80;
  server_name example.com;
  return 301  https://$host$request_uri;
} 

server {
  listen      443 ssl;
  listen      [::]:443 ssl;
  listen      8448 ssl;
  listen      [::]:8448 ssl;

  server_name example.com;

  ssl_certificate     /path/to/fullchain.pem;
  ssl_certificate_key /path/to/privkey.pem; 

  location / {
    proxy_pass http://synapse:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $host;
    client_max_body_size 50M;
  }
}

If you run Nginx as a Docker container remember also to expose port 8448 alongside 443.

Synapse uses port 8008 for HTTP communication, to which we forward requests received on both the secure client and federation ports. In the example above, synapse is the name of the container that runs my homeserver, as we’ll cover next. Again, depending on your setup, and whether you choose to use Docker, you may need to change this value so that your reverse proxy can route through to port 8008 on your homeserver.

Generate a configuration file

The next step is to generate your homesever config file. I recommend firstly creating a directory to hold your synapse data (e.g. mkdir synapse_data). We’ll mount this to /data on the target container in order for the configuration file to be created.

The configuration file can be generated using Docker:

docker run -it --rm \
  -v synapse_data:/data
  -e SYNAPSE_SERVER_NAME=example.com \
  -e SYNAPSE_REPORT_STATS=yes \
  matrixdotorg/synapse:latest generate

Once this completes, your synapse_data directory should contain a homeserver.yaml file. Feel free to read through this and check out the documentation for ways in which it can be modified.

Run the homeserver

Finally, we can now run the homeserver. Depending on your reverse proxy setup (and whether you are containerising anything else), you may need to configure your Docker networks, but generally you can just execute the following to get your homeserver running:

docker run -d \
  -v synapse_data:/data
  --name synapse \
  --restart always \
  matrixdotorg/synapse:latest

If everything went well (and assuming your reverse proxy is also now up and running), you should be able to use your web browser to visit your Matrix domain (we used “example.com” above) and see a page that looks like this:

Creating your user account

As long as your homeserver is configured to accept user registrations (via the enable_registration directive in homeserver.yaml), you should be able to download a client (or use the Element webapp) and register your first user account.

Once logged-in you can join rooms, invite people, and begin communicating with others.

Conclusion

This post aims to be a rough introduction to running your own Matrix homeserver. The Synapse software offers a variety of ways to tailor your instance, and so it is certainly worth becoming familiar with some of the documentation to ensure you have configured things the way you need.

If you want to get in touch then you can send me a message using Matrix (@wilw:matrix.wilw.dev) or on Mastodon (@wilw@fosstodon.org).

Blood, Sweat, and Pixels by Jason Schreier

Wed, 17 Mar 2021 19:23:00 +0000

This post contains some of my thoughts on the book Blood, Sweat, and Pixels by Jason Schreier.

This book contains a number of stories about how some of the most well-known (and other less well-known) video games are made. The book’s subtitle, “The Triumphant, Turbulent Stories Behind How Video Games Are Made”, sums it up pretty well.

Working in the software industry myself, I often hear about the notion of “crunch time”, which is a term we’ve borrowed from the game devleopment industry at times when critical updates, fixes, or deadlines are pressing. However, after reflecting on the stories in this book, it makes me realise that the “crunches” we suffer are nothing to the crunch and stresses experienced by game developers in many small teams and large development studios alike.

Every chapter explains in detail the pain and reward faced by game developers and management teams on an ongoing basis. The developer skill and expertise required by game studios, and the time and size of the required resource, helps to explain the huge financial impact these projects have.

It’s no wonder why such harsh deadlines are set. In many cases it’s a matter of “life or death”: either the game gets released on time or there is no game at all and everyone has to lose their job - even in large well-funded companies.

I loved the stories of the groups of developers that ended up leaving their well-paid (but stressful) jobs in order to start something by themselves as a smaller group - not quite realising at the start what they were letting themselves in for.

I enjoyed the story behind the development of the game Stardew Valley. This is a game I love and have played for hours on my Switch - not knowing really (or fully appreciating) where the game came from and all the time spent by its solo developer and the stress that went on behind the scenes.

The background to the development of The Witcher 3 was also fascinating; how the relatively small but super-ambitious studio CD Projekt Red successfully brought to the world stage the Polish much-loved fantasy world.

The book was great, and well-narrated by Ray Chase (I listened to the Audible version). I only wish there were more stories (it only took a few days to get through), but I appreciate the effort the author went into with researching and interviewing some of the key people involved. It is an excellent insight into how parts of the game industry work.

The Tildeverse

Mon, 15 Mar 2021 11:05:00 +0000

The last twenty years of internet evolution

Although I was still somewhere between being of single-digit age and a young teen back in the ’90s and early ’00s, I still fondly remember discovering and becoming a small part of the flourishing community of personal, themed, and hobby websites that connected the web.

We were even given basic server space in school and the wider internet was thriving with GeoCities and communities grew around services like Neopets. Everyday, after school, we’d go home and continue our playground conversations over MSN Messenger (after waiting for the dial-up modem to complete its connection, of course). The internet felt small and personal (even if you didn’t use your real name or identity) and exciting.

For those more tech-aware than I during those days there were also the established BBS systems, IRC (which is still very much in active use), and several other types of available internet and communication services.

Over the years since then we’ve obviously seen the introduction and growth of tech companies, which have exploded into nearly every corner of the internet. Some have come and gone, but many are here still and continue to grow. We’re now at a point where many of these services are almost a full “internet” (as it was back in the day) by themselves: on Facebook you can host a page for yourself or your business, you can engage with any nunber of other apps through your Facebook account, you can chat in real-time with individuals or groups of friends and family, and much more.

In the developing world, many people see the internet and Facebook as being entirely analogous such that new mobile handsets are sold with the app pre-installed on their device and cellular carriers sometimes provide free access to the platform as part of their data plan.

This boom (invasion?) has completely changed the way the internet works for day-to-day users. Although these companies and their huge marketing teams have facilitated the growth of adoption of technology for community and communication, it has come at a cost. When using these services, the internet no longer feels personal and exciting.

For many people - particularly those who grew up with this state of the world or those who never fully engaged before Web 2.0 - this is fine and not a problem. They would likely laugh at the simplicity and “slowness” of the “old internet” compared to the flashy, speedy and engaging platforms they are used to interacting with for several hours every day.

Community through the Tildeverse

However, there are also many of us who miss the quality and meaningfulness of the smaller and slower web. Since joining Mastodon a couple of years back, it’s been great to be part of a movement that actively encourages the growth and maintenance of personal websites, blogs, distributed systems, and the self-hosted services that help promote these ideologies.

Movements and concepts such as the Small Web, the Indie Web, and even initiaives like Project Gemini have all helped to raise awareness around the fact that there is still a large number of people interested in promoting the ideas around the slow web, and building a real sense of community.

Also part of this movement is the notion of the Tildeverse. The Tildeverse draws some inspiration from PubNix and stems from building community through “belonging” - similar to how one might feel when interacting with the Fediverse.

The Tildeverse is an opportunity for people to donate server resources by provisioning and managing a *nix system (e.g. Linux, BSD, or similar), on which members of that tilde community can have a user account that they can access using programs such as SSH.

The name is derived from the fact that the tilde symbol (~) is used to denote a user’s home directory on UNIX-like systems that offer multiuser functionality (e.g. ~will). On such servers, users can use their account and home directory to publish a website, a Gemini capsule, use tools to chat with other members via IRC or internal mail, or take advantage of any number of other services the server administrators may offer.

To join, it is recommended to first identify a community you feel you can contribute positively towards. Many servers don’t require payment to join (although there are often options to make donations to help contribute towards the running costs), but it is usually expected that you help foster the sense of community by actively engaging with others, posting interesting or useful content, or by abiding by other “rules” that may be in place.

If you have found a community you’d like to join, a typical registration is often achieved by emailing the server administrators with your desired username and an SSH public key. If and when your registration is accepted, you can then use the corresponding private key to login and begin to engage with the community.

Many such communities, such as tilde.club, list some of the users’ home directories as webpages. This lets you get an idea of the community before choosing to join. Many homepages (though this isn’t limited to the Tildeverse) include a webring, which you can use to navigate to other user websites belonging to the same webring.

Others, such as tanelorn.city, are more focused on publishing Gemini content if this is more interesting to you.

Either way, I’d recommend browsing from tildeverse.org as a starting point if you’re interested in getting involved. It helps explain some of the concepts and lists some of the Tildeverse member servers.

The Hunt for Red October by Tom Clancy

Wed, 10 Mar 2021 20:18:00 +0000

I recently finished reading The Hunt for Red October by Tom Clancy.

This genre of novel (sort of military thriller fiction) is not usual for me and this is the first Clancy book I have read. That being said, the book has been on my “to-read” list for a fair amount of time and so I am glad I got round to reading it.

I also hadn’t seen the movie (starring Sean Connery and Alec Baldwin) by the time I read it and so I didn’t have any pre-perceived ideas about the story and could read afresh.

Side note: I have now since watched the movie, and whilst the core plot is mostly the same there are a lot of differing details throughout (in terms of both angle and storyline), and so I can certainly recommend both media if you’ve previously seen one or the other or neither.

In general, I very much enjoyed the book. It was an exciting read from start to finish, with interesting characters, relationships and story arcs. I was fascinated by all of the technical detail and also felt that it helped explain and justify many of the core concepts and features of the story. The character development was good, and you quickly build a connection with many of the different people involved.

Though I do not think this a fault of the author (I imagine the work is an accurate reflection given the time of the setting), I would hope that if it were written in modern times there would be improved gender diversity and more female representation in the novel - as it is I do not remember there being a single female character (aside from mentioning wives and family members who do not appear in the story directly).

Either way, I can certainly recommend the book to others who also enjoy an exciting story and lots of technical detail. I thought the run-up to the ending was great and I am definitely intrigued to further my reading in this genre.

Thoughts on minimalism, and what happens when I get mail

Mon, 08 Mar 2021 19:23:00 +0000

Minimising possessions

Like many people I these days try and live a minimal life when it comes to possessions. Having more stuff means there is a greater level of responsibility required to look after it. I love the principles involved in “owning less”.

Although I am in a very different situation to Pieter Levels, I find the ideas behind his 100 Thing Challenge (and other related pieces) to be inspiring.

Although my home contains items that are technically mine - furniture, kitchenware, decorations, etc. - I consider these as belonging to the house itself rather than as my personal belongings. Personal items are essentially the things I can fit into my backpack and are things I actually need on a daily or weekly basis: my laptop, my phone, some of my clothes, my toothbrush, passport, and a few other smaller items.

Non-essential things - although a “luxury” - are also a liability (and an anchor).

This also helps to keep emotional attachment out of ownership. I know that if I were to lose or break my phone, I could get another and continue on as before. The main concepts here for me are portability and replaceability.

I consider my data and communications to be personal belongings, too. For example, emails I’ve sent and received, documents, images, and so on. Since these are all digital, I can just stick them on my Nextcloud (or pCloud), and I can access them any time through my phone or laptop.

I also strive for digital minimalism too, where possible. However, since this data storage methodology is scalable and I can keep things very organised I don’t mind holding onto data and documents should they be useful in the future. Even with thousands of stored documents, the collection is still portable and it fits with my model.

Mail and paperwork

Many of the world’s organisations - including insurance companies, banks, lawyers, and public services - still love doing business with physical documents and through physical mail. Also, these are typically the types of documents you are supposed to keep hold of for long periods of time for the purposes of financial records, insurance certification, and so on. Over time this paperwork builds up and quickly becomes disorganised.

Some people keep boxes or filing cabinets of documents and mail. This turns into something else to be responsible for. It’s not portable (in the “backpack” idea mentioned earlier) or replaceable. If there was a fire it would be lost, and if moving home it’s something else to “worry” about.

Until a couple of years ago, I kept documents in ring-binders. My process would include holepunching documents (retro, I know), finding the section of the ringbinder most appropriate for that document, placing the document, and then putting the ringbinders back on the shelf.

I had years’ worth of utility bills, insurance documents, bank statements, pay-slips, and more that I would need to bring with me whenever I moved and always ensure there was a phycial space for them in my life somewhere.

I began to realise that - for the vast majority of these documents - I would never really need the original version. Apart from things like my passport and paper certificates containing security features, document copies would be fine. And since I already had a system for storing digital documents, I could extend this to maintain a more organised (and searcahble) collection of digitised paper documents too.

Digitising paperwork

Phone cameras these days are more than capable of creating high-quality digital replicas of paper documents. There are also many scanner apps and software available to make this easier.

I personally use Scanner Pro on my iPhone, which is very useful. It automatically detects paper edges (even documents with weird dimensions) and straightens the image sensibly too. It also has settings to help configure further; for example, I only need greyscale copies and not the highest resolution - both of these factors help decrease the size of the eventual file.

The official iOS Files app also has a “Scan Documents” feature, which looks pretty good. I’ve not used this extensively myself yet.

After downloading the scanner app, I went through my ring-binders and piled up all the documents to throw out - stuff I just didn’t need any record of but had, for some reason, kept anyway. I then went through each remaining section in turn and scanned each document in - storing each PDF to my Nextcloud.

The process was surprisingly quick and by the end I had a nicely organised collection of files on Nextcloud and a large pile of paper documents I could throw out. As I mentioned earlier, about the only physical things I did keep were certificates, my passport, and a handful of other items.

It was a weirdly therapeutic exercise!

My process now

Jumping back to the present and my more minimalism-focused self, I am now very strict about what paperwork I keep. In fact, I don’t think I’ve kept hold of a physical document that I’ve received in the last year (and probably longer).

I have a simple process:

I receive the document/paperwork and open it;
I use my phone to scan the document;
I sync the file to an 0 Unfiled directory on my Nextcloud, titled by date, sender, and short subject (e.g. 2021-03-02_BritishGas_Statement.pdf);
I throw the document out (shredding first if sensitive);
If the paperwork requires action, I either do so immediately or set a reminder to do so;
Once a month or so I go through my 0 Unfiled directory and categorise properly according to my personal filesystem.

I use a “holding” directory (0 Unfiled) to make the process quicker (for example, if there are several documents to scan) and it ensures I have actually actioned the files once I come round to organising them later. I use a 0 at the start of the directory name so that it sits at the top of my filesystem root in order to improve efficiency (and I try and use the Johnny.Decimal concepts as much as possible).

I also use the holding directory for other important documents - such as email attachments I want to include in this system. To me, it doesn’t matter which medium was used to receive the document: it’s all just data to be categorised and stored.

It’s a satisfying process. I now feel more organised, I can easily find a particular document - even from several years ago - without needing to trawl through piles of paper; I can ensure longevity and integrity of the data (i.e. it can’t get torn or damaged); I can back the collection up with added redundancy; and I can easily view and share the documents from anywhere.

If you currently keep lots of paper records and are interested in minimising your physical footprint then I can recommend trying a similar process yourself.

Easily set up discoverable RSS feeds on a Gatsby website

Thu, 04 Mar 2021 22:17:00 +0000

RSS has had a bit of a resurgence for personal websites and blogs in recent years, especially with the growing adoption of Small Web and IndieWeb ideologies.

Many static site generators - including Hugo, Jekyll, and Eleventy - can easily support the automatic generation of RSS feeds at build time (either directly, or through plugins).

The same is true for Gatsby - the framework currently used to build this static website - and the good news is that setting up one feed, or multiple ones for different categories, only takes a few minutes.

Your Gatsby blog structure

This article talks about RSS feeds for blogs (a typical use-case), but is also relevant for other notes, podcasts, or anything else that is published periodically to your Gatsby site.

In Gatsby, the typical blog set-up involves the blog entries in markdown format, and a template “page”, which is used to render the markdown blog posts.

You’ll also probably have a “blog” page which lists or paginates your posts for visitors to find them, and a createPages function in your gatsby-node.js that generates the pages from the template and markdown.

All this sounds way more complicated than it is in practice, and there are lots of guides available to help set this up.

At the very least, this article assumes you have blog posts written in a directory containing markdown for each post similar to the following:

---
date: "2021-03-04T22:17:00Z"
title: "Easily set up discoverable RSS feeds on a Gatsby website"
description: "How to set up multiple discoverable RSS feeds for your static Gatsby website."
tags: [100daystooffload, technology, javascript]
---

The post content starts here...

The metadata (frontmatter) doesn’t need to be exactly as shown, but having useful metadata (e.g. tags) in-place helps make your feeds richer.

Creating your feeds

To create the feeds, we’ll use a Gatsby plugin called gatsby-plugin-feed, which will do most of the heavy-lifting for us (as long as you have a blog in place structured similarly to the way described above).

First off, add the plugin as a dependency: yarn add gatsby-plugin-feed. I also recommend installing moment to help with formatting dates for the feed (as we’ll see later): yarn add moment.

Next, you’ll need to create some code in gatsby-config.js. If you have a blog already then you likely already have content in this file (e.g. gatsby-source-filesystem configuration). Your file probably looks a little like the following:

module.exports = {
  siteMetadata: {
    title: 'My Cool Website',
    siteUrl: 'https://my.cool.website',
  },
  plugins: [
    {
      resolve: 'gatsby-source-filesystem',
      options: { ... },
    },
    'gatsby-plugin-react-helmet',
  ],
};

Along with any other plugins you may have.

To create the feed we’ll make use of a GraphQL query, and a function which will create a feed object. If we define these separately (as below), it will give us more flexibility later.

In the same file (gatsby-config.js), at the top, first require the moment library we installed earlier, define the query we’ll use, and a function to create a feed object:

const moment = require('moment');

// Query for all blog posts ordered by filename (i.e. date) descending
const rssPostQuery = `
{
  allMarkdownRemark(
    sort: { order: DESC, fields: [fileAbsolutePath] },
    filter: { fields: { slug: { regex: "/blog/" } } }
  ) {
    edges {
      node {
        html
        fields { slug }
        frontmatter {
          title
          description
          date
          tags
        }
      }
    }
  }
}
`;

const createRssPost = (edge, site) => {
  const { node } = edge;
  const { slug } = node.fields;
  return Object.assign({}, edge.node.frontmatter, {
    description: edge.node.description,
    date: moment.utc(`${node.frontmatter.date}`, 'YYYY/MM/DDTHH:mmZ').format(),
    url: site.siteMetadata.siteUrl + slug,
    guid: site.siteMetadata.siteUrl + slug,
    custom_elements: [{ "content:encoded": edge.node.html }],
  });;
};

The rssPostQuery assumes your blog posts are rendered at /blog/filename in your built site. If not, then just change this value in the regex. Likewise, the createRssPost function assumes the dates in the frontmatter of your posts are formatted like YYYY/MM/DDTHH:mmZ - if not, just change this string to match your own format (I use UTC here as we’re dealing with global audiences!).

Essentially, the GraphQL query string returns all markdown files ordered by descending filename (I title my blog posts by date, so this gives a reverse chronological ordering of posts, with the newest first), and gives us the post content, slug (“path”), and selected fields from the posts’ frontmatters.

We use a regex in the query to discern between different types of markdown files. For example, you may have a collection of notes - also written in markdown - which we want to ignore for the purposes of creating an RSS feed for just blog posts.

The createRssPost function (which we’ll call later), accepts a markdown file (edge) and information about the website (site), and returns a fresh object representing this information to be eventually embedded in the feed.

The guid field is a globally-unique ID for this post on your blog and reader software will use this to, for example, determine if the user has already seen the post and should mark it as “read”. Since all of my posts have a unique path (“slug”), I just use this for the ID.

Finally, we need to add a section to our plugins array to tell gatsby-plugin-feed how to build our feed using the query and function we created above. In the same file, make the following changes:

module.exports = {
  siteMetadata: { ... }, // omitted for brevity
  plugins: [
    {
      resolve: 'gatsby-source-filesystem',
      options: { ... }, // omitted for brevity
    },
    { // Add this object to your "plugins" array:
      resolve: 'gatsby-plugin-feed',
      options: {
        feeds: [
          {
            serialize: ({ query: { site, allMarkdownRemark } }) =>
              allMarkdownRemark.edges.map(e => createRssPost(e, site)),
            query: rssPostQuery,
            output: '/rss.xml;,
            title: 'My Cool Blog',
            description: 'All of my blog posts'
          },
        ],
      },
    },
    ...
  ],
};

The gatsby-plugin-feed plugin only runs when the site is actually built. If you have your Gatsby site running locally, just run gatsby build in a separate Terminal window and then navigate to /rss.xml on your local development website to view the feed.

Creating multiple feeds

The example configuration in the previous section creates a single feed containing all blog posts.

However, you may have noticed that the feeds attribute is an array; this means that the plugin can be used to create multiple feeds. I do exactly that on this website: I have different feeds for different audiences (e.g. for technology, life, books, etc.).

Since we’ve already broken our code out into a separate query and function, it is easy to add new feeds by filtering on the markdown edges before passing them to map in the serialize function.

If you modify the same file again (gatsby-config.js), you can create a feed for all of your posts that contain a tag named “technology” as follows:

  ... // omitted for brevity
    {
      resolve: 'gatsby-plugin-feed',
      options: {
        feeds: [
          { ... }, // omitted for brevity
          {
            serialize: ({ query: { site, allMarkdownRemark } }) =>
              allMarkdownRemark.edges.filter(e => {
                const tags = e.node.frontmatter.tags;
                return tags && tags.length > 0 && tags.indexOf('technology') > -1;
            }).map(e => createRssPost(e, site)),
            query: rssPostQuery,
            output: '/technology.xml',
            title: 'My Technology Blog',
            description: 'Posts in my blog tagged with "technology".'
          },
        ],
      },
    },
  ...

This will create a new feed at /technology.xml containing these tech posts.

Since it’s just plain old JavaScript, you can use any of the available information to craft a number of flexible feeds for your visitors to subscribe to. You can then list these feeds on a page on your site, like this one.

Feed discovery

The gatsby-plugin-feed plugin has one more trick up its sleeve: without any extra work it will automatically inject the relevant tags to your site’s HTML at build-time to list the feeds that you have configured.

This means that your visitors just need to add your site’s root URL (e.g. “https://my.cool.website”) into their feed reader and it will suggest the available feeds to them.

The image above shows the Reeder macOS app automatically listing the available feeds on my website after entering just the root URL for the site. Visitors can then just add the ones they want.

Making your Python Flask app serverless

Sun, 28 Feb 2021 22:05:00 +0000

Python’s Flask framework is an easy and excellent tool for writing web applications. Its in-built features and ecosystem of supporting packages let you create extensible web APIs, handle data and form submissions, render HTML, handle websockets, set-up secure account-management, and much more.

It’s no wonder the framework is used by individuals, small teams and all the way through to large enterprise applications. A very simple, yet still viable, Flask app with a couple of endpoints looks as follows.

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
  return 'Hello, World!'

@app.route('/')
def greet(name):
  return 'Hello, ' + name

Flask apps like this can easily be deployed to a server (e.g. a VPS) or to an app-delivery service (e.g. Heroku, AWS Elastic Beanstalk, and Digital Ocean App Platform). In these scenarios, the server/provider often charges the developer for each hour the app is running. Additionally, as traffic increases or reduces, the provider can automatically scale up and down the resources powering your app in order to meet demand. However this scaling can sometimes be a slow process and also means that the developer is charged even when the app is not being used.

If you want to set-up your app so that it can automatically scale from 0 to thousands of concurrent users almost instantly, where you are not charged when users aren’t using your app, where it is highly-available (keep up your uptime to meet SLAs), and where there is no server set-up or maintenance required (and there is nothing for bad actors to try and SSH into), then migrating to a more serverless architecture might be of interest to you.

Also, given that most providers offer a pretty generous free tier for serverless apps, you may not end up paying much at all (up to a few dollars max a month) until you start generating enough traffic.

Note: in this article I use Flask as an example, however the same should apply to any WSGI-compatible framework, such as Bottle and Django, too.

What is a serverless web app?

“Serverless” is the generic term for a family of cloud-based execution models where the developer does not need to worry about provisioning, managing, and maintaining the servers that run their application code. Instead, the developer can focus on writing the application and can rely on the cloud provider to provision the needed resources and ensure the application is kept highly-available.

Although services such as Heroku and Digital Ocean App Platform can be considered “serverless” too (in that there is no server to configure by the developer), I refer more to delivery via function as a service as the particular serverless model of interest in this article, since this offers the benefits listed at the end of the previous section.

“Function as a service” (FaaS) - as its name suggests - involves writing functions, which are deployed to a FaaS provider and can then be invoked. Such systems are event-driven, in that the functions are called as a result of a particular event occurring - such as on a periodic schedule (e.g. a cron job) or, in the web application case, an HTTP request.

There are many FaaS providers, such as Azure Functions, Google Cloud Functions, Cloudflare Workers, and IBM Cloud Functions.

Probably the most famous (and first major) FaaS provider offering is AWS Lambda. In this article I will focus on using Lambda as the tool for deploying Flask apps, but many of the concepts discussed are generic across providers.

Serverless apps written using AWS Lambda usually also involve Amazon API Gateway, which handles the HTTP request/response side of things and passes the information through as code to the Lambda function. The event argument received by the function describes - among other things - the information about the request that can be used to generate an appropriate response, which is then returned by the function.

import json
        
def lambda_handler(event, context):
  name = event['queryStringParameters']['name']
  return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json"},
    "body": json.dumps({"Hello": name})
  }

As long as your function(s) return a valid object to generate a response from API Gateway, applications on Lambda can use a separate function for each request path and method combination, or use one function for all invocations from API Gateway and use the event parameter and code logic to provide the needed actions.

Either way, this is a different pattern to how Flask structures its functions, requests, and responses. As such, we can’t simply deploy our Flask app as-is to Lambda. I’ll now talk about how we can do it without too much extra work.

Using Serverless framework to describe a basic app

The Serverless framework, along with its extensive library of plugins, is a well-established tool for provisioning serverless applications on a number of providers. It bundles your code and automates the deployment process, making it easy to create a serverless app.

Configuration of apps deployed using Serverless is done through the serverless.yml file. The example configuration below would, when deployed, create an API Gateway interface and a Lambda function using the code in app.py, and would invoke the lambda_handler function (above) each time a GET request is made to /hello:

service: my-hello-app

provider:
  name: aws
  runtime: python3.8
  region: eu-west-1
  memorySize: 512

functions:
  hello:
    handler: app.lambda_handler
    events:
      - http:
          path: hello
          method: get

Deploying an existing Flask app to AWS Lambda

The good news is that we can also leverage the Serverless framework to deploy Flask apps - and without needing much change to the existing project. This section assumes that you have an AWS account already that you can use. If not, then you can sign-up from their website.

First off, we need to install the Serverless framework itself. This can be achieved through NPM: npm install -g serverless.

Next, we need to configure credentials that will allow Serverless to interact with your AWS account. To do so, use the IAM manager on the AWS console to generate a set of keys (an access key and secret access key) and then use the following command to configure Serverless to use them:

serverless config credentials --provider aws --key  --secret

While you should try and restrict access as much as possible, the fastest (yet riskiest) approach is to use an IAM user with Administrator Access permissions. If you want to configure more security I recommend reading the Serverless docs.

Once the above groundwork has been completed, you can proceed to create a new serverless.yml file in the root of your Flask project:

service: my-flask-app

provider:
  name: aws
  runtime: python3.8

plugins:
  - serverless-wsgi

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - http: ANY /
      - http: ANY {proxy+}

custom:
  wsgi:
    app: app.app

Don’t worry too much about the wsgi_handler.handler and events parts - essentially these ensure that all HTTP requests to the service will get routed through to your app via a special handler that Serverless will setup for us.

This setup assumes your root Flask file is named app and that your Flask instance within this file is also named app (in the custom.wsgi attribute above), so you may need to change this if it doesn’t match your project setup.

Another thing to note is the new plugins block. Here we declare that our application requires the serverless-wsgi plugin, which will do much of the heavy lifting.

To make use of the plugin, you’ll need to add it to your project as a dependency by running serverless plugin install -n serverless-wsgi. As long as your Flask project dependencies are listed in a requirements.txt file, you can now deploy your app by simply running serverless deploy. After a few minutes, the framework will complete the deployment and will print out the URL to your new service.

Tweaking the deployment

There are various ways to adjust the environment of your deployed service. For example, you can change the amount of memory assigned to your function, make use of environment variables (e.g. for database connection strings or mail server URLs), define roles for your functions to work with other AWS services, and much more.

I recommend taking a look at the Serverless documentation to understand more about what options are available.

If you want to use a custom domain for your service, then you can either set this up yourself in API Gateway through the AWS console or by using the serverless-domain-manager plugin. Either way you will need to have your domain managed using Route 53.

Serverless caveats

Whilst the benefits offered by serverless delivery are strong, there are also some things to bear in mind - particularly when it comes to avoiding unexpected costs. Lambda functions bill per 100 milliseconds of execution time, and so long-running functions may be cut short (unless you tweak the duration allowance on the Lambda function, which can be up to 5 minutes long).

Additionally, if your Flask app makes use of concurrency (e.g. if you use threads to background longer-running tasks, like email-sending), then this may not play nicely with Lambda, since the function may get terminated once a response is generated and returned.

I outlined some extra things to watch out for in a recent article, so take a look through that if you want to read more on these.

Generally speaking, however, serverless apps are quite a cheap and risk-free way to experiment and get early prototypes off the ground. So, if you’re familiar with Flask (or other WSGI frameworks) and want an easy and scalable way to deploy your app, then perhaps this approach could be useful for your next project.

Migrating from Google Photos: Nextcloud, Piwigo, Mega, and pCloud

Wed, 24 Feb 2021 13:46:00 +0000

By now I’m sure everyone has heard the horror stories about people (seemingly-) randomly losing access to their Google accounts. Often the account closures are reported to have been accompanied with vague automated notifications from Google complaining that the account-holder violated their terms in some way, but without any specific details or an offer of appeal or process to resolve the “issues” and reinstate the accounts.

As such, these events usually mark the end of the road for the victims’ presence and data on Google platforms - including Gmail, Drive, Photos, YouTube - without having any option to extract the data out first. This could be years’ worth of documents, family photos, emails, Google Play purchases, and much more (ever used “Sign in with Google” on another service, for example?).

Some affected people are fortunate to have a large social media following to ensure that their posts describing this treatment can traverse the networks enough to get through to someone close to Google who can try and elevate it in order to get accounts reinstated. However, for most people this is not possible.

The creator of Stardew Valley recently found himself locked out of his 15-year-old Google account - even whilst involved in a key ongoing deal with Stadia, which he has since pulled out from due to feeling mal-treated. There are many similar stories available, and probably thousands more we never hear about.

Of course, I am sure there are legitimate reasons for many accounts to be removed and that the original intentions behind these automated systems were good. Either way, this still just worries me. Whilst I haven’t (at least, I don’t think?) done anything to violate any terms, I just don’t want to take the risk and wake up one morning to find I have lost 15 years’ worth of emails, photos, and documents.

What are the alternatives?

These days there are so many services that compete with Google’s own offerings. For example, DuckDuckGo is excellent for web search - though several times a day I do need to revert to Google search for more complex queries (which I can do by prepending DuckDuckGo search queries with g!). There are many websites that list good alternatives to Google services, and I won’t bang on about these ideas here - it’s up to you what you prefer to use, of course, and this type of thing has been covered many times before.

Personally, I’ve used my own domain to send and receive email (using Fastmail) for several years now, and self-host my files and documents using Nextcloud. I don’t really use YouTube or have much data tied-up in the other Google offerings.

However the one service I do rely on still is Google Photos. To be fair, this is a fantastic service - the apps seamlessly back everything up, the search is great (it’s Google’s bread-and-butter, after all), and I can easily and instantly find specific photos from two decades ago, or from any time in-between. It’s also super fast. I’d never found a good-enough replacement for media storage and so I never made the leap.

The problems with media storage

Images and videos - espcially with modern cameras and phones - take up a huge amount of space. I take a few pictures every day, and on my messenger apps I sometimes like to save images I receive from friends and family too.

This has resulted in a collection of over 84,000 pictures and videos in a mostly-continuous stream since 1998 - the year our family got our first digital camera. There are also digitised versions of photos from as early as 1959 on there too. Whilst this is not a massive collection by any standards these days, it forms a significant part of my own data footprint.

Whilst I was happy with using Google for this one area, I would get so nervous every time I read one of those “deleted accounts” stories that it got to the point where last month I finally committed to making a change.

In the meantime I needed to try and get my stuff out of Google Photos. The service lets you download 500 images at a time from the web interface, but that would have taken forever. The other option was to use Google Takeout. I did this and shortly after received an email containing links to download 48 different archives of data.

When I downloaded a couple of examples, I saw that they seemed to contain a lot of JSON metadata files and not many actual photos. I imagined I’d have to download the whole lot to try and make sense of it all and manually piece bits together. I thought I’d leave that for now whilst I continued my search for an alternative service, and come back to that problem later.

The search for a Google Photos alternative

The first job was to identify a new process/system for media storage. I had a few acceptance criteria in mind;

It needed to be affordable (not necessarily as cheap as Google One, but not bank-breaking either).
I needed it to be quickly and easily navigable (i.e. to easily move to a particular date to find photos).
It had to have some type of auto-sync from my phone’s photo gallery (I am too lazy to remember to manually “back-up” things - I need automation!).

I was already using Nextcloud for my documents anyway, and the Nextcloud app (which is brilliant) also includes an auto-upload feature for photos. However, I find Nextcloud gets a bit slow and grinds my server to a halt when viewing (I guess it processes things in the background?) large and recently-uploaded photos. Also, my VPS provider’s pricing (not unreasonable) would mean forking out the best part of $400 a year for the required block storage - which would only increase every year as my library gets bigger.

I also considered Piwigo, which looks great and is reported to be very fast. However the self-hosted option would have the same pricing implications as Nextcloud (above), and the hosted offering would be significantly more if I was to include videos too. I think Piwigo is aimed more at photographers maintaining and sharing albums rather than for use as a personal photo storage solution.

I recently tooted out to the community about this problem and I got some great responses back. One idea in particular caught my eye: Mega. I had used Mega before a while back, and the apps and web interfaces seemed to have come on a long way in recent years. After a bit of research I decided to choose this option. It seemed secure (with client-side encryption), quick, and the apps had the features I needed.

I went to pay for Mega (using the web browser), and it redirected me to a very dodgy-looking payment website - this threw me a little. I went back to the checkout page to see if I had clicked the wrong thing, clicked “confirm” again, and this time it took me to an entirely different (but still sort of dodgy-looking) payment site. I’ve set-up Stripe a few times before, and know it’s pretty trivial these days to accept card payments on your own site, and so alarm bells began to ring a little. My paranoid security-focused self was put off enough to continue my search.

Migrating to pCloud

That’s when I stumbled upon the Swiss-based pCloud on a Reddit thread discussing storage alternatives. It seems to be pretty feature-matched with Mega, despite not offering client-side encryption out-of-the-box - but then neither does Google Photos. Additionally, pCloud offers both US and European servers.

pCloud’s apps have similar functions to Mega, and the service also has the added bonus of offering a Google Drive integration! Hopefully this would mean I wouldn’t need to spend ages traversing that Google Takeout mess. The service also offers integrations with Dropbox, OneDrive, and some social networking platforms.

I signed-up and paid - without needing to go to any dodgy sites. I then linked my Google account and waited for the magic to happen.

It was a little slow. I know there was a fair amount of data, and I imagine the combination of this plus Google rate-limiting and other factors contributed to the speed too. I checked every few hours on the progress; there’s a sort of indicator (a folder count), but otherwise there was no way to really check what was going on. After a couple of days I noticed it had stopped (or “aborted”) by itself.

I had a quick browse through what pCloud had brought through and could see it had got to around July 2019 before it had had enough. This was OK - it had imported the vast majority and I was happy enough to run through the last couple of years’ worth of content on Google Photos, downloading 500 photos at a time to manually upload to pCloud in order to plug the gap.

I then un-linked my Google account from pCloud. I turned off Google Photos auto-upload from my phone and instead all new media now gets auto-uploaded to pCloud. Job done.

Final thoughts

pCloud’s navigation seems to be pretty quick, and uploading content is also very fast. It’s not perfect, though (is anything?) - viewing photos on the app can take a few seconds to generate/retrieve thumbnails, and it doesn’t have the smoothness that Google Photos offers.

However, it’s great for now. I have a “tangible” folder of media that feels more portable in case I ever need to move again. pCloud also has clear channels for communication if I do ever need to get in touch, and I certainly feel as though I am less subject to automated judgments from unruly algorithms.

The Glamour of Cyberpunk and the Road to Solarpunk

Sat, 20 Feb 2021 21:42:00 +0000

A few months ago I stumbled across this article: Beyond Cyberpunk: Towards a Solarpunk Future. It was posted on the excellent blog Tales from the Dork Web, by Steve Lord, which I can certainly recommend subscribing to.

I had never heard of the term “Solarpunk” before, but I read up more about it and the more I researched the more intrigued I became. Essentially it is defined as - more or less - the opposite to the Cyberpunk subculture, and I think we’re at a bit of a fork in the road from which either future could become a reality.

Cyberpunk

Cyberpunk (not the game by CD Projekt) is a term that describes a potential future setting that is pretty dystopian: there is a large “wealth gap” between the rich and poor; people live in dark and cramped accommodations, have mostly unhealthy existences, and are governed by a small number of large private corporations. The growth of these companies, however, allows citizens of the Cyberpunk future to be equipped with some pretty nice pieces of technology for communication, leisure & media, travel, automation, and anything else.

In a nutshell, it’s often described as “high-tech, low-life”.

Whilst it sounds (to some?) like a gloomy outlook, I love the dark and lonely imagery, the artwork, stories and subculture that has emerged from other people who are also fascianted by this movement. You’ve probably seen such scenes yourself in pictures, movies, books, and games that adopt the Cyberpunk setting. The r/ImaginaryCyberpunk subreddit community also often posts excellent and emotive content.

I love this image: Oris City by Darko Mitev and I can certainly recommend checking out more of his work and tutorials too. I love all of the atmosphere and detail.

Despite the “glamour”, interesting and exciting stories and movies, politics, and other cultural pieces that emerge from it, Cyberpunk describes a gloomy future that I imagine most people do not want to actually experience.

Solarpunk

I think we’re at a bit of a weird, but pivotal, point in time right now - from (geo-)political, societal and technological perspectives - in that the Cyberpunk dystopia is becoming a little unblurred. With ever-mounting consumerism, capitalism, bad choices regarding energy production, mass surveillance (from both private companies and governments), and much more, our reality certainly feels as though it is moving towards a point where some of the elements that comprise Cyberpunk do not feel too far-fetched at all.

The present feels pivotal because whilst there are excellent efforts being made to reverse some of these positions around the world (from local recycling schemes and zero-waste manufacturers through to fights for human rights and rallies around liberal activists), these processes only become effective and impactive if they are considered and actioned by society as a whole. While there are are still enough members that continue to wallow in seemingly-backward ideologies and refuse to become involved or make any of the needed adjustments, then change as a society cannot happen.

However, on a more positive note, if such challenges can be solved - and the right choices made now and in the near future - then a whole new potential future opens its doors: one that might be described as Solarpunk.

In a Solarpunk future humanity is much more in-tune with the world around it, maintaining a focus on sustainability (in terms of energy production, consumerism, ecology, and education), locality (in terms of sourcing materials and food, manufacturing, and the “do it yourself” movement), and - perhaps most importantly - an attitude that promotes sharing and positivity.

To me it’s not “hippyish” or necessarily to do with the adoption of socialism or the outright rejection of capitlism and associated ideologies - it’s more concerned with sensible balances across many facets of society and its politics. Competitiveness and drives to “do better” are parts of what make us human, and can very much live hand-in-hand with the other points and aesthetics we’re talking about here.

Nor is it a rejection of technology. In fact, from a technological perspective, forward-thinking efforts surrounding the free and open-source software movement and privacy-first companies are certainly components I see that can help contribute to (and become a focus within) a more sustainable and fair world. Technology can continue to innovate, develop, and improve in either setting.

Solarpunk isn’t about doing your bit to save the world from climate collapse. Solarpunk is about building the world you want your grandchildren to grow old in. - Steve Lord

We’ve already seen some fantastic real-world efforts that can be considered part of this movement - from architecture and transport through to self-repair and home agriculture. I love the bottle farm idea included in the post I mentioned at the start of this article, and want to try this myself.

There are also the more obvious reflections, such as to fully embrace solar energy (and other renewables) as a source of power - both at an individual and industrial scale - and efforts concerned with maintaining green spaces in developing and urban areas. I think that the more mainstream and ubiquitous we can make all of these actions the more realistic a Solarpunk world can become.

–

Note: this article only scratches the surface of the Cyberpunk and Solarpunk subcultures. It is aimed to be more of a primer to introduce the concepts behind these ideas and to perhaps pique the interest of readers enough to continue their own research.

A Year Without Answering my Phone

Thu, 18 Feb 2021 21:01:00 +0000

This month marks a year from when I decided to (mostly - see below) stop answering my phone. This was not because I wanted to be antisocial (quite the opposite), but because it’s become the wrong form of communication for me.

Why did I stop?

Like many people, I am inundated with sales-y and spammy phonecalls. I have had the same mobile phone number since 2001 (that’s 20 years this year), which I am sort of proud of and would prefer to keep. However, careless (or malicious) entities over the years (and more than likely mistakes also made by my younger self) have meant that my number and name are now in the databases of many different types of agents - from insurance/legal company sales teams through to dodgy Bitcoin spam companies.

It got to the point that the signal/noise ratio (“real” phone calls vs. unwanted) probably dropped to around 5%. At first, spam calls were easier to spot (they’d call from random UK cities), but recently calls started to come in from numbers starting with “07” (which designates a mobile number in the UK) and also more and more from the area code of the city where I live - probably in the hope of appearing more legitimate to me.

I also find talking on the phone sort of stressful. I’m sure I’m not alone in that the Phone app is probably the least-used part of my smart"phone". For some reason, to me it just doesn’t feel natural, and - with the exception of close friends and family (and even them sometimes) - I’d much rather “talk” to people via IM or live text chat.

I’m naturally pretty introverted so I get on better with channels that enable me to think and formulate comms in my own time.

Unexpected and unscheduled calls are also pretty rude, I think. Stephen Fry sums up essentially what I feel about this in this short but great clip from QI - that phoning someone out of the blue is really the equivalent to going up to that person and yelling at them, “speak to me now, speak to me now, speak to me now”. Without caring that they might be busy, stressed, not in the right frame of mind, or any number of other states.

This is incredibly invasive to do to someone you don’t even know.

What was the result?

In the end I made a pact that I would no longer answer the phone unless it was a pre-arranged call or from a number that I recognised - and only then close friends and family.

I feel far more empowered and in control of my own time when I hear/see my phone ring - and I just silence it and let it ring out. The decision has already been made to purposefully miss the call and so there is no need for any anxiety that might accompany such unexpected calls.

I sometimes choose to avoid calls from numbers I do recognise. These callers (usually businesses I deal with) just follow-up with an email anyway to which I can respond when I’m ready - usually within the hour. If there is an emergency they can leave a voicemail, which I will get notified about and then choose how best to respond. Friends and family either feel the same as me or know me well enough so that I don’t need to miss their calls.

Either way, I haven’t (knowingly) missed any events, appointments, insurance renewals, or whatever. I am going to carry on as I have been and I can certainly recommend this approach to you too if you feel the same way as me about unwanted phonecalls.

The Midnight Library by Matt Haig

Sat, 13 Feb 2021 21:17:00 +0000

Last week I read The Midnight Library by Matt Haig. The book won the 2020 Goodreads Choice Award for Fiction.

“Set” in Bedford, England, the story starts by introducing the main character - Nora Seed - who feels completely down. She is depressed and thinks that she has nothing further to contribute to her own life or to the lives of the few people around her.

On the day she decides she no longer wants to live, she is fired from her job, her cat dies, and other events occur which help cement her decision. However, as she dies she is transported to a place that exists between life and death: The Midnight Library.

Here she is presented with the infinite number of books that make up the lives that could have been had she made different choices in the past - whether those were big or small (such as choosing whether to have a tea or coffee) or something more obviously impactful. Either way, they can contribute to a complete change in life direction.

She has the option to begin living these different lives by considering the regrets she has about the decisions she made in her root life. As she “visits” her other lives she reflects on the decisions that led her to that point, and also realises the power in the choices she makes in their ability to also drastically affect the lives of those around her.

Whilst I feel that the book was perhaps not as deep as it could have been, it is my opinion that this was an intentional design by the author as it made the experience more of a canvas in order for the reader to make their own reflections.

Some of the key thought takeaways for me were around the knowledge that whilst the decisions one person makes may benefit them, they may not be beneficial for everyone. Whilst of course it is important to consider the happiness and wellbeing of yourself as well as those affected by your decisions, one needs to live and experience the variety of life without feeling paranoid about making the decisions that you feel are the right ones.

The book made me reflect on some of my own decisions. I know the grass isn’t always greener but that the choices are always there to be made if I want or need a change - it is never too late.

The premise of the story sounds like it could be depressing, however I did not find that at all. In many ways, it was the complete opposite: having an understanding of the power in your choices helps you realise that even when things feel at their worst, you are not powerless. There is always something you can do to make a change and choices to be made to gear yourself towards where you need to be.

We all have regrets in our own lives, and decisions we wish we did (or didn’t) make, but these should not be dwelled upon or worried about. Instead we can consider them as the useful tools they are to help us make different or better decisions as we look forward and continue into the future.

SSH Jumping and Bastion Hosts

Wed, 10 Feb 2021 22:11:00 +0000

For many small or personal services running on a VPS in the cloud, administration is often done by connecting directly to the server via SSH. Such servers should be hardened with firewalls, employ an SSHd config that denies root and password-based login, run fail2ban, and other services and practices.

Linode has some great getting-started guides on the essentials of securing your server.

Protecting sensitive servers

In more complex production scenarios heightened security can be achieved by isolating application (webapp, API, database, etc.) servers from external internet traffic. This is usually done by placing these “sensitive/protected” servers in a private subnet, without direct internet-facing network interfaces. This means that the server is not reachable from the outside world.

In this type of scenario, outbound traffic from the sensitive server can be routed through a NAT gateway and inbound traffic can be funnelled through a load-balancer or reverse proxy server. In both these cases the NAT gateway and load-balancer would exist in public subnets (with internet-facing network interfaces) and can reach the sensitive server through private network interfaces in order to forward requests (e.g. web traffic).

Now the question is around how one does manage the services running on the protected server, since it is no longer available to connect to. Traditionally this is done by introducing bastion hosts into your network.

Bastion hosts

Bastion hosts - like the NAT gateway and load balancers - sit in the public subnet and so they are available to the outside world. They often accept SSH connections, from which one can “jump” through to the protected servers through the bastion’s private networking interface.

Bastion hosts should be hardened as much as possible (with firewalls and other network rules), and should run a limited set of services - in many cases simply SSHd.

This server then enables administrators to connect through to the protected servers in order to carry out maintenance, upgrades, or other tasks.

Connecting through a bastion host

SSH port-forwarding is a widely-used concept, in which a secure tunnel to a service running on the protected server is opened via a port on the local machine (using ssh’s -L option).

Another option is to use proxy jumping (with the -J option):

ssh -J bastion.company.com protected.company-internal.com

In this example the user connects via the bastion through to the protected server at protected.company-internal.com. Since you should be using key-based authentication to connect, you may also need to specify the private key path (with the -i option), and also tell SSH to forward your agent to the bastion (using -A) so it can continue the connection.

This can make things hard to remember each time. You could write the command in a script, however it’s probably easier to use SSH’s own local configuration. To do so, you can add the following to your local ~/.ssh/config file:

Host *.company-internal.com
  ProxyJump bastion.company.com
  User username
  IdentityFile /home/username/.ssh/identity

With that in place you can now simply run the following when you want to connect to the protected server:

ssh protected.company-internal.com

Note: depending on your system you may need to add the key to your local agent first, but you just need to do this once per login (ssh-add ~/.ssh/identity).

A note on DNS

Generally I would probably avoid assigning public domain names to a bastion host, as this may invite unwanted attention and traffic (even if the host is secured). Instead you can just include the IP address directly in the ProxyJump line of .ssh/config. I used domain names in the examples above to make the process clearer.

Also, in the above example I refer to the company-internal.com domain for use within the private network. This domain should only be resolvable within members of the private network - either by using an internal DNS server or by simply modifying /etc/hosts on the bastion. Alternatively you can just use the private IP address for the protected server on the Host line of .ssh/config.

Additional notes

In setups like this you may also want to consider the following:

Private keys

Don’t provision these on your bastion host. Instead use agent forwarding (as described above). You’ll need to add your public keys to both the bastion and protected servers.

Restrict source network

For extra security you can restrict SSH connections to your bastion only from trusted networks (e.g. your office network or a VPN).

Similarly, restrict protected servers such that they only accept SSH traffic from the bastion, and not from other servers on the network.

Make use of managed services if/when possible

For extra security you can use managed services when they are available. For example, if you use AWS then you can make use of a combination of VPCs, subnets, NAT gateways, elastic load balancing, security groups, Route 53, and other services to secure your hosts and control your network. You can of course set this up on your own servers without relying on managed services.

Either way, I hope this post has helped shed light on some simple ways to improve network security for your applications and services.

Using Monica to Help Manage your Personal Relationships

Sun, 07 Feb 2021 19:31:00 +0000

Many people no longer feel comfortable using Facebook. Whether you were never a member to begin with or you’ve had an account but chosen to remove yourself from the service, or you’ve simply tried to start using it less - either way, it’s no surprise given the way that they, across their family of products (including Instagram and WhatsApp), operate in terms of your own data and time.

This is a huge subject on its own and it’s really up for everyone to make their own minds up when it comes to their own stance. It’s been widely discussed pretty much everywhere, and there are loads of resources available on this handy website if you’re interested in understanding more about what goes on behind the scenes on these platforms.

Staying in the loop

Anyway this isn’t another post about Facebook, but one of the things that is useful about that particular platform is its birthday reminder system in which you automatically receive an email from Facebook if it happens to be one of your friend’s birthdays that day. In itself, this is of course simply a mechanism to try and get you to re-engage with the platform - such as to send your friend a direct message on Messenger or to post something on their timeline.

However, it is nice to get messages on your birthday, and nice to imagine that someone you only speak to a couple of times a year has the headspace to remember that today is your special day. Even though you both know that it’s because Facebook has sent a reminder with an easy CTA.

The good news is that there are still lots of services that help you remember key events without needing to rely on Facebook. Of course you can set-up calendars (many mail providers have in-built calendar facilities that can sync to your client with CalDAV), but you may want to remember other things too - such as anniversaries, friends’ pets’ names, that time you helped your cousin move house, and more. Quickly all of this info ends up distributed between a number of systems and becomes hard to look-up and manage (unless you’re super organised).

Monica: the “Personal Relationship Manager”

What we need is a personal CRM (“customer relationship manager”), which can do all of this for us. And thankfully such systems exist - such as Monica.

Monica is the single best investment you can make to have better relationships. - monicahq.com.

Monica is a piece of open-source software that can handle all of this for you as a “Personal Relationship Manager” (in their words) - and much more. You can sign-up on their website and pay a small ongoing subscription fee to cover the server costs. Alternatively, you can easily self-host it on your own server.

I’ve been using it (the self-hosted option) for some time now, and love its features. I get automatic email notifications in-time to remind me about key events, I can keep track of the birthdays of my friends’ kids, remember gifts I have been given, friend life events, jobs, and more.

Although I still want to spend some further time setting it up and adding more details about the people I know, it already helps me to include richer information when I message friends and family and to remember the things I really should be anyway.

Monica looks great, works fine on my phone web browser as well as my desktop browser, and also has an API that allows you to build your own workflows or to connect it to other services.

If you find yourself forgetting birthdays and important information about friends and family, or if you just want to log relationships more effectively, then I certainly recommend giving it a go.

How to self-host Monica

I host Monica on a relatively small VPS. It’s lightweight and it happily runs alongside a few other services.

I usually prefer using Docker to host things like this as it helps keep things isolated when running multiple services on the same machine. I have an Nginx container (with several virtual hosts) that proxies requests through to the appropriate services.

The Monica Team kindly maintain an official Docker image. I went for the Apache version (as I already have Nginx in-place for TLS, etc.) for which there is an example Docker Compose config available on the official Monica image page. The documentation also explains how to get your first user setup.

One of the main advantages of Monica is its ability to keep you updated without you needing to login and check-up on things. It does this by sending you emails, and for this to work you’ll need to add a bit of extra configuration to your Docker Compose file, as described on this page. Just add the extra variables to your environment section in docker-compose.yml. The article mentions Amazon SES, however you can use your own mail provider’s SMTP/IMAP server settings here (e.g. Mailgun).

If you plan to use Linode to host your Monica service (which is a great choice), you may just need to open up a quick support ticket with them so that they can make sure your account is allowed to send traffic on standard email ports (e.g. 25 and 587), which they sometimes restrict on new accounts to help fight spam.

Contribute

If you want to contribute to this great open-source project, then there are guides available on GitHub.

Add icing to your websites using pattern.css

Sat, 06 Feb 2021 19:34:00 +0000

Shapes and patterns can be leveraged in user interfaces to guide your users, draw attention to content, lend weight or emphasis, or just for aesthetics and decoration.

Layout and styling on the web is typically handled using CSS, however mastering CSS to the level where you can confidently take advantage of more advanced features is definitely not easy. I’ve been developing for the web almost full-time for a decade and I’m still pretty crap when it comes to doing complex stuff with CSS.

That said, some people have done some mindblowing things using CSS and just a single div element.

The Subtle Patterns website has been around for years - it’s a great resource for discovering and accessing nice textures and backgrounds for your creations. There are also some nice libraries in CSS that let you describe patterns programmatically, and this comes with the added performance advantages that CSS provides (as browsers are pretty performant when it comes to CSS).

pattern.css (created by bansal-io) is a great little CSS-only library for adding simple, but effective, patterns to your websites. For example, backgrounds for elements, false “block” shadows, or even within the text itself. All that’s needed are a few extra classes on your elements and the small (less than 1Kb when gzipped) library will do the rest.

To get started, you can add the library to your project using your normal JavaScript package manager (e.g. yarn add pattern.css). Then either include the CSS file in your HTML or, if you’re using React or another framework/builder that allows you to import CSS directly, you can:

import 'pattern.css/dist/pattern.css'

Once that’s done it’s just a matter of adding classes to your markup. All the pattern.css classes start with pattern, followed by the type of pattern (e.g. -diagonal-stripes), followed by the “size” of the pattern (e.g. -sm).

For example, to build a div with a chunky zig-zag patterned background you just need to use:

<div class="pattern-zigzag-lg">
  ...
div>

To change the colour of the pattern just set a color style on the element. If the element also has a backgroundColor then this will display through the transparent bits:

<div class="pattern-diagonal-stripes-md" style="color: red; backgroundColor: yellow">
  ...
div>

Have a read through the documentation for examples and further pattern types. It’s quick to get the hang of and far more effective to use if - like me - you find some of the complexities of CSS hard to get your head around!

React State Management with Zustand

Fri, 05 Feb 2021 23:46:00 +0000

React state

React state management is what gives the library its reactiveness. It’s what makes it so easy to build performant data-driven applications that dynamically update based on the underlying data. In this example the app would automatically update the calculation result as the user types in the input boxes:

import React, { useState } from 'react';

function MultiplicationCalculator() {
  const [number1, setNumber1] = useState(0);
  const [number2, setNumber2] = useState(0);
  return ( <>
    <input value={number1} onChange={e => setNumber1(parseInt(e.target.value))} />
    <input value={number2} onChange={e => setNumber2(parseInt(e.target.value))} />
    <p>The result is {number1 * number2}.p>
   );
}

The entire function will re-run on each state change (the setNumber1 and setNumber2 functions) in order to reactively update the result text. The multiplication itself could be calculated in a useEffect but it is simpler to look at it as shown.

This is totally fine for many apps, however this quickly becomes unmanageable when you need to share state (e.g. number1) between this component and another component - and ensure that a state change in the former can be reflected in the latter - whether it’s an ancestor, descendant, or a more distant component. Of course, you can pass the state variables (and the associated setState functions) from a parent down as props to child components, but as soon as you’re doing this more than a handful of times or in cases where state needs to be shared across distant components this quickly becomes hard to maintain or understand.

An example of shared state might be to store the details about the currently logged-in user in an app. A navigation bar component would need to know about the user state to show a link to the correct profile page, and another component may need access to the same state in order to allow the user to change their name.

Context and Redux

This is by no means a new problem. Many of these issues are solved using React’s Context API and there are also libraries like Redux that are useful in perhaps more complex scenarios - it’s much more opinionated and involves a fair bit of extra code that may be overkill in many apps. Adding just a small piece of state (e.g. a new text input), and the ability to alter it, to Redux involves updating reducers, creating an action, dispatchers, and wiring things through to your components using connect, mapStateToProps, and mapDispatchToProps. Plus you’ll need the relevant provider higher up.

Redux is certainly a fantastic library, however, and I use it in many apps. This post is useful and discusses the cases in which you may (or may not) want to use Redux.

Zustand

In this post I want to talk about another option that is perhaps quicker and easier to use, expecially for those newer to React (though it’s also great for more seasoned React developers) - zustand. Not only is this the German word for “state”, it’s also a nice and succinct library for state management for React.

The zustand library is pretty concise, so you shouldn’t need to add too much extra code. To get started just add it as a dependency to your project (e.g. yarn add zustand). Now let’s rewrite the earlier multiplication example but using zustand.

First, define a store for your app. This will contain all of the values you want to keep in your global state, as well as the functions that allow those values to change (mutators). In our store, we’ll extract out the state for number1 and number2 we used in our component from earlier, and the appropriate update functions (e.g. setNumber1), into the store:

import React from 'react';
import create from 'zustand';

const useStore = create((set) => ({
  number1: 0,
  number2: 0,
  setNumber1: (x) => set(() => ({ number1: x })),
  setNumber2: (x) => set(() => ({ number2: x })),
}));

Now - in the same file - we can go ahead and rewrite our component such that it now uses this store instead of its own local state:

function MultiplicationCalculator() {
  const { number1, number2, setNumber1, setNumber2 } = useStore();
  return ( <>
    <input value={number1} onChange={e => setNumber1(parseInt(e.target.value))} />
    <input value={number2} onChange={e => setNumber2(parseInt(e.target.value))} />
    <p>The result is {number1 * number2}.p>
   );
}

That’s it - we now have a React app that uses zustand. As before, the component function runs each time the store’s state changes, and zustand ensures things are kept up-to-date.

In the example above the two blocks of code are in the same file. However, the power of zustand becomes particularly useful when the store is shared amongst several components across different parts of your app to provide “global state”.

For example, the useStore variable could be declared and exported from a file named store.js somewhere in your app’s file structure. Then, when a component needs to access its variables or mutator functions it just needs to - for example, import useStore from 'path/to/store' - and then use object destructuring (as on line 11 above) to pull out the needed variables and functions.

It’s worth checking out the documentation since zustand is super flexible and can be used in ways that help improve performance, such as taking advantage of memoizing and state slicing. It also makes what can be tricky in other such libraries - e.g. asynchronous state updates - trivial.

If you’ve already got an established app using another state management system it may not be worth migrating everything over. But give zustand a go in your next project if you’re looking for straight forward, yet powerful, state management.

RSS: The Rise and Fall... and Rise Again

Wed, 03 Feb 2021 22:16:00 +0000

Many people would consider RSS - Really Simple Syndication - to be a relic of the past. However I think it has been making a comeback.

RSS is a mechanism by which people can automatically receive updates from individual websites, similar to how you might follow another user on a social networking service. Software known as RSS readers can be used to subscribe to RSS feeds in order to receive these updates. As new content (e.g. a blog post) is published to an RSS-enabled website, its feed is updated and your RSS reader will show the new post the next time it refreshes. Many RSS readers have an interface similar to an email client, with read/unread states, folders, favourites, and more.

The rise

RSS was first released in early 1999, and it steadily gained popularity amongst content producers and consumers, with adopters from media outlets and software implementations making their way into early Internet Eplorer and Firefox versions, amongst others. These were the days before the “real” Web 2.0 hit, and in which websites were very much more silos of information. Tools like RSS were powerful then because they enabled the easy aggregation of information from multiple sources.

Not too long after this (Web 2.0 ‘began’ in the mid 2000’s), and during the years ever since, mainstream social networks became ubiquitous. Many people flock(ed) to these as a way to share and subscribe (by following others) to receive updates in real time, several times a day and from lots of different people and organisations. These services enabled features far beyond aggregation by allowing easy sharing, rating (e.g. likes) and commentating such that today such services have become the primary means of information and news sharing and reception for many people.

The fall(?)

At that time RSS was still very much “a thing” for many people (though the discontinuation of the hugely popular Google Reader in 2013 was a bit of a bummer to these communities). However new people now joining the web scene would be far more likely to instead engage with these extremely well-funded, well-marketed, and centralised social platforms - perfectly engeineered to be addictive, entirely driven and propagated by FOMO, and focused on content-sharing (even if the content is often misinformation) - where you are the product, rather than spend the time researching and subscribing to individual RSS feeds.

To some commentators in this space the concept behind all of these social platforms is known as the fast web - a web that tells you when and what information to consume rather than letting you make that decision for yourself. Facebook, Twitter, Instagram, and others all started as just a chronological timeline of interesting content from friends and family. On all of these services today the “algorithm” determines what (and who) goes in your timeline, and it constantly learns what to feed you - and when - in order to get those few extra minutes from you each day. This is literally its business model.

What used to be an innocent bookmarking tool, Twitter’s “favouriting” mechanism is now essentially a game of retweet roulette in which the algorithm will every now and again choose to include your “bookmarks” (not just retweets) on the feeds of people who follow you. If that’s not anxiety-inducing or user-hostile then I don’t know what is!

Of course this is something Facebook has done for a while too, except perhaps in a more sinister way - such as implying a user has liked something when they haven’t at all.

Other social networking tools can be more user-friendly. For example, the open-source Mastodon software powers distributed social networks that aren’t fuelled by addiction and instead give you more control over what you receive and where your posts go. However these tools still have some way to go to becoming anywere near mainstream.

I want to caveat some of the above: I obviously don’t think any of this is the fault of the individual. These social platforms are fantastically easy places to set-up a web presence. Creating an Instagram, Facebook, or Tik-Tok account for you (or your business) takes literal seconds. Within a minute you can have your profile setup and be following a dozen people, and already getting engagement and “reactions” from others (remember those “Your friend, X, is now on Instagram!” type notifications?).

With all of this power and efficiency at the fingertips it’s no wonder that people don’t create their own personal websites anymore, or feel the need to reach out to actively keep-up with other such sites. What’s the point in re-inventing the wheel when I can easily create a Facebook page for myself that includes an inbuilt blog “feed”, a space for links, photos, and more? And it’s “free”! The barrier to creating a self-owned personal space on the internet for yourself is considered too high for most people, and is probably still seen as “geeky” even if it does come with all the benefits of privacy and control.

And I’m not saying that self-owned spaces, RSS, and that whole ecosystem are related, or the opposite to, mainstream social media; more that it is a useful way to compare and contrast different ways of accessing and disseminating information and the level of control one has over this.

This probably feels like I’m going way off-piste, and I sort of have, but my key meaning here is that for several years the concept of RSS has evaporated from popular knowledge because people haven’t needed it - either as a tool for receiving or disseminating information. Ask your non-tech friends and family if they’ve heard of RSS (and know what it is for a bonus point) - I bet the positive response rate will be low in most cases, especially in younger respondents.

Also, I don’t think this is solely the fault of the social giants. Online media outlets - which would have needed to rely on RSS for years before online social media became more mainstream - now often completely ignore it or treat it as a second-class citizen.

The BBC News website happily displays large friendly icons for Facebook, Twitter, and the like, but no mention of RSS (try ctrl-F). In fact, you’ll probably need to search the web for “bbc rss” in order to find the RSS feeds that are listed on a page that hasn’t been updated for over a decade and which still lists IE7 and the long-discontinued Google Reader as sensible options (though ironically I suppose this does indicate the stability and robustness of the RSS system).

The rise again

Anyway, all that sounds a bit doom and gloom but I definitely think we are starting to see a shift in people’s attitude towards and - importantly, trust in - these big tech companies. Facebook’s recent attitiude towards information collection (and subsequent sharing) has hit mainstream headlines and everyone must have seen Whatsapp’s popup about data sharing. Too much uncertainty undermines the trust in these platforms, and people have understandably sought out other options. A few weeks ago Telegram reported an addition of 25 million new users within 72 hours as a result of these policy “changes”.

My parents aren’t really tech-aware at all but even they were telling me last week on a video call about this “new app Signal” they had downloaded and begun to use with their friends - without any of my input.

I’m not sure what it is, but people seem to care more about their data these days. Whether that’s because of GDPR, the fact that coronavirus means people aren’t endlessly scrolling through social feeds on their daily commutes anymore, or something else or a mixture of everything. And that extends to being more picky about the information they receive too.

Either way, I’ve noticed more and more posts like this (and the subsequent reactions and discussions) recently, and the #100DaysToOffload movement has brought about a surge in people - myself included, really - creating their own longer-form content, for which RSS is a perfect distribution mechanism.

I think we’re on a bit of a brink representing a general - but real - change in attitude from people towards data, and the time that they choose to give to now lesser-trusted platforms. It is our responsibility to help educate about the alternative options so that those around us can make their own decisions. Whilst I am relatively new to RSS in the grand scheme of things (having only really started properly engaging with it about a year ago), it already makes me feel more in control of what I view, and when.

Whilst this concept doesn’t need to be limited to RSS, it’s a great starting point as it’s easy to understand. It “feels” friendly, it helps power connections to the decentralised and small web.

It, as a concept, has no business model. Though of course you can pay for the software you use, and websites can make money through ads, but at least you have a choice regarding who you subscribe to and the software you use to do it through (and there are lots of choices). You aren’t tied into anything and it respects your privacy - you don’t need to “sign-up”, provide your details, and sites don’t know that you personally have subscribed.

RSS may be age-old, but it is an excellent way to still get the information you need as you begin to use mainstream social media less, and - although it doesn’t need to be slow in itself - it is a fantastic tool to combine with the growing and user-respecting world of the slow web, in which timeliness (where you’re in control) is far more important than “real-time”.

–

Edit

I’ve received some replies to this post that talk about the lack of mentions of the Atom standard and podcasts. RSS certainly is (and has been) a fantastic way to subscribe to podcasts. Its flexibility and ease of use has been a great tool for both content creators and consumers, and has helped to build the ecosystem of podcast apps and services we see today. And of course, there are other very useful distribution mechanisms and standards available for distributing information, such as Atom. This post was focused more on contrasting this family of systems with what many people may consider “mainstream” services, and how the wide adoption of the latter has perhaps had an effect on the former.

Blogging for Devs

Tue, 02 Feb 2021 20:31:00 +0000

A few months ago I discovered Blogging for Devs - I think through Product Hunt when it made it to #1 Product of the Day back in August last year.

At the time blogging was something I had been thinking about quite a lot. I actively followed several other blogs - both from people I know and from others in the tech community - and it was clear that, in addition to producing content that was interesting to read by others, writing was something these bloggers actually enjoyed and found valuable too for their own learning and engagement with the community.

I have also always enjoyed writing (you have to if you’re ever involved in research!). I was still posting things occasionally, and had been doing so for several years, but blogging had just never really got round to forming any part of my normal routine. It was certainly something I wanted to do more of - to write, engage more with likeminded people, and for all the other personal and professional benefits associated with consistent and frequent writing - and so this was clearly a habit I needed to learn to form.

Blogging for Devs is a course and newsletter created by Monica Lent and, with all of this running through my head, I signed-up almost straight away.

I don’t want to give away too much about Monica’s course or content (it’s free to sign up yourself!), but one thing I found really valuable was actually something that happened right at the start of the course. After I signed-up I received an automated email asking why I had chosen to sign-up and what I wanted to learn. Of course, I know this is largely to help Monica shape her course and to get an understanding of people’s needs, but I actually found it a super-helpful self-reflection.

Why didn’t I blog more? What was blocking me, even though it was something I actively wanted to do? After a while of thinking it boiled down to one main thing, which was my confidence and, in particular a fear of what people would think if they read it (especially if they knew me!) and also a worry of writing about things no-one is actually interested in (“why would anyone want to read this?”). I summarised this and wrote it back to Monica’s email, and she got back to me with a nice personal reply not long after.

The course covers lots of topics - from SEO and branding through to actual blog content. However, my issue was still very much the whole confidence thing. One thing that became clear to me during the course is that the most important step in getting over that barrier, and then forming a habit - whether it’s getting up early, doing more exercise, or writing blog posts - is just to start doing it.

And I don’t mean tomorrow or next week, I mean today. Just pick something to write about. If you’re just getting started it can be a quick post introducing yourslf (WriteFreely is a great platform if you need one). If you’ve already got something going and want to write more (like me) then write a short post about something you’ve learned today - tech or not. The important thing is just to start doing it.

Of course, not everything you post will be enjoyed by everyone, but that’s OK. It’s not always solely about your audience; you’re doing it for yourself too, remember.

And also remember to sign-up to Blogging for Devs today too. It’s a fantastic course. If you look back at my writing history you’ll notice the difference it’s had on me. I’ve blogged much more consistently and effectively since taking the course, and I’m still working through some of the content even today.

Even if you’re already a seasoned blogger I’m sure you’ll pick up some extra tips and helpful insights, and the Blogging for Devs website also has a great Trends section to help you discover new blogs to follow.

Why not SQLite?

Mon, 01 Feb 2021 21:06:00 +0000

If you need a database for your next project, why not first consider if SQLite might be a good option? And I don’t mean just for getting an MVP off the ground or for small personal systems; I mean for “real” production workloads.

Many people will be quick to jump on this with chimes of “it’s not designed for production”, but I think it depends on what is actually meant by “production”? Sure, it’s not the right choice for every scenario - it wouldn’t work well in distributed workloads or for services expected to receive a very high volume of traffic - but it has been used successfully in many real-world cases.

What made me feel the need to write this article was seeing this sentence in the README of the Synapse Docker repo:

By default it uses a sqlite database; for production use you should connect it to a separate postgres database. - matrixdotorg/synapse

Don’t get me wrong. I totally get its meaning, but at the same time do personal Matrix servers or home Nextcloud servers not count as “production”?

Pieter Levels famously used SQLite to help drive revenues from some of his products to well over six-digit dollar values, and SQLite’s own ‘appropriate uses’ list explains where it can be useful:

SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites) - sqlite.org

Even if your site or service does eventually outgrow SQLite (which will be a nice problem to have), your application code will still be using SQL and so it should be relatively easy to migrate to something like PostgreSQL.

As Paul Graham said, “do things that don’t scale”.

Of course, it is backed by disk and so is subject to the usual I/O constraints applicable to any file, but nearly all VPS providers offer SSD-backed instances these days and SQLite claims to be faster than filesystem I/O anyway.

It’s worth remembering that there can be huge overheads and costs in setting up “production-ready” database servers. You’ll need to think about provisioning the instance itself, installation of dependencies, certificates, the usual networking hardening (firewalls, ports, etc.) - and then keeping all of this up-to-date too. Even when using managed database services there are still user roles, authentication and rotating credentials to worry about, along with securely provisioning your applications with the connection strings.

Having all of these things to worry about carries the additional risk of encouraging people to become lazy or to not have the time needed to make sure everything is done properly; an easy way to accidentally introduce security issues. Plus, if you have multiple environments (e.g. for staging or testing) then these factors, and the associated costs, amplify.

There is also some interesting discussion on the topic in this Hacker News thread from last year.

I just think it’s definitely worth a go before jumping straight into alternative heavier options. It’s free, has a smaller footprint, has easily accessible bindings for many languages, and you can get started in minutes - all you need is a file.

Dirty Little Secrets by Jo Spain

Sun, 31 Jan 2021 18:15:00 +0000

Recently I finished reading Dirty Little Secrets. This is the first book I have read by Jo Spain and the first time I have known of the author.

The book first appears as though it’s a typical murder mystery set in a relatively wealthy gated community in Ireland - however the intricacies of the characters and narrative quickly made it hard to put down. The story begins with the discovery of the long-dead body of the woman who lives at number 4 and continues with the involvement of the detectives as they investigate the strange incident.

The narrative primarily focuses on and is told from the perspectives of the neighbours and the police. It becomes clear that everyone - including the detectives - has hidden backgrounds and the story cleverly interwines past and present timelines (along with later repeated scenes told from different viewpoints) such that open-ended questions and arcs are often eventually resolved.

I really enjoyed this book, which I listened to as an audiobook well narrated by Michele Moran. It helps reinforce the reality that everyone has obscured backgrounds or secret parts to them, which can be prematurely forced into the open by external events.

Interestingly, another recent book I read - The Guest List by Lucy Foley - is a similar murder mystery also set in Ireland (Goodread’s content recommender systems clearly working at their best). Although on paper it is similar (in terms of its location and multiple character-based perspectives) and it is similarly well-reviewed by others, I personally didn’t really enjoy it. The Guest List is certainly more of a suspenseful “thriller” in the traditional sense (largely given its setting) and the conclusion is probably more shocking, so I am not surprised it received good reviews. However, I just found the story to be a little dull and the characters a bit uninteresting and un-relatable. Each to their own, but I felt Jo Spain’s storytelling and character development to be far more compelling.

Out with the Old: Moving to Gitea

Sat, 30 Jan 2021 15:31:00 +0000

If you’ve visited my geminispace (gemini://wilw.capsule.town) you’ll have noticed that I’ve recently been on a mission to decentralise the every-day tools and services I use, and will understand the reasons why. This post will likely become part of a series of posts in which I talk about taking control and responsibility for my own data.

One of the changes I’ve made more recently is to move many of my own personal projects (including the source for this site) over to a self-hosted Gitea service. I chose Gitea personally, but there are many other self-hosted solutions available (see this post for examples and comparisons).

The “problem” with GitHub

I’ve been a GitHub member for as long as I can remember, and will continue to be so and actively use it in my more professional work and when contributing to other projects. However, I don’t think I’m alone in that athough I try and develop things in the public and keep many home projects open-source, I usually don’t do it with the intention of receiving contributions from others. The discoverability on GitHub is great (though some may argue that its size means that things can get a bit “diluted”), but many of the projects I develop are for my own use - and while anyone is free to take the code and use it as they want, the powerful tools offered by GitHub (and other centralised services) just never get used for these types of projects.

The other thing is that GitHub seems to have gradually become the LinkedIn of the software world, and many people use it as the basis of their CV or portfolio. This is great as it allows other people and potential employers to get an idea of the kinds of things a developer works on, coding style, and so on, but there’s always a certain feel of pressure (or sometimes subconscious competitiveness) that people on any socially-focused platforms might get.

When Twitter introduced their Fleets feature they mentioned that one of the motivators behind the project is that they understand that some people get a fear of posting tweets when things feel so public. I’ve seen the same thing with GitHub in that people feel put-off contributing or publishing their own work in public repositories “in case someone sees” - is this a barrier to entry for more introverted developers? Inversely, re-engagement mechanisms - like the contributions graph on each user’s profile - may make developers just publish for the sake of it.

None of these things are necessarily problems or wrong (private repos are always an option, for example), but these days it just feels more appropriate to be responsible for your own data as much as possible - especially when not making the most of what alternatives can provide you with, and it’s always good to use and encourage alternative options so that one service doesn’t become the expected norm.

My experience so far

Since migrating many projects over to the smaller “world” that is my own git server, I get the feel that things are slower (in a good way) and I have been spending more time curating projects and working on the things I actually want to work on (though many are still private “for now”!).

If you’re interested in trying your own self-hosted Gitea server, then it’s pretty straight forward if you have a VPS (I just used the official Docker images, for which there are instructions in the documentation).

To move existing repositories over it’s as simple as changing the remote (or adding a new one) in your local git configuration for the project and then re-pushing. Gitea also includes a migration service to automatically pull repositories through, and can also be set-up to mirror other remote repos.

In terms of performance, I’ve found it quick to use and navigate (certainly faster than GitHub’s web interface) on a $10 VPS from Linode that I had anyway and on which I host many other services too.

It’s definitely worth a try if this is something you’re interested in. Let me know how you get on.

100 Days to Offload Challenge

Fri, 29 Jan 2021 17:46:00 +0000

I know that I’ve been a bit crap at updating my blog properly and consistently over the past few years. One of my new year’s resolutions this year is to get into the habit of writing more, and so #100DaysToOffload seems a good opportunity to challenge myself to make sure I do.

The guidelines for and the ideas behind the challenge are on the challenge’s website. There aren’t any rules really, but the essential message is to “Just. Write.”. So, I’ll do my best before the end of 2021, and given that I’ve already published two posts this year I’ll count this number 3.

I will try to keep things tech-related as much as possible. There’s only so much there that I can write about though, so I will probably also include bits from my life, books I read, things I’ve watched, etc.

If you want to follow along, you can subscribe to my RSS feed. If you need an RSS reader, I can definitely recommend Reeder 5, which is available for macOS and iOS - it’s fab. If you’re trying the challenge too, then let me know so I can check out your posts!

Project Gemini

Wed, 20 Jan 2021 17:00:00 +0000

Over the past few months I have been trying to use centralised “big tech” social media platforms less and instead immerse myself into the more community-driven “fediverse” of decentralised services that are connected (“federated”) using common protocols (e.g. ActivityPub). If you like, you can follow me on Mastodon (@wilw@fosstodon.org, recently migrated over from my old mastodon.social account) and Pixelfed (@wilw@pixelfed.social).

I’ve loved spending my time on these platforms - mainly due to the lack of noise and fuss, and more of a focus on sharing relevant content and interesting interactions with likeminded people (though of course this does depend on the instance you join).

One of the things I’ve seen talked about more and more is Gemini - and having learned about it and participated myself - I have come to love the ideas behind it.

Some people will remember the Gopher protocol - a lighter alternative to the web that was ultimately sidelined by most of the world in favour of the HTTP World-Wide Web in the early 90s. The Gemini protocol is newer, having started in 2019, but is inspired by Gopher. In particular, it aims to solve some of the problems experienced by the modern HTTP(S) web we know today - around complexity, privacy, and “bloat” - and focuses on providing a graph of usefully connected content.

Gemini “sites” (known as “capsules” or “stars”/“starships”) - the resources that form Geminispace - are located using the gemini:// URL scheme. Servers typically listen on port 1965: a homage to the NASA’s Gemini Project. Gemini text resources are similar to traditional HTML web pages in the sense that they can include links to other resources and provide structure and hierarchy through a markdown-like syntax. All Gemini resources must also be transferred using TLS.

The image below shows my own capsule (found at gemini://wilw.capsule.town). If you can’t open that link yet then read to the end of this post.

However, there are also significant differences. These .gmi files (and files with similar extensions served with the text/gemini MIME type) cannot indicate any styling instructions (as HTML often does with CSS) - and instead leaves the display and rendering of the file up to the client. In addition, whilst images can be served over the protocol, they cannot be included in (and rendered within) Gemini text files like they can in HTML. Similarly, there is no client-side dynamicity for these resources, such as the JavaScript included with most HTML web pages. It’s simple; the client just renders whatever is provided by the server, and that’s it.

The simplicity of the protocol - without styling, embedded images, client-side scripts, and more - offers a lightweight, ad-free, and content-oriented experience that is also available for low-powered devices and machines on slower networks. There is more of a focus on privacy (servers can’t track or fingerprint you beyond knowing your IP address), and the relative “smallness” of it and absence of big-tech presence certainly brings back some of the fun and novelty of the early web as we remember it.

I certainly recommend visiting the project’s website for more useful information.

Getting involved

Since Gemini resources are not served using HTTP(S), you can’t access them using a normal web browser (although you can use HTTP proxies such as Mozz’s Portal).

Instead, you’ll need a Gemini client. I use Amfora on my Mac and the Gemini Browser on my phone.

Once you have a client, you can view my own capsule by visiting gemini://wilw.capsule.town. If you’re interested in starting your own and want to see how mine is formed, you can view the project source.

I also recommend trying out the Gemini search-engine to discover what else lies in Geminispace. It is located at gemini://gus.guru.

Scaling serverless apps: some lessons learned

Sun, 03 Jan 2021 17:00:00 +0000

Building apps on serverless architecture has been a game-changer for me and for developers everywhere, enabling small dev teams to cheaply build and scale services from MVP through to enterprise deployment.

Taking advantage of serverless solutions - such as AWS’ Lambda, Google’s Cloud Functions, and Cloudflare’s Workers - means less resource is spent on traditional dev-ops and deployment and, especially when combined with tools like Serverless framework and its rich ecosystem of plugins, you can use the time instead to better develop your products. Let the provider worry about deploying your code, keeping your services highly available, and scaling them to meet the needs of huge audiences.

Serverless providers often bill by number of “invocations” (the number of times your functions are executed) and total execution time. This means you’re only paying for your services while they are actively being used, as opposed to the traditional server(-full?) model in which you have to pay to keep your services available 24/7. Given providers often have generous free tiers, you might not even start paying for anything until your service gets popular enough, which is always a nice problem to have.

However, before you get to that stage, it’s worth also being aware of some of the pitfalls that come with adopting a serverless-only approach and knowing you may later need to prepare to migrate your service further down the line. Many of these are by design; to help keep things quick to deploy and run, but others are less obvious or predictable. I’ll refer to AWS and Lambda (and Serverless framework) in this post, as it’s what I’m most familiar with, but the concepts should mostly translate to other providers.

It might not stay cheap

If deploying a web service to Lambda, it’s likely you’ll front it using API Gateway, and Serverless framework will set this all up for you at deploy-time. Whilst offering lots of nice features (authorizers, automatic TLS termination, integrations with Certificate Manager, and more), API Gateway also comes with a nice free tier.

However, you will start paying once your service gets popular enough. This means that on top of Lambda billing you for invocations and compute time, you’re also getting hit by bandwidth and API call costs. If by good (or bad) marketing your service ends up suddenly getting a lot of visitors - perhaps whilst you’re away for a week on holiday - AWS will happily and seamlessly scale to meet the demand, but you might get hit with a nasty bill on your return.

There are a number of posts online on this matter (example here), so it’s worth doing some research beforehand.

Performance

The link mentioned just above also makes note of the performance of Lambda functions. Whilst the technology makes development and delivery super easy, it comes with the downside of reduced performance.

To save costs Lambda keeps your function code out of memory. It will only provision compute power, locate and retrieve your code, and finally execute it when the function is actually invoked. This is known as a cold start, and invocations that start cold will cause your requests to take longer. If the function is executed frequently, Lambda will keep it “warm” and you’ll notice the requests are quicker.

Note that cold starts may not be a problem for Cloudflare Workers and Serverless framework has plugins to help mitigate them too.

AWS Lambda also has other overheads which may mean you notice a performance reduction compared to when running your app in a container or on a VM.

Limitations

Lambda imposes a number of limitations that you would not be subject to when deploying using a VM or containers. Many can be increased, but this would come with cost implications and require a chat with AWS support.

Language runtimes

Historically, Lambda only offered a fixed set of language runtimes and versions - including some of the most popular, such as Python, Node, Ruby, and Go. You can now provide your own runtime (e.g. if you wanted to write a service in Rust), but this requires you to do your own configuration first.

Function file size

AWS has a default limit of 75GB for storing all of your Lambda functions, dependencies, and layers. This seems like quite a lot, but if you had multiple NodeJS Lambda applications, each with large node_modules directories for dependencies, and considering Serverless framework versions each release by default, you might hit this limit sooner than you think and become unable to release any new code until you can find something to delete.

Request and response data

When using a Lambda function as part of a web service, it will naturally be involved with handling request data and emitting response data. This is one that got me confused for a while in an API I was building: there is a 6MB limit to the size of the request and response.

Whilst this won’t (and shouldn’t) be an issue for most REST APIs, a client for a project I worked on needed a large volume of data returned by a single call for daily automated data exports, which one day just stopped working by itself. Cloudwatch logs didn’t reveal any particular problems and API Gateway was only responding with non-descriptive errors. After some investigation it turned out that the response had trickled over the payload size limit. It’s worth considering and remembering this if you ever get hit with a similar problem.

Response structure

Remember that if your Lambda function does not return a correctly-formatted response, API Gateway will respond to the client with an error status code - even if the function executed successfully. Always return a response from your function since Cloudwatch won’t give much away in this scenario either (the function may have still executed successfully), which is another pain to debug.

Resource limits

When deploying an application using Serverless framework, a number of AWS resources are created on your behalf in a single “stack” using CloudFormation. On the surface, a single function could be associated with a large number of “resources” (e.g. the function itself, layers, IAM roles, Cloudwatch log groups, authorizers, API Gateway resources/methods, etc.).

CloudFormation limits stacks to 500 resources each, and if your application has more than a handful of functions (e.g. for handling different types of events) then you might very well this limit.

It can be mitigated by combining similar functions together (using arguments to define any differences), or by separating functions out to another Serverless application. Either way, you’d need to consider your strategy with this.

Closing remarks

Despite some of these fallbacks, I can still very much recommend taking advantage of serverless architecture when creating your apps, and I will certainly continue to build on these platforms in many areas of my work.

The purpose of this post is just to raise awareness around some of the invonveniences I have faced during my time working with this technology, but I still love its power and flexibility.

React Query

Tue, 15 Dec 2020 17:00:00 +0000

If you write React web apps that interface with a backend web API then definitely consider trying React Query.

The library makes use of modern React patterns, such as hooks, to keep code concise and readable. It probably means you can keep API calls directly inside your normal component code rather than setting-up your own client-side API interface modules.

React Query will also cache resolved data through unique “query keys”, so you can keep transitions in UIs fast with cached data without needing to rely on redux.

You’ll still need to write the actual requests yourself (e.g. using fetch or axios) but using this certainly takes away a lot of the pain in building smooth asynchronous web apps.

To try it out install it with your usual Node package manager and begin with the quick start guide.

JS Tidbit: Nullish Coalescing

Fri, 20 Nov 2020 17:00:00 +0000

This short post introduces a useful JavaScript operator to help make your one-liners even more concise.

The specification was added formally in the 11th edition of ECMAScript. It is implemented as a logical operator to selectively return the result of one of two expressions (or operands) based on one of the expressions resolving to a “nullish” value. A nullish value in JavaScript is one that is null or undefined.

In particular, the operator - given by ?? - will return the right-hand side if the left-hand expression resolves to null or undefined, and otherwise returns the left-hand side.

const x = y ?? z;

In the example above, z will be returned if y is “nullish”, and otherwise y will be returned.

Nullish coalescing is similar to (but stricter than) the more commonly-seen logical OR operator - given by || - which returns the result of the right-hand side expression if the left-hand side resolves to any falsy value, which includes nullish ones in addition to the boolean false, empty string (''), 0, NaN, etc.

More info is available on MDN and Wikipedia discusses null coalescing in other languages.

JS Tidbit: Optional Chaining

Sat, 10 Oct 2020 17:00:00 +0000

JavaScript has lots of handy tools for creating concise code and one-liners. One such tool is the optional chaining operator.

The optional chaining operator is useful for addressing an attribute of a deeply-nested object in which you cannot be fully certain that the successive levels of the object are valid at run-time.

For example, consider the following object.

const person = {
  name: 'Harry',
  occupation: 'student',
  enrolmentInformation: {
    contactDetails: {
      email: 'harry@hogwarts.ac.uk',
      address: {
        firstLine: '4 Privet Drive',
        postCode: 'GU3 4GH'
      }
    }
  }
};

In order to safely (i.e. if you cannot guarantee each object level at run-time) read the nested postCode attribute, you could do so like this, using the logical AND operator:

const enrolmentInfo = person && person.enrolmentInformation;
const contactDetails = enrolmentInfo && enrolmentInfo.contactDetails;
const address = contactDetails && contactDetails.address;
const postCode = address ^^ address.postCode;

Even in its single-line equivalent, this very quickly gets un-readable and messy.

The optional chaining operator (?.) makes this much easier:

const postCode = person?.enrolmentInformation?.contactDetails?.address?.postCode;

Or, using object destructuring and a logical OR:

const { postCode } = person?.enrolmentInformation?.contactDetails?.address || {};

Essentially, at each successive level, the optional chaining operator checks that the current attribute is non-nullish (i.e. not null or undefined) before proceeding to the next. If it does try to access a nullish attribute, the expression short-circuits and returns undefined, ignoring the rest of the chain.

Optional chaining also works in other scenarios:

const x = doubleArray[3]?.toFixed(2); // accessing array elements
const y = maybeArray?.[0]; // accessing an array that may exist
car.openDoors?.(); // calling methods on nested objects

For more info, see the MDN reference.

Command-line bookkeeping in Animal Crossing

Sat, 23 May 2020 17:00:00 +0000

I recently stumbled across an article on Hacker News discussing the pros of basic personal accounting using GnuCash - a free and open-source desktop accounting program. The article was interesting as the data geek in me resonated with the notion of being able to query the information in useful ways, particularly after having used the system for enough time to accumulate enough financial data.

The comments on the article’s post also mentioned another tool, Ledger. Whilst GnuCash allows users to input transactional and account information as well as reports, Ledger’s focus is only on the reports - a key feature of this CLI tool is that the actual bookkeeping is made directly (or through other tools) into a text file, which Ledger only reads from and never otherwise touches. Both programs work on the principle of double-entry bookkeeping, but some of the key positives of Ledger are its speed (even when working with several decades’ worth of financial data) and its innate ability to be combined with other useful UNIX tools - both for data input and, if necessary (Ledger’s own reporting outputs are very powerful), output.

The fact that it relies on only a single human-manageable text file, which can contain the data for any number of accounts, and which can easily be version-controlled, is also a bonus in my eyes. The author of Beancount - a similar program worth a look at - also wrote a good article about the benefits of command-line accounting.

I do not come from an accounting or bookkeeping background, but I was keen to start trying it out for myself, and wanted to make sure that I could get the grips of the basics before diving in too far and making too many mistakes that I’d have to go back and change later. I currently use Xero for some business work, which happily handles (nearly) everything for me - invoicing, payments, automatic bank feeds and reconciliations, and more. It sometimes feels a bit slow and clunky, though, and its general ease of use does teach much about what actually goes on behind the scenes. Ledger seemed like a great alternative to use for personal finance - a powerful tool for viewing key information at a glance, combined with a familiar and standard method to “input” data. It also gives me the feel that I am more intimate with my accounts and also a richer understanding of my financial model.

Moving forwards from my initial experiments, I had three main aims:

Learn the basics of bookkeeping (specifically with Ledger in mind);
Learn how to run reports on “the books” using Ledger (e.g. profit & loss, and balance sheets);
Learn how to actually apply this to something tangible (real life assets).

This post represents some of my learnings and attempts to apply them to a game that features a number of usefully illustrative financial transaction examples, as we’ll see later.

To reiterate - I am not a bookkeeper, so my explanations are simply the way I describe the key concepts to myself. I acknowledge that my descriptions may not be 100% technically accurate from an accounting perspective, but I hope they will provide a decent understanding to others also starting out. This is still a learning exercise for me, so if anyone has any tips or points for improvement it’d be great to hear from you.

The rest of this post assumes that Ledger is installed. There are instructions for this on GitHub.

Command-line accounting & bookkeeping basics

I’m not going to go into too much detail here, as there is lots of information on Wikipedia about bookkeeping, double-entry bookkeeping and debits and credits. For me (probably due to my background) the best introduction was actually Ledger’s own documentation, which I definitely recommend reading.

Essentially, in someone’s double-entry world, the core concept is that money (or whatever assets/currencies/commodities you want to journal, but I’ll talk about money here and, later, bells) - like matter and energy - can’t be created or destroyed. In every transaction money must move from one place (account) to another such that the balance of the transaction equals zero, and therefore the balance of all transactions and accounts (books) is also zero (“balancing the books”). To put it another way, each transaction involves debiting one or more accounts and crediting one or more other accounts, such that the total amount of the credits equals the total amount of the debits. When adding a transaction to a ledger a bookkeeper signifies this by indicating the account(s) that have been debited and the account(s) that have been credited in the transaction.

In Ledger we do exactly the same by writing the transaction in the plain text file for our ledger:

2020/05/21  Payee or description
  Account1  £50
  Account2  £-50

In this example, we show that money has been taken from (credited) Account2 and given to (debited) Account1. Note that in bookkeeping a “debit” is when an account receives a benefit and a “credit” is when an account gives a benefit. This is often the opposite to how people understand debit and credit, and there is a good explanation about this on Wikipedia.

Note that the balance of the transaction is 0 (there is a credit of £50 and a debit of £50).

In bookkeeping, accounts don’t have to (and usually don’t) refer to just “bank accounts”. Instead, accounts can be thought of as “categories” or “buckets” of money, of which there are four main types (e.g. for a person):

Asset (what the person actually owns, such as cash, bank accounts and property)
Liability (what the person owes to other people/entities)
Income (where money comes to the person from)
Expenses (where money going from the person goes)

There are also “equity” accounts, which indicates the person’s net worth. I won’t talk much about these types of accounts, but when first using Ledger yourself this type of account is what you’ll use to represent your current “net worth” being transferred into your accounted-for assets and liabilities.

Note also that in Ledger we deal with positive and negative numbers to represent debit and credit respectively. Often bookkeepers don’t use signs to indicate debit/credit and instead rely on the understanding that debit and credit mean different things for different types of accounts. For example, that a debit to an asset account is an increase, whereas a debit to a liability account is a decrease. This can be confusing so I won’t go further into this!

In Ledger, we can indicate the account category and name to make the ledger more useful for reporting. E.g. to make the previous transaction better:

2020/05/21  Tesco
  Expenses:Groceries   £50
  Assets:Bank Account

We can leave one value line blank in each transaction and Ledger will automatically balance it when running reports - in this scenario by marking a negative (“credit”) £50 in the final line. This transaction shows account changes in two categories - an expense account and an asset account. When we went to Tesco we used our bank card to buy groceries, which debited our Expenses:Groceries account £50 and credited our Assets:Bank Account account the same amount. The expenses account does not refer to anything we “own”; moreover a debit in an expense represents a loss.

On the Tesco side, they may have something along these lines:

2020/05/21  Customer123
  Assets:Bank Account     £50
  Income:Sales:Groceries  £-50

Tesco might associate my card with a unique customer number (Customer123) and mark the transaction as a credit in their income account and a debit in their bank account - essentially the opposite of my own record. Note that in this example Tesco (a more complex entity than myself) have categorised their income account further by indicating an additional level of precision, allowing them to run reports separately on all income, income from sales, and income specifically from groceries sales.

Why Animal Crossing?

I (like many others around the world) have recently become engrossed in Animal Crossing: New Horizons - a video game for the Nintendo Switch. The primary concept of the game is that your character, with the help of travel company Nook Inc, moves to a deserted island, along with a couple of other characters (at first), in order to build a new home and life.

When you first arrive at your new island, Tom Nook (owner of Nook Inc) introduces you to the other islanders who have also come along, and to Timmy and Tommy (Tom Nook’s apprentices). Tom, Timmy, Tommy (and other characters who later join) stay with you on your island to help you and your fellow islanders out on your new adventure.

Throughout the game the player is involved in a number of different types of financial transaction and must take out loans, make sales and purchases in order to progress. As I played, it occurred that this would be a great way to learn and explain basic bookkeeping with Ledger.

In the game the player works with a number of different asset types. The main currency (money) is Bells but there is also a loyalty program called Nook Miles which allows the player to gain points (like air miles or a points card for a store). Luckily Ledger can handle different currencies and commodities with no problem - and will even make conversions where it can too - so we can deal with all of our Animal Crossing assets in a single ledger.

The rest of this post walks through some of the early stages of the game, using it to illustrate basic bookkeeping with Ledger.

Starting off the ledger

After landing, Tom Nook reveals your accomodation - a tent - and that you have inadvertently already got yourself into debt (from his perspective!). He tells you that the cost of your travel and moving has cost you some Nook Miles loyalty points, which you’ll need to pay back in order to move to the next stages of the game. He does give you the option to pay nearly ten times the amount in Bells instead, but it’s quicker to get the Miles.

We now have a financial position (despite it being “negative”). This means that we can add it to our ledger so that we have a clear record of the transaction and can begin to account for all of our finances. Even though you haven’t physically given or received anything yet, you now have a new liability (loan) that has been credited in turn for your moving expense (cost). Inidcate this by creating a new text file (e.g. ledger.dat) and writing the following:

2020/05/20  Moving Fee
  Expenses:Travel      5000 MILE
  Liabilities:NookInc  -5000 MILE

Our expenses account has been debited (to show that we have “paid” for something), and our liabilities account has been credited (to indicate that someone else paid on our behalf, and to whom we now owe something). In the real world this would be a similar transaction to when paying for something using a credit card.

Now we have a transaction entry we can start to run reports. As long as you have Ledger installed, you can run the following to see a register (sort of a transaction statement):

~ ledger -f ledger.dat register
20-May-20 Moving Fee            Expenses:Travel          5000 MILE   5000 MILE
                                Liabilities:NookInc     -5000 MILE           0

And a current balance (“balance sheet”) of all of our accounts:

~ ledger -f ledger.dat balance
          5000 MILE  Expenses:Travel
         -5000 MILE  Liabilities:NookInc
--------------------
                   0

We can see that Ledger has created the two accounts for us and has checked that they are correctly balanced. At a glance we can easily observe the current state of our finances.

In future I may sometimes omit the -f ledger.dat for brevity. You can also do this, for which there are simple instructions on the Arch Wiki.

Paying off the first loan

Nook Miles can be earned by unlocking achievements in the game (such as picking weeds, fishing, and catching insects). As you unlock an achievement its Nook Miles value will immediately be given to you (i.e. “debited to your assets” - these Miles are now something we “own”). For example, if you were to receive 200 Nook Miles for an achievement then this can be recorded as a new transaction (beneath the first one) in your ledger:

2020/05/20  Unlocked achievement
  Assets:NookMiles     200 MILE
  Income:Achievements

We can check our balances again to see the new accounts. Note that we don’t need to declare our accounts anywhere; Ledger will automatically create them for us as it finds them.

~ ledger bal
            200 MILE  Assets:NookMiles
           5000 MILE  Expenses:Travel
           -200 MILE  Income:Achievements
          -5000 MILE  Liabilities:NookInc
--------------------
                   0

We can now also use the balance report to display our current profit and loss statement (P&L) by limiting its response to just our income and expense accounts:

~ ledger bal Income Expense
           5000 MILE  Expenses:Travel
           -200 MILE  Income:Achievements
--------------------
           4800 MILE

Income accounts have a negative balance in Ledger to indicate how you have credited someone else in order to debit another account (e.g. an asset). Expense accounts have positive balance to show that you have debited someone else in order to pay for something or a service. Therefore, this “positive” P&L balance means that we are still loss-making for the report period.

I have used quotation marks (") a bit to try to illustrate meaning without upsetting accountants. It’s a complex world and I find that thinking of things - like Ledger does - in positive and negative numbers makes things easier at first.

Eventually, after playing the game enough, we will have earned enough Miles to pay back our loan to Nook Inc. This state will be represented by additional transactions in the ledger and an eventual balance that shows that our current assets are greater than our liabilities:

~ ledger bal Assets Liabilities
           6200 MILE  Assets:NookMiles
          -5000 MILE  Liabilities:NookInc
--------------------
           1200 MILE

and the associated P&L to show that we’re now in profit by the same amount:

~ ledger bal Income Expense
           5000 MILE  Expenses:Travel
          -6200 MILE  Income:Achievements
--------------------
          -1200 MILE

When we visit Resident Services to pay back our Miles we log the transaction beneath the first two in the ledger file:

2020/05/21  Moving fee repayment
  Liabilities:NookInc  5000 MILE
  Assets:NookMiles

And can now show that we now have no more liabilities:

~ ledger bal Assets Liabilities
           1200 MILE  Assets:NookMiles

For interest, we can view a register (reg) of all transactions on our assets accounts to date, where the final value is the balance indicated above:

~ ledger reg Assets
20-May-20 Unlocked achievement  Assets:NookMiles           200 MILE     200 MILE
20-May-21 Unlocked acheivements Assets:NookMiles          6000 MILE    6200 MILE
20-May-21 Moving fee repayment  Assets:NookMiles         -5000 MILE    1200 MILE

Depending on your play style or needed accuracy you may want to just record achievements (and anything else) in bulk in a single transaction (as in the second posting above) rather than as individual transactions.

From now on, this post assumes that new transactions continue to be entered beneath each other in your ledger text file.

Multiple currency and commodity types

As you play more you’ll quickly realise that you spend a lot of game time with liabilities, which should be a familiar concept to those who have real-life mortgages and credit cards. As soon as you’ve finished paying off your moving fee you’ll be invited to take out a new loan (this time in Bells - the main currency) to upgrade your tent to a house, which you can represent as expected:

2020/05/23  House upgrade
  Assets:Property      98,000 BELL
  Liabilities:NookInc

Running a balance sheet report now shows your assets across both Bells and Miles:

~ ledger -f bal
         98,000 BELL
           1200 MILE  Assets
           1200 MILE    NookMiles
         98,000 BELL    Property
           5000 MILE  Expenses:Travel
          -6200 MILE  Income:Achievements
        -98,000 BELL  Liabilities:NookInc
--------------------
                   0

From your perspective, Bells are “currency” and Miles are “commodities”. From Ledger’s view they’re just two types of “things” that you can account for, and doesn’t differentiate between them. Note also that Ledger has learned how we want things displayed based on our early transactions: the value always comes before the commodity symbol, and Bells are displayed with commas to separate every third digit whereas Miles are displayed uninterrupted.

Although your new house is an asset (subclassed as a property), it also has the associated liability (loan) that you’ll gradually pay off - this time through your ABD (as we’ll see later) - as you did with the Nook Miles. As such, your net worth (assets minus liabilities) is still currently only the leftover Miles you have from your first loan:

~ ledger bal Assets Liabilities
         98,000 BELL
           1200 MILE  Assets
           1200 MILE    NookMiles
         98,000 BELL    Property
        -98,000 BELL  Liabilities:NookInc
--------------------
           1200 MILE

Income and expenses

At the start of the game, Timmy will buy items from you (e.g. things you find around the island or are given) in exchange for Bells.

Selling such items (e.g. fruit you find on the island) is simply represented as a debit in your assets and a credit in your income (which you’ll remember is always “negative”). When you make the sale to Timmy (or in Nook’s Cranny) the Bells are transferred directly to you in person (i.e. “cash”):

2020/05/24  Timmy
  Assets:Cash   8,000 BELL
  Income:Sales

When you sell items in this way you increase your net worth, since your assets have increased but your liabilities have not changed.

As the game progresses you’ll eventually unlock Nook’s Cranny - Tommy and Timmy’s shop for the island’s residents. Here you can continue to sell things, but you can also now buy items from their rotating stock. When you buy items (e.g. furniture) from the shop, the cash account is instead credited:

2020/05/25  Nook's Cranny
  Expenses:Furniture   3,000 BELL
  Assets:Cash

Savings accounts

Now you have accumulated some Bells you can pay off your house loans. You’ll be able to continue to take out progressively larger (though thankfully interest-free) loans to upgrade your home a number of times each time you pay one off, until you own a house with multiple rooms, an upstairs and a basement.

Paying off your house loans is done through the ABD (Automatic Bell Dispensor) function available on the Nook Stop machine in Resident Services. In the ABD you have something similiar to a savings account, which you can make Bell deposits to and withdrawals from at any time.

To pay off your loans you first need to deposit the Bells that you have on you into your savings account. This would be recorded in your ledger as two postings involving asset accounts (a debit in your savings and a credit in your cash balances):

2020/05/26  Savings transfer
  Assets:Savings  4,000 BELL
  Assets:Cash

You can now credit the savings balance to pay off some of the loan:

2020/05/26  Loan repayment
  Liabilities:NookInc  4,000 BELL
  Assets:Savings

Note that these two transactions have not affected your net worth: even though you have paid off some of the loan, your own assets have also reduced by the same amount. Taking out and repaying loans only “moves” money between assets and liabilities accounts.

Accounts receivable

Nook’s Cranny closes at 10PM each evening, and purchases and sales that day should ideally be made before that time. However, the shop also offers a “drop-off box” facility that allows you to sell items (at a reduced price) whilst the shop is closed.

Sales made this way do not transfer cash directly to you, as they do when you sell directly when the shop is open; instead the Bells accumulated by selling to the drop-off box are transferred to your savings account when the shop opens the following day.

As such, when using this service, although you have sold items, the Bells are not tangibly (physically) yours to spend straight away. They are still assets (i.e. they increase your net worth), but they should be recorded separately to your cash and savings accounts by using an “accounts receivable” assets account - which represents money you are owed:

2020/05/27  Nook's Cranny
  Assets:Accounts Receivable  8,000 BELL
  Income:Sales

If you run a business that makes sales through invoices, checking a receivables account is a nice way to view money that (hopefully) will be paid to you soon:

~ ledger reg Receivable
20-May-27 Nook's Cranny    Assets:Accounts Receivable   8,000 BELL   8,000 BELL

The following day, when the shop re-opens, you get a call to let you know that you have now received the funds to your savings account, which you add to your ledger:

2020/05/28  Nook's Cranny
  Assets:Savings               8,000 BELL
  Assets:Accounts Receivable

Ledger actually has nicer ways of managing this for you (e.g. effective dates), but this is a simple example to illustrate receivable accounts.

Accounts payable are the opposite; they are classed as liabilities since this is money you owe. In many cases you wouldn’t normally have a generic “payable” account - instead opting to use specific payables for individual liabilities (e.g. a particular credit card or mortgage).

Joint accounts, payees and tags

Your island will contain a number of streams and cliffs that need to be traversed using tools that you can make yourself. However, Tom Nook also gives the option to purchase bridges and inclines to allow you and fellow island residents to more easily get around.

Once the plan for the building work (e.g. a bridge) is laid-down, Lloid collects funds for the work before it can begin. Funds can be contributed by yourself, NPC islanders, or by other players who visit your island. Like me, you’ll likely find that your islander neighbours are less than generous when it comes to infrastructure investment, however it is interesting to consider this concept as a joint account shared by multiple people or entities.

From my perspective, there is probably no right or wrong way to bookkeep this. The constructed bridge is not really an asset that you own (since it is a public object), however you’ll certainly need to contribute towards it if you want it built any time soon. As such, you may want to simply consider it as an expense:

2020/06/01  Lloid
  ; :bridge1:
  Expenses:Infrastructure   20,000 BELL
  Assets:Cash

The commented :bridge1: is a tag for the transaction (read more about how Ledger can handle tags). If I later made further payments towards this bridge I could re-use the same tag and then, when runing reports later, use Ledger to view all of my payments towards this particular bridge:

~ ledger reg %bridge1
20-May-30 Lloid                 Expense:Infrastructure  20,000 BELL  20,000 BELL
                                Assets:Cash            -20,000 BELL            0
20-Jun-01 Lloid                 Expense:Infrastructure   5,000 BELL   5,000 BELL
                                Assets:Cash

I think it’d be tricky to account for this yourself in other ways, since you do not know who has contributed what. As such your ledger would not be aware of the total cost of the bridge before it has been built.

Considering the process from Lloid’s perspective, my first contribution from above could be marked as the following in his own ledger:

2020/06/01  Bridge payment
  ; :infrastructure:southBridge:
  ; Payee: Will
  Assets:Cash             20,000 BELL
  Income:Contributions

Here Lloid has also tagged the transaction (though he refers to the bridge differently) and also adds a special Payee tag to the transaction. When more people have contributed he can clearly report on who has contributed what for this particular bridge:

~ ledger reg %southBridge --by-payee
20-Jun-02 Cleo                  Assets:Cash             10,000 BELL  10,000 BELL
                                Income:Contributions   -10,000 BELL            0
20-Jun-03 Punchy                Assets:Cash             15,000 BELL  15,000 BELL
                                Income:Contributions   -15,000 BELL            0
20-Jun-01 Will                  Assets:Cash             30,000 BELL  30,000 BELL
                                Income:Contributions   -30,000 BELL            0

He can also quickly see all infrastructure contributions made by me:

~ ledger bal @Will and %infrastructure
         30,000 BELL  Assets:Cash
        -30,000 BELL  Income:Contributions
--------------------
                   0

Note that if we omit the and, Ledger would read this as “@Will OR %infrastructure”, which gives a different balance (i.e. a balance of any transactions involving me OR infrastructure):

~ ledger bal @Will %infrastructure
         55,000 BELL  Assets:Cash
        -55,000 BELL  Income:Contributions
--------------------
                   0

Later, Lloid would use his contributed assets to pay for the materials and labour required to build the bridge (keeping some in his savings account as profit, of course):

2020/06/05  Bridge build
  ; :infrastructure:southBridge:
  Expenses:Materials             60,000 BELL
  Expenses:Labour                40,000 BELL
  Assets:Savings                 29,800 BELL
  Assets:Cash

Tagging transactions in these ways allows for powerful queries when reporting later on. Lloid can now easily build comprehensive reports for all transactions for a particular build, or across multiple builds. In the real world this type of tagging could be useful if you let out property. For example, you could tag all transactions involving the property (e.g. rent, expenses, mortgage payments, etc.) to make tax returns much less painful.

Investments and capital gains (& losses)

Each Sunday morning your island will be visited by Daisy Mae, who sells her family’s special turnips. The main focus of this feature is to take part in the “Stalk Market”, which is arguably one of the quickest ways to accumulate Bells in the game.

The player buys the turnips with Bells (the exchange rate for which varies every week, but is usually at around 100 Bells per turnip). Then, during the week, Nook’s Cranny can buy the turnips from you. Nook’s Cranny’s turnip prices change twice per day, and can vary wildly. The goal of the player is therefore to try to sell the turnips at a greater price than what they were bought for.

There is a catch, however, for if the player does not sell the turnips within a week, they rot and can then only ever be sold at a fraction of what was initially paid for them.

Ledger has first-class support for understanding investments like these. Let’s say that we buy 100 turnips at a price of 100 Bells each:

2020/06/07  Daisy Mae
  Assets:Investments  100 TURNIP @ 100 BELL
  Assets:Cash

Ledger will automatically calculate the Bell cost based on the turnip volume and lot price, and will credit the cash account accordingly to balance.

When buying turnips from Daisy Mae, despite them being a type of food that you can eat in the game, they are usually considered a non-edible commodity. As such we treat them as just another type of asset that we still own (e.g. similar to buying oil, gold, or stock in a company). If you were to buy the turnips for purposes other than selling later then you’d probably treat the transaction as an expense instead.

If we now run a balance report of our assets (we include liabilities here so we don’t forget about what we still also owe), we can see Ledger accounting for all of our different commodities:

~ ledger bal Assets Liabilities
        132,000 BELL
           1200 MILE
          100 TURNIP  Assets
         26,000 BELL    Cash
          100 TURNIP    Investments
           1200 MILE    NookMiles
         98,000 BELL    Property
          8,000 BELL    Savings
        -94,000 BELL  Liabilities:NookInc
--------------------
         38,000 BELL
           1200 MILE
          100 TURNIP

The plan now is to try and sell the turnips at a lot price of more than 100 Bells, and so we check at Nook’s Cranny each day for the current turnip price.

Eventually, we find a price we are happy with (e.g. 200 Bells), and we sell all of our turnips. Since Ledger doesn’t let us create Bells out of nowhere, to indicate this appreciation (we would now have twice as many as we did before) we need to make a credit in an income account to represent the capital gains:

2020/06/06  Nook's Cranny
  Assets:Cash         20,000 BELL
  Assets:Investments  -100 TURNIP {100 BELL} @ 200 BELL
  Income:Capital Gains

In Ledger, we indicate the cost that we originally bought the turnips for, as well as what we sold them for. As usual it will work out the capital gains for us. If we were instead to make a loss on our turnips, we would debit our cash account with whatever we did make back and also an equity account to indicate a loss in our net worth.

Now, if we run a profit and loss report for the past few days, we would see that we are now in profit!

~ ledger bal Income Expenses
         28,000 BELL
           5000 MILE  Expenses
          3,000 BELL    Furniture
         25,000 BELL    Infrastructure
           5000 MILE    Travel
        -86,000 BELL
          -6200 MILE  Income
          -6200 MILE    Achievements
        -10,000 BELL    Capital Gains
        -76,000 BELL    Sales
--------------------
        -58,000 BELL
          -1200 MILE

Ledger allows you to track the the prices for various commodities using a separate “price database” file, which provides an historical record of the prices for commodities in your portfolio, and will be automatically consulted by Ledger when running reports. Read more about this in the documentation.

“Currency” exchange

I won’t go into much detail here, since we’ve already covered exchanging assets a few times. However, the game provides in-built mechanisms for exchanging Nook Miles for Bells. From the Nook Stop one can buy Bell vouchers for 500 Miles each, which provide a convenient way to convert your Miles to Bells.

Please note: there is probably a more elegant approach to this process, so if you have any better ways of accounting for these transactions it’d be great to hear them!

However, in my view, when we buy the vouchers we are essentially spending our own Miles:

2020/06/07  Nook Stop
  Expenses:NookInc  1000 MILE
  Assets:NookMiles  -1000 MILE
  Assets:Vouchers   2 BV @ 500 MILE
  Income:Vouchers   -2 BV @ 500 MILE

And then, when we sell the vouchers in Nook’s Cranny we are spending the vouchers in return for 3,000 Bells each:

2020/06/07  Nook's Cranny
  Expenses:NookInc  2 BV @ 3000 BELL
  Assets:Vouchers   -2 BV @ 3000 BELL
  Assets:Cash       3000 BELL
  Income:NookInc    -3000 BELL

This approach means our accounts remain balanced and we don’t have outstanding assets reported.

Reporting balance sheets & profit and losses

That’s all I’m going to cover in this post about recording transactions, but I will finish off with a couple of final useful types of reports using Ledger we can run now that we have a fuller set of accounts.

When running reports it is often useful to scope to a particular time period - e.g. to focus on a particular set of transactions in a given tax year. Ledger nicely supports querying specific timeframes for transactions and balances. For example, to view our P&L for only May:

~ ledger bal Income Expenses -b "2020/05/01" -e "2020/06/01"
         23,000 BELL
           5000 MILE  Expenses
          3,000 BELL    Furniture
         20,000 BELL    Infrastructure
           5000 MILE    Travel
        -76,000 BELL
          -6200 MILE  Income
          -6200 MILE    Achievements
        -76,000 BELL    Sales
--------------------
        -53,000 BELL
          -1200 MILE

Or to view only the transactions involving bridge1 from June onwards:

~ ledger reg %bridge1 -b "June 2020"
20-Jun-01 Lloid                 Expenses:Infrastructure    5,000 BELL    5,000 BELL
                                Assets:Cash               -5,000 BELL             0

If I wanted to see all capital gains income for the year-to-date (e.g. for calculating capital gains tax):

~ ledger bal "Capital Gains" -b "January"
        -10,000 BELL  Income:Capital Gains

Again, I’ll mention the Ledger documentation, which goes into reporting in much more detail.

Conclusion

I hope this has been a useful insight into how command-line accounting can work and how it might be beneficial when using it for managing your own real-world (or non-real!) finances.

Although there are definitely “wrong” ways to do bookkeeping, as long as you follow the basic principles then the way you keep your books is down to what works for you. If this is something you want to try yourself then, like a diet, it’s important to find an approach that means you can stick to it - by choosing a set of accounts that gives enough detail without being too specific to manage, and by finding the right time balance spent in logging your transactions and reconciling against your real world accounts.

I try to update my personal ledger every few days, and now find it much quicker to simply update the text file than navigating around GUI tools and Xero - even with their fancy auto-reconciliation. Running reports is lightning fast and having a single text file as a source-of-truth means it can easily be synced between devices for editing, or checked into source control.

I’ve been using Ledger for a while now (I don’t have a very complicated financial life), and I’ve found myself agreeing with others who have tried it - I feel that I understand my finances more thoroughly, and it is hugely satisfying adding postings to my ledger and observing how this affects the accounts and the reflections to their real-world counterparts.

Kubernetes Cluster: Essentials

Sun, 02 Feb 2020 17:00:00 +0000

This note documents the set-up of a k8s cluster from scratch, including ingress and load-balanced TLS support for web applications. It’s mainly for myself to revisit and reference later on. The result of this note is not (quite) production-grade, and additional features (e.g. firewalls/logging/backups) should be enabled to improve its robustness.

Several cloud providers offer managed k8s services (including Amazon EKS, GKE, Digital Ocean, etc.). Whilst these would be recommended for sensitive or production workloads, I wanted to create my own provider-independent cluster in order to understand the ins and outs.

Infrastructure

Create the underlying infrastructure for the cluster by provisioning at least two new instances - all with at least 2GB memory and 2 CPUs. I used the latest Ubuntu LTS.

One of the new instances (kube1) will be the “control” node, and the others (kube2+) will be the “worker” nodes. In this scenario the “control plane” is made up of just one node - kube1.

In larger clusters the control plane can be made to be highly available by using multiple control nodes in different availability zones.

Enable private networking between the nodes and ensure they can all access the internet using a public IP or through another interface (e.g. a NAT). If you use firewalls ensure the nodes can communicate with each other via their private IPs.

Dependencies

On all nodes (control and workers), install Docker and k8s binaries:

Add Google apt keys: wget -qO - https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Add Google repo: add-apt-repository ‘deb http://apt.kubernetes.io/ kubernetes-bionic main'
Update local repos: apt update
Install docker.io, kubelet, and kubeadm: apt install docker.io kubelet kubeadm

On the control node only (kube1) also:

Install kubectl: apt install kubectl

Set-up the cluster

On the control node (kube1) prepare the cluster:

As root, initialise the cluster: kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.131.106.38
- Replace apiserver-advertise-address with the private IP for the master node
- Setting --pod-network-cidr allows Flannel (the CNI plugin we’re using here) to later allocate IP addresses to pods in your cluster from this range
Create a new user to manage kubernetes: useradd kubeuser
Create a new .kube directory: mkdir /home/kubeuser/.kube && chown kubeuser:kubeuser /home/kubeuser/.kube
Copy in the default config file: cp /etc/kubernetes/admin.conf /home/kubeuser/.kube/config && chown kubeuser:kubeuser /home/kubeuser/.kube/config
As the kubeuser user, install the Flannel network fabric by apply the appropriate YAML: kubectl apply -f kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Finally, back as root, get and copy the kubernetes join command: kubeadm token create --print-join-command

Now join the worker nodes to the new cluster:

As root, copy and paste and run the output from the last command on the control node on each worker node.

After a few minutes kubectl get nodes (as the kubeuser user on the control node) should list the other nodes in the cluster.

From now on, all kubectl commands should be issued by kubeuser on the control node, and all kubeadm commands by root on the control or worker nodes.

At this point you have a working cluster ready to create pods, deployments and services. The next sections go on to set up the cluster to allow exposing, securing, and load-balancing services for access from outside the cluster.

Install Helm

It’s a good idea to install Helm (a good way to install more complex packages and services on your cluster) according to the official website.

Load-balancing and ingress

We’re now ready to route external traffic to the cluster.

Create a load balancer

Use your provider’s load balancer service (or create your own load balancer using a reverse-proxy like Nginx) and make a note of the IP address (or host name) of the load balancer server. Add your kubernetes worker nodes to your load balancer’s distribution group.

If you have a domain name you want to associate with a service on your cluster, update your DNS settings to point the domain to the load balancer.

Next install the nginx-ingress k8s ingress controller via Helm (inserting the IP address of your load balancer).

helm install nginx-ingress stable/nginx-ingress \
--set controller.replicaCount=2 \
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP=“”

Check the services (kubectl get services) now running and note the HTTP/HTTPS ports used by the new ingress controller. Update your load balancer to use these ports as the forwarding ports in the load balancer settings (e.g. 80 -> 32345, 443 -> 31454).

Finally create an ingress resource for the domain/service by applying the following (replacing the relevant details for your domain name (host) and the service name and port requests should be routed to).

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: primary-ingress
annotations:
  kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: 
    http:
      paths:
          - backend:
            serviceName: 
            servicePort: 
    - http:
      paths:
          - backend:
            serviceName: nginx # this is default
            servicePort: 80

This ingress forwards all requests to the host through to on . All non-matched requests (i.e. to other domains) will be sent to the default backend - in this case a simple nginx service that returns 200 response codes (this is important to ensure load balancer health-checks pass).

Visiting in your web browser should now enable you to reach the service.

Securing your ingress

Currently the ingress we created only accepts traffic over HTTP on port 80. To secure traffic (i.e. use HTTPS to access services), either terminate TLS at your load balancer and add relevant certificates there, or allow your cluster to handle the TLS connection, as described below.

You can add HTTPS for services you want to expose through your load balancer. To begin, ensure that the domain is pointing to your load balancer (as above), and that you can reach your service through the load balancer using your domain name. Finally ensure that port 443 traffic to your load balancer is routed to the HTTPS port of the ingress controller service.

We use cert-manager to automatically manage certificates for us through LetsEncrypt. Install cert-manager:

Create a new namespace: kubectl create namespace cert-manager
Apply the needed definitions: kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml
Verify pods come online (the webhook one may take longer): kubectl get pods --namespace cert-manager

Once installed, cert-manager is formed from Issuer objects (determines how and where certificates should be issued from) and Certificate objects (determines the Secrets which contain the certificate values - these can be autorenewed by cert-manager).

First, create an issuer that uses letsencrypt’s production ACME service:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: 
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    # An empty 'selector' means that this solver matches all domains
    - selector: {}
      http01:
        ingress:
          class: nginx

Next, create a certificate that uses this issuer:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: certificate-prod
  namespace: default
spec:
  secretName: domain-tls
  issuerRef:
    name: letsencrypt-prod
  commonName: 
  dnsNames:
  -

Finally, update the ingress we created earlier to add a tls section that references the secret created by the Certificate:

...
spec:
  tls:
  - hosts:
    - 
    secretName: domain-tls
  rules:
  - host: 
    http: ...

After a few seconds your certificate should be issued and your service will be available via HTTPS through the load balancer.

Go backends on Now

Tue, 20 Aug 2019 17:00:00 +0000

ZEIT’s Now service is great for deploying apps and APIs that are able to make use of serverless execution models, and I use it for many of my projects (including this website, at the time of writing).

I recently needed to deploy a backend written in Go and kept running into problems when trying to read data from the HTTP request body. The client-side app I was developing to communicate with the backend is also written in Go and everything seemed to work fine when running the backend locally (using now dev), but the exact same requests failed when running it in production. The client’s request body was available when in development, but returned empty strings when running in production.

It eventually boiled down to two things:

Now’s dev environment does not seem to care about discrepancies between the Content-Length header and the actual request body. In production, however, if the Content-Length does not exist (or is 0) then the request body will not be read even if it exists (and I admit that this should probably be the expected behaviour and is probably a symptom of the underlying cloud architecture rather than Now itself).
When creating new client requests, Go’s http package automatically sets the Content-Length header based on the length of the request body (and will overwrite/remove the header even if it is set manually), but it will only do this under certain circumstances. In my scenario, a request was being generated with a valid body but without the Content-Length header. To fix the issue, rather than passing in an instance of os.File directly (which I was using to form my request), I needed to read the contents into a buffer before passing it into the request.

Hue: Security Lights

Fri, 18 Aug 2017 17:00:00 +0000

A previous note about Philips Hue bulbs got me thinking that the API exposed by the bridge might be used to warn if the house lights are left on too late at night, or even if they get turned on at unexpected times - potentially for security.

I put together a simple program that periodically checks the status of known Hue bulbs late at night. If any bulbs are discovered to be powered on during such times then an email notification is sent. It runs as a systemd service on a Raspberry Pi.

Currently the project is quite basic, but it could be further extended - perhaps to implement ignore lists or to automatically turn off specific sets of bulbs if they are found to be powered on.

For those interested, the project source and setup info is available on GitHub.

Alexa, ask Sherlock...

Wed, 19 Jul 2017 17:00:00 +0000

I have recently posted about CENode and how it might be used in IoT systems.

Since CENode is partially designed to communicate directly with humans (particularly those out and about or “in the field”) it makes sense for inputs and queries to be provided via voice in addition to or instead of a text interface. Whilst this has been explored in the browser (including in the previous Philips Hue control demo), it made sense to also try to leverage the Alexa voice service to interact with a CENode instance.

The Alexa Voice Service and Alexa Skills Kit are great to work with, and it was relatively straight forward to create a skill to communicate with CENode’s RESTful API.

This short video demonstrates this through using an Amazon Echo to interact with a standard, non-modified CENode instance running on CENode Explorer that is partly pre-loaded with the “space” scenario used in our main CENode demo. The rest of the post discusses the implementation and challenges.

Typical Alexa skills are split into “intents”, which describe the individual ways people might interact with the service. For example, the questions “what is the weather like today?” and “is it going to rain today?” may be two intents of a single weather skill.

The skill logic is handled by AWS Lambda, which is used to associate each intent with an action. When someone gives a voice command, the Alexa Voice Service (AVS) determines which intent is being called for which service, and then passes the control over to the appropriate segment in the Lambda function. The function returns a response to the AVS, which is read back out to the user.

The strength of Alexa’s ability to recognise speech is largely dependent on the information given to build each intent. For example, the intent “what is the weather like in {cityName}?”, where cityName is a variable with several different possibilities generated during the build, will accurately recognise speech initiating this intent because the sentence structure is so well defined. A single intent may have several ways of calling it - “what’s the weather like in…”, “tell me what the weather is in…”, “what’s the weather forecast for…”, etc. - which can be bundled into the model to further improve the accuracy even in noisy environments or when spoken by people with strong accents.

Since CENode is designed to work with an entire input string, however, the voice-to-text accuracy is much lower, and thus determining the intent and its arguments is harder. Since we need CENode to handle the entire input, our demo only has a single intent with two methods of invocation (slots):

ask Sherlock {sentence}
tell Sherlock {sentence}

Since ‘Sherlock’ is also provided as the invocation word for the service, both slots implicitly indicate both the service and the single intent to work with. I used ‘Sherlock’ as the name for the skill as it’s a name we’ve used before for CENode-related apps and it is an easy word for Alexa to understand!

sentence is the complete body to be processed by CENode - e.g. “Jupiter is a planet” or “what is Jupiter?” - giving a typical full Echo invocation: “Alexa, tell Sherlock Jupiter is a planet”. The Alexa segment tells the Echo to begin listening, the tell Sherlock component determines the skill and intent to use, and the remainder of the sentence is the body provided to CENode.

Since we only have a single intent, using either ‘ask’ or ’tell’ in the invocation is irrelevant since it is CENode that will try and work out what is meant from the sentence body - whether a question or an input of information. The two slots are only used for the benefit of the human user and so invocations such as “tell Sherlock what is Jupiter?” still work.

At this stage, the AWS Lambda function handling the intent makes a standard HTTP POST request to a CENode instance, and the response is directly passed back to the Alexa service for reading-out to the user. As such, CENode itself provides all of the error-handling and misunderstood inputs, making the Alexa service itself combined with the Lambda function, in this scenario, very ’thin’.

The skill has not yet been published to the Alexa skills store for general use, but the code for this project, including the Alexa Skills Kit configuration and the AWS Lambda code (written using their Node environment) is available on GitHub.

CENode in IoT

Mon, 26 Jun 2017 17:00:00 +0000

In a previous note I discussed CENode and briefly mentioned its potential for use in interacting with the Internet of Things. I thought I’d add a practical example of how it might be used for this and for ’tasking’ other systems.

I have a few Philips Hue bulbs at home, and the Hue Bridge that enables interaction with the bulbs exposes a nice RESTful API. My aim was to get CENode to use this API to control my lights.

A working example of the concepts in this note is available on GitHub (as a small webapp) and here’s a short demo video (which includes a speech-recognition component):

The first step was to generate a username for the Bridge, which CENode can use to authenticate requests through the API.

I use CE cards to supply instructions to a CENode agent, since this is the generally recognised method for interaction between CE-capable devices. When instantiating a node, any number of CE ‘models’ may be passed in order to form a base knowledge set to work from. Here is such a model for giving CENode a view of the Hue ‘world’:

const lightModel = [
  'conceptualise a ~ hue bridge ~ h that has the value V as ~ address ~ and has the value W as ~ token ~',
  'conceptualise a ~ hue bulb ~ h that has the value C as ~ code ~ and has the value V as ~ strength ~',
  'conceptualise an ~ iot card ~ I that is a card and ~ targets ~ the hue bulb D and has the value P as ~ power ~ and has the value B as ~ brightness ~ and has the value S as ~ saturation ~ and has the value H as ~ hue ~ and has the value C as ~ colour ~',
  'there is a hue bridge named bridge1 that has \'192.168.1.2\' as address and has \'abc123\' as token',
];

The model tells the node about Hue Bridges, bulbs, and a new type of card called an iot card, which supports properties for controlling bulbs. Finally, we instantiate a single bridge with an appropriate IP address and the username/token generated earlier.

Next the CENode instance needs to be created and its agent prepared:

const node = new CENode(CEModels.core, lightModel);
const hueBridge = node.concepts.hue_bridge.instances[0];
updateBulbs();
node.attachAgent();
node.agent.setName('House');

The updateBulbs() function (see it here) makes a request to the Bridge to download data about known Hue bulbs, which are added to the node’s knowledge base. For example;

there is a hue bulb named 'Lounge' that has '7' as code

The code property is the unique identifier the bridge uses to determine the bulb on the network.

Finally, all that was needed was to include a handler function for iot cards and to add this to the CENode agent:

node.agent.cardHandler.handlers['iot card'] = (card) => {
  if (card.targets){
    const data = {};
    if (card.power) data.on = card.power === 'on';
    if (card.brightness) data.bri = parseInt(card.brightness)
    if (card.saturation) data.sat = parseInt(card.saturation)
    if (card.hue) data.hue = parseInt(card.hue)
    request('PUT', hueBridge, '/lights/' + card.targets.code + '/state', data);
  }
};

The function makes an appropriate request to the Hue Bridge based on the properties of the iot card. Now, we can submit sentences like this in order to interact with the system (e.g. to turn the ‘Lounge’ bulb on):

there is an iot card named card1 that is to the agent House and has 'instruction' as content and targets the hue bulb 'Lounge' and has 'on' as power

And that’s it, really. This note contains only the more interesting components of the experiment, but hopefully provides an indication of how the library may be used for simple inter-device communication. The full demo includes extra code to handle the UI for a webapp and extra utility functions.

CENode

Thu, 22 Jun 2017 17:00:00 +0000

Whilst working on the ITA Project - a collaborative research programme between the UK MoD and the US Army Research Laboratory - over the last few years, one of my primary areas has been to research around controlled natural languages, and working with Cardiff University and IBM UK’s Emerging Technology team to develop CENode.

As part of the project - before I joined - researchers at IBM developed the CEStore, which aims to provide tools for working with ITA Controlled English. Controlled English (CE) is a subset of the English language which is structured in a way that attempts to remove ambiguity from statements, enabling machines to understand ‘English’ inputs.

Such a language was developed partly to support multi-agent systems consisting of a mixture of humans and machines, and to allow each agent to be able to communicate with one another using the same protocol in coalition scenarios. In these systems, there may be agents on the ground who submit information to the CEStore in CE, which is able to parse and understand the inputs. The CEStore may then pass the information on to other interested parties or may give an agent (such as a drone, camera, sensor, or other equipment) a task (follow, intersect, watch, etc.) based on the complement of the existing knowledge and the new input.

An old example we use combines the CEStore with a system capable of assigning missions to sensors or equipment (see this paper). This example focuses on ‘John Smith’, who is known to the CE system as a HVT (high-value target) owning a black car with licence plate ‘ABC 123’. A human agent on the ground may later observe a speeding car and issue information into the system through an interface on their mobile device or via a microphone;

there is a car named car1 which has black as colour and has 'ABC 123' as licence plate and is travelling north on North Road

The system receiving the message can put together that this speeding car most likely contains John Smith (since it’s known that he owns a car with this licence plate), and so can task a nearby drone to follow it based on the coordinates of the road and the direction of travel.

A human agent being able to type or speak this precise type of English is unlikely, particularly in emergency or rapid-response scnearios, and so the CEStore has a level of understanding of ’natural’ language, and is able to translate many sentences from natural language English into CE - enabling agents to, largely, speak in a more native fashion.

The usefulness of the CEStore project led us to consider possibilities of a (lighter) version of a CEStore that could run on mobile devices in a decentralised network of CE-capable devices without relying on a centralised node responsible for parsing and translating all CE inputs. Such a system would also have the benefit of supporting a network of distributed ’nodes’, each with the ability to maintain their own distinct knowledge bases and to understand and ‘speak’ CE - and thus the concept for CENode was produced.

A key motivation for this was to support those agents who may not have a consistent network connection to a central server, but who still need knowledge support and the ability to report information - thus building the local knowledge base and improving inferences. Then, once the agent can re-establish a connection to other nodes, new information can propagate through the network.

The CENode project (with source hosted on GitHub) began with a focus on supporting our SHERLOCK experiments, which had traditionally been powered using the CEStore. Using CENode, users of SHERLOCK experienced benefits such as auto-correct and typing suggestions, the ability to continue working offline (with information syncing when a network is re-established), and the display of a personalised ‘dashboard’ indicating the local agent’s view of the world represented by the game.

The SHERLOCK experiment was even covered by the BBC.

Since then, the CENode project has grown, and many of the features enjoyed by the CEStore (which is written in Java and deployed using Apache Tomcat) have been re-implemented for CENode. The library supports rules that fire given specific inputs, simple natural language understanding and parsing, querying through CE inputs, the CE cards blackboard architecture, and policies - enabling CENode instances to communicate with each other in different topologies.

CENode is written in JavaScript, since this allows it to be downloaded to and cached on any JavaScript-supporting browser (for example, on a mobile phone or tablet), and to run as a Node app.

In addition to using the CE-based (‘cards’) interfaces, CENode can be interacted-with using the JavaScript bindings and can expose RESTful APIs when run as a Node app, enabling several types of CENode deployments to work together as part of a single system.

Check out a demo of the library here, which wraps a simple user interface around the library’s JavaScript bindings. In the demo, the local CENode agent is preloaded with some knowledge about planets and stars. Try asking it questions or teaching it something new. Additionally, we have deployed a service called CENode Explorer which can launch cloud-based CENode instances and allows you to browse the knowledge base.

We hope to continue to maintain CENode as part of the project, and to discover more interesting use-cases. There are already clear pathways for its use in voice assistants, bots, and as a protocol for communication in IoT devices (some work for which is already underway). Those interested in developing with the library can get started using the CENode Wiki.

Two Year Update

Thu, 16 Mar 2017 17:00:00 +0000

I haven’t written a post since summer 2015. It’s now March 2017 and I thought I’d write an update very briefly covering the last couple of years.

I finished researching and lecturing full-time in the summer of 2015. It felt like the end of an era; I’d spent around a third of my life at the School of Computer Science and Informatics at Cardiff University, and had experienced time there as an undergraduate through to postgrad and on to full-time staff. However, I felt it was time to move on and to try something new, although I was really pleased to be able to continue working with them on a more casual part-time basis - something that continues to today.

In that summer after leaving full-time work at Cardiff I went interailing around Europe with my friend, Dan. It was an amazing experience through which I had a taste of many new European cities where we met lots of interesting people. We started by flying out to Berlin, and from there our route took us through Prague, Krakow, Budapest, Bratislava, Vienna, Munich, Koblenz, Luxembourg City, Brussels, Antwerp, and then finished in Amsterdam (which I’d been to before, but always love visiting).

Some photos from the Interrail trip.

After returning, I moved to London to start a new full-time job with Chaser Having met the founders David and Mark at a previous Silicon Milkroundabout, Chaser was so great to get involved with - I was part of a fab team creating fin-tech software with a goal to help boost the cashflows in small-medium sized businesses. Working right in the City was fun and totally different to what seemed like a much quieter life in Cardiff. Whilst there, I learned loads more about web-based programming and was able to put some of the data-analysis skills from my PhD to use.

At the end of 2015 I was to move back to South Wales to begin a new job at Simply Do Ideas as a senior engineer. Again, this was a totally different experience involving a shift from fin-tech to ed-tech and a move from the relentless busy-ness of London to the quieter (but no less fun) life of Caerphilly - where our offices were based. Since I was to head the technical side of the business, I was able to put my own stamp on the company and the product, and was able to help decide its future and direction.

Myself and Josh representing Simply Do Ideas at Bett 2017 in London.

In February 2016 I was honoured to be promoted to the Simply Do Ideas board and to have been made the company’s Chief Technology Officer. Over the last year myself and the rest of the team have been proud to be part of a company growing very highly respected in a really interesting and exciting domain, and we’re all very excited about what’s to come in the near (and far) future!

I still continue to work with Cardiff University on some research projects and to help out with some of the final-year students there, I hope to write a little more about this work soon.

I feel so lucky to have been able to experience so much in such a short time frame - from academic research and teaching, being a key part of two growth startups, heading a tech company’s technology arm, being a member of a board along with very highly-respected and successful entrepreneurs and business owners, and getting to meet such a wide range of great people. I feel like I’ve grown and learned so much - both professionally and personally - from all of my experiences and from everyone I’ve met along the way.

Android: Consuming Nintendo Hotspot Data

Wed, 27 May 2015 17:00:00 +0000

I recently blogged about Nintendo Hotspot data and mentioned it could be more usefully consumable in a native mobile app.

As such, I wrote a small Android app for retrieving this data and displaying it on a Google Map. The app shows nearby hotspots, allows users to also search for other non-local places, and shows information on the venue hosting the zone.

The app is available on the Play Store and its source is published on GitHub.

Nintendo's Hotspot 'API'

Tue, 12 May 2015 17:00:00 +0000

Since getting a DS, StreetPass has become quite addictive. It’s actually pretty fun checking the device after walking through town or using public transport to see a list of Miis representing the people you’ve been near recently, and the minigames (such as StreetPass Quest) that require you to ‘meet’ people in order to advance also make it more involved. Essentially the more you’re out and about, the further you can progress - this is further accentuated through Play Coins, which can be used to help ‘buy’ your way forward and are earned for every 100 steps taken whilst holding the device.

The DS systems can also use relay points in Nintendo Zone hotspots to collect StreetPass hits. These zones are special WiFi access points hosted in certain commercial venues (e.g. in McDonalds and Subway restaurants), and allow you to ‘meet’ people around the world who also happen to be in another Nintendo Zone at the same time. As such, users can get a lot of hits very quickly (up to a maximum of 10 at a time). There are various ways people have found to set up a ‘home’ zone, but Nintendo have also published a map to display official nearby zones.

However, their map seems a little clunky to use while out and about, so I wanted to see if there could be an easier way to get this information more quickly. When using the map, the network logs revealed GET requests being made to:

https://microsite.nintendo-europe.com/hotspots/api/hotspots/get

The location for which to retrieve data is specified through the zoom and bbox parameters, which seem to map directly to the zoom level and the bounds reported by the underlying Google Maps API being used. For some reason, the parameter ummary_mode=true also needs to be set. As such, a (unencoded) request for central Cardiff may look like this:

/hotspots/api/hotspots/get?summary_mode=true&zoom=18&bbox=51.480043,-3.180592,51.483073,-3.173028

Where the coordinates (51.480043,-3.180592 and (51.483073,-3.173028<) respectively represent the lower-left and upper-right corners of the bounding box. The response is in JSON, and contains a lat/lng for each zone, a name, and an ID that can be used to retrieve more information about the host’s zone using this URL format:

https://microsite.nintendo-europe.com/hotspots/#hotspot/<ID>

When the map is zoomed-out (to prevent map-cluttering) a zone ‘group’ might be returned instead of an individual zone, for each of which the size is indicated. Zooming back in to a group then reveals the individual zones existing in that area.

It seems that this server endpoint does not support cross-origin resource-sharing (CORS), which means that the data is not retrievable for a third-party web-app (at least, without some degree of proxying) due to browser restrictions. However, and especially since the endpoint currently requires no session implementation or other kind of authentication, the data seems very easily retrievable and manageable for non-browser applications and other types of systems.

Using Weka in Go

Fri, 01 May 2015 17:00:00 +0000

A couple of years ago I wrote a blog post about wrapping some of Weka’s classification functionality to allow it to be used programmatically in Python programs. A small project I’m currently working on at home is around taking some of the later research from my PhD work to see if it can be expressed and used as a simple web-app.

I began development in Go as I hadn’t yet spent much time working with the language. The research work involves using a Bayesian network classifier to help infer a tweet’s interestingness, and while Go machine-learning toolkits do exist, I wanted to use my existing models that were serialized in Java by Weka.

I started working on WekaGo, which is able to programmatically support simple classification tasks within a Go program. It essentially just manages the model, abstracts the generation of ARFF files, and executes the necessary Java to make it quick and easy to train and classify data:

model := wekago.NewModel("bayes.BayesNet")
...
model.AddTrainingInstance(train_instance1)
...
model.Train()
model.AddTestingInstance(train_instance1)
...
model.Test()

Results from the classification can then be examined, as described.

Media and volume keys in i3

Tue, 28 Apr 2015 17:00:00 +0000

As is the case with many people, all music I listen to on my PC these days plays from the web through a browser. I’m a heavy user of Google Play Music and SoundCloud, and using Chrome to handle everything means playlists and libraries (and the way I use them through extensions) sync up properly everywhere I need them.

On OS X I use BearededSpice to map the keyboard media controls to browser-based music-players, and the volume keys adjusted the system as they should. Using i3 (and other lightweight window managers) can make you realise what you take for granted when using more fully-fledged arrangements, but it doesn’t take long to achieve the same functionality on such systems.

A quick search revealed keysocket - a Chrome extension that listens out for the hardware media keys and is able to interact with a large list of supported music websites. In order to get the volume controls working, I needed to map i3 through to alsa, and this turned out to be pretty straight-forward too. It only required the addition of three lines to my i3 config to handle the volume-up, volume-down, and mute keys:

bindsym XF86AudioRaiseVolume exec amixer -q set Master 4%+ unmute
bindsym XF86AudioLowerVolume exec amixer -q set Master 4%- unmute
bindsym XF86AudioMute exec amixer -q set Master toggle

And for fun added the block below to ~/.i3status.conf to get the volume displayed on the status bar:

volume master {
    format = "♪ %volume "
    device = "default"
    mixer = "Master"
    mixer_idx = 0
}

Web and Social Computing

Wed, 18 Feb 2015 17:00:00 +0000

his week I begin lecturing a module for Cardiff School of Computer Science and Informatics’ postgraduate MSc course in Advanced Computer Science.

The module is called Web and Social Computing, with the main aim being to introduce students to the concepts of social computing and web-based systems. The course will include both theory and practical sessions in order to allow them to enhance their knowledge derived from literature with the practice of key concepts. We’ll also have lots of guest lectures from experts in specific areas to help reinforce the importance of this domain.

As part of the module, I will encourage students to try and increase their web-presence and to interact with a wider community on the Internet. They’ll do this by engaging more with social media and by maintaining a blog on things they’ve learned and researched.

Each week, the students will give a 5-minute Ignite-format talk on the research they’ve carried out. The quick presentation style will allow everyone in the group to convey what they feel are the most important and relevant parts in current research across many of the topics covered in the module.

We’ll cover quite a diverse range of topics, starting from an introduction to networks and a coverage of mathematical graph theory. This will lead on to social networks, including using APIs to harvest data in useful ways. Over the last few weeks, we’ll delve into subjects around socially-driven business models and peer-to-peer finance systems, such as BitCoin.

During the course, I hope that students will gain practical experience with various technologies, such as NetworkX for modelling and visualising graphs in Python, Weka for some machine learning and classification, and good practices for building and using web APIs.

Developing Useful APIs for the Web

Thu, 05 Feb 2015 17:00:00 +0000

Yesterday, I gave a talk about my experiences with developing and using RESTful APIs, with the goal of providing tips for structuring such interfaces so that they work in a useful and sensible way.

I went back to first principles, with overviews of basic HTTP messages as part of the request-response cycle and using sensible status codes in HTTP responses. I discussed the benefits of ‘collection-oriented’ endpoint URLs to identify resources that can be accessed and modified and the use of HTTP methods to describe what to do with these resources.

NHS Hack Day

Tue, 27 Jan 2015 17:00:00 +0000

This weekend I took part in the NHS Hack Day. The idea of the event is to bring healthcare professionals together with technology enthusiasts in order to build stuff that is useful for those within the NHS and for those that use it. It was organised by AnneMarie Cunningham, who did a great job in making the whole thing run smoothly!

This was our team! The image is released under a Creative Commons BY-NC2.0 license by Paul Clarke.

I was asked to go along and give a hand by Martin, who also had four of his MSc students with him. Matt, previously from Cardiff CS&I, also came to provide his data-handling expertise.

We built a webapp, called Health Explorer Wales, that attempts to visualise various data for health boards and communities in Wales. One of the main goals of the app was to make it maintainable, so that users in future could easily add their own geographic or numeric data to visualise. For this, it was important to decide on an extensible data schema for describing data, and suitable data formats.

Once the schema was finalised, we were able to go ahead and build the front-end, which used D3.js to handle the visualisations. This was the only third-party library we used in the end. The rest of the interface included controls, such as a dataset-selector and controls for sliding back through time (for timeseries data). The app is purely front-end, which means it can essentially be shipped as a single HTML file (with linked scripts and styles).

We also included an ‘add dataset’ feature, which allows users to add a dataset to be visualised, as long as the schema is observed. In true hackathon style, any exceptions thrown will currently cause the process to fail silently ;) The GitHub repository for the app contains a wiki with some guidance on data-formatting. Since the app is front-end only, any data added is persisted using HTML5 local storage and is therefore user-specific.

Generally, I am pleased with the result. The proof-of-concept is (mostly) mobile-friendly, and allows for easily showing off data in a more comprehensible way than through just using spreadsheets. Although we focussed on visualising only two datatypes initially (we all <3 #maps), we hope to extend this by dropping in modules for supporting new formats in the future.

There were many successful projects completed as part of the event, including a new ’eye-test’ concept involving a zombie game using an Oculus Rift and an app for organising group coastal walks around Wales. A full list of projects is available on the event’s website. I really enjoyed the weekend and hope to make the next one in London in May!

End of an Era

Tue, 20 Jan 2015 17:00:00 +0000

I recently received confirmation of my completed PhD! I submitted my thesis in May 2014, passed my viva in September and returned my final corrections in December.

I was examined internally by Dr Pete Burnap and also by Dr Jeremy Pitt of Imperial College London.

The whole PhD was an amazing experience, even during the more stressful moments. I learnt a huge amount across many domains and I cannot thank my supervisors, Dr Stuart Allen and Prof Roger Whitker, enough for their fantastic support and guidance throughout.

Talk on Open-Source Contribution

Wed, 26 Mar 2014 17:00:00 +0000

Today I gave an internal talk at the School of Computer Science & Informatics about open-source contribution.

The talk described some of the disadvantages of the ways in which hobbyists and the non-professional sector publicly publish their code. A lot of the time these projects do not receive much visibility or use from others.

Public contribution is important to the open-source community, which is driven largely by volunteers and enthusiasts, so the point of the talk was to try and encourage people to share expert knowledge through contributing documentation (wikis, forums, articles, etc.), maintaining and adopting packages, and getting more widely involved.

Node.js Contribution to Heroku's Dev Center

Mon, 17 Mar 2014 17:00:00 +0000

I recently wrote a new article for Heroku’s Dev Center on carrying out asynchronous direct-to-S3 uploads using Node.js.

he article is based heavily on the previous Python version, where the only major change is the method for signing the AWS request. This method was outlined in an earlier blog post.

The article is available here and there is also a companion code repository for the example it describes.

Seminar at King's College London

Tue, 28 Jan 2014 17:00:00 +0000

Last week, I was invited to give a seminar to the Agents and Intelligent Systems group in the Department of Informatics at King’s College London.

I gave an overview of my PhD research conducted over the past two or three years, from my initial research into retweet behaviours and propagation characteristics through to studies on the properties exhibited by Twitter’s social graph and the effects that the interconnection of users have on message dissemination.

I finished by outlining our methods for identifying interesting content on Twitter and by demonstrating its relative strengths and weaknesses as were made clear by crowd-sourced validations carried out on the methodology results.

There was some very interesting and useful questions from the audience, some of which is now being taken into consideration in my thesis. It was also good to visit another computer science department and to hear about the work done independently and collaboratively by its different research groups.

The slides from the seminar are available here and there is a blog post about it on the Department of Informatics’ website.

Direct-to-S3 Uploads in Node.js

Fri, 17 Jan 2014 17:00:00 +0000

A while ago I wrote an article for Heroku’s Dev Center on carrying out direct uploads to S3 using a Python app for signing the PUT request. Specifically, the article focussed on Flask but the concept is also applicable to most other Python web frameworks.

I’ve recently had to implement something similar, but this time as part of an Node.js application. Since the only difference between the two approaches is literally just the endpoint used to return a signed request URL, I thought I’d post an update on how the endpoint could be constructed in Node.

The front-end code in the companion repository demonstrates an example of how the endpoint can be queried to retrieve the signed URL, and is available here. Take a look at that repository’s README for information on the front-end dependencies.

The full example referenced by the Python article is in a repository hosted by GitHub and may be useful in providing more context.

Workshop Presentation in Germany

Sat, 05 Oct 2013 17:00:00 +0000

Last week I visited Karlsruhe, in Germany, to give a presentation accompanying a recently-accepted paper. The paper, “Inferring the Interesting Tweets in Your Network”, was in the proceedings of the Workshop on Analyzing Social Media for the Benefit of Society (Society 2.0), which was part of the Third International Conference on Social Computing and its Applications (SCA).

Although I only attended the first workshop day, there was a variety of interesting talks on social media and crowdsourcing. My own talk went well and there was some useful feedback from the attendees.

I presented my recent work on the use of machine learning techniques to help in identifying interesting information in Twitter. I rounded up some of the results from the Twinterest experiment we ran a few months ago and discussed how this helped address the notion of information relevance as an extension to global interestingness.

I hadn’t been to Germany before this, so it was also a culturally-interesting visit. I was only there for two nights but I tried to make the most of seeing some of Karlsruhe and enjoying the traditional food and local beers!

CasaStream

Sat, 14 Sep 2013 17:00:00 +0000

In my last post I discussed methods for streaming music to different zones in the house. More specifically I wanted to be able to play music from one location and then listen to it in other rooms at the same time and in sync.

After researching various methods, I decided to go with using a compressed MP3 stream over RTP. Other techniques introduced too much latency, did not provide the flexibility I required, or simply did not fulfill the requirements (e.g. not multiroom, only working with certain applications and non-simultaneous playback).

To streamline the procedure of compressing the stream, broadcasting the stream, and receiving and playing the stream, I have started a project to create an easily-deployable wrapper around PulseAudio and VLC. The system, somewhat cheesily named CasaStream and currently written primarily in Python, relies on a network containing one machine running a CasaStream Master server and any number of machines running a CasaStream Slave server.

The Master server is responsible for compressing and broadcasting the stream, and the Slaves receive and play the stream back through connected speakers. Although the compression is relatively resource-intensive (at least, for the moment), the Slave server is lightweight enough to be run on low-powered devices, such as the Raspberry Pi. Any machine that is powerful enough to run the Master could also simultaneously run a Slave, so a dedicated machine to serve the music alone is not required.

The Master server also runs a web interface, allowing enabling of the system and to disable and enable Slaves. Slave servers are automatically discovered by the Master, though it is possible to alter the scan range from the web interface also. In addition, the selection of audio sources to stream (and their output volumes) and the renaming of Slaves are available as options. Sound sources are usually automatically detected by PulseAudio (if it is running), so there is generally no manual intervention required to ‘force’ the detection of sources.

My current setup consists of a Master server running on a desktop machine in the kitchen, and Slave servers running on various other machines throughout the house (including the same kitchen desktop connected to some orbital speakers and a Raspberry Pi connected to the surround sound in the living room). When all running, there is no notable delay between the audio output in the different rooms.

There are a few easily-installable dependencies required to run both servers. Both require Python (works on V2.*, but I haven’t tested on V3), and both require the Flask microframework and VLC. For a full list, please see the README at the project’s home, which also provides more information on the installation and use.

Unfortunately, there are a couple of caveats: firstly, the system is not reliable over WLAN (the sound gets pretty choppy), so a wired connection is recommended. Secondly, if using ethernet-over-power to mitigate the first caveat, then you may experience sound dropouts every 4-5 minutes. To help with this problem, the Slave servers are set to restart the stream every four minutes (by default).

This is quite an annoying issue, however, since having short sound interruptions every few minutes is very noticeable. Some of my next steps with this project, therefore, are based around trying to find a better fix for this. In addition, I’d like to reduce the dependency footprint (the Slave servers really don’t need to use a fully-fledged web server), reduce the power requirements at both ends, and to further automate the installation process.

Zoned Network Sound-Streaming: The Problem

Mon, 02 Sep 2013 17:00:00 +0000

For a while, now, I have been looking for a reliable way to manage zoned music-playing around the house. The general idea is that I’d like to be able to play music from a central point and have it streamed over the network to a selection of receivers, which could be remotely turned on and off when required, but still allow for multiple receivers to play simulataneously.

Apple’s AirPlay has supported this for a while now, but requires the purchasing of AirPlay compatible hardware, which is expensive. It’s also very iTunes-based - which is something that I do not use.

Various open-source tools also allow network streaming. Icecast (through the use of Darkice) allows clients to stream from a multimedia server, but this causes pretty severe latency in playback between clients (ranging up to around 20 seconds, I’ve found) - not a good solution in a house!

PulseAudio is partly designed around being able to work over the network, and supports the discovery of other PulseAudio sinks on the LAN and the selection a sound card to transmit to through TCP. This doesn’t seem to support multiple sound card sinks very well, however.

PulseAudio’s other network feature is its RTP broadcasting, and this seemed the most promising avenue for progression in solving this problem. RTP utilises UDP, and PulseAudio effecively uses this to broadcast its sound to any devices on the network that might be listening on the broadcast address. This means that one server could be run and sink devices could be set up simply to receive the RTP stream on demand - perfect!

However, in practice, this turned out not to work very well. With RTP enabled, PulseAudio would entirely flood the network with sound packets. Although this isn’t a problem for devices with a wired connection, any devices connected wirelessly to the network would be immediately disassociated from the access point due to the complete saturation of PulseAudio’s packets being sent over the airwaves.

This couldn’t be an option in a house where smartphones, games consoles, laptops, and so on require the WLAN. After researching this problem a fair bit (and finding many others experiencing the same issues), I found this page, which describes various methods for using RTP streaming from PulseAudio and includes (at the bottom) the key that could fix my problems - the notion of compressing the audio into MP3 format (or similar) before broadcasting it.

Trying this technique worked perfectly, and did not cause network floods anywhere nearly as severely as the uncompressed sound stream; wireless clients no longer lost access to the network once the stream was started and didn’t seem to lose any noticeable QoS at all. In addition, when multiple clients connected, the sound output would be nearly entirely simultaneous (at least after a few seconds to warm up).

Unfortunately, broadcasting still didn’t work well over WLAN (sound splutters and periodic drop-outs), so the master server and any sound sinks would need to be on a wired network. This is a small price to pay, however, and I am happy to live with a few Ethernet-over-power devices around the house. The next stage is to think about what to use as sinks. Raspberry Pis should be powerful enough and are significantly cheaper than Apple’s equivalent. They would also allow me to use existing sound systems in some rooms (e.g. the surround-sound in the living room), and other simple speaker setups in others. I also intend to write a program around PulseAudio to streamline the streaming process and a server for discovering networked sinks.

I will write an update when I have made any more progress on this!

A rather French week

Sat, 31 Aug 2013 17:00:00 +0000

I recently spent a week in France as part of a holiday with some of my family. Renting houses for a couple of weeks in France or Italy each summer has almost become a bit of a tradition, and it’s good to have a relax and a catch-up for a few days. They have been the first proper few days (other than the decking-building adventure back in March) I have had away from University in 2013, so I felt it was well-deserved!

This year we stayed in the Basque Country of southern France, relatively near Biarritz, in a country farmhouse. Although we weren’t really within walking distance to anywhere, the house did come with a pool in the garden, with a swimmable river just beyond, and an amazing, peaceful setting.

Strangely enough, there was no Internet installation at the house, and no cellular reception anywhere nearby. This took a bit of getting-used to, but after a while it became quite relaxing not having to worry about checking emails, texts, and Twitter. The only thing to cause any stress was a crazed donkey, living in the field next door, who would start braying loudly at random intervals through the nights, waking everyone up.

As might be expected, the food and drink was exceptional. Although we did end up eating in the house each evening (to save having someone sacrifice themselves to be the designated driver), the foods we bought from the markets were very good, and the fact that wine cost €1.50 per bottle from the local Intermarché gave very little to complain about.

The majority of most days was spent away from the house, visiting local towns, the beaches and the Pyrenees. We spent a few afternoons walking in the mountains, with some spectacular scenery.

Gower Tides v1.4

Wed, 31 Jul 2013 17:00:00 +0000

Last week I released a new version of the tides Android app I’m currently developing.

The idea of the application was initially to simply display the tidal times and patterns for the Gower Peninsula, and that this should be possible without a data connection. Though, as the time has gone by, I keep finding more and more things that can be added!

The latest update saw the introduction of 5-day surf forecasts for four Gower locations - Llangennith, Langland, Caswell Bay, and Hunts Bay. All the surf data comes from Magic Seaweed’s API (which I talked about last time).

he surf forecasts are shown, for each day they are available, as a horizontal scroll-view, allowing users to scroll left and right within that day to view the forecast at different times of the day (in 3-hourly intervals).

Location selection is handled by a dialog popup, which shows a labelled map and a list of the four available locations in a list view.

The backend support for the application was modified to now also support 30-minute caching of surf data on a per-location basis (i.e. new calls to Magic Seaweed would not be made if the requested location had been previously pulled in the last 30 minutes). The complete surf and weather data is then shipped back to the phone as one JSON structure.

Other updates were smaller but included an overhaul of the UI (the tide table now looks a bit nicer), additional licensing information, more speedy database interaction, and so on.

If you are interested in the source, then that is available here, and the app itself is on Google Play. If you have any ideas, feedback or general comments, then please let me know!

Magic Seaweed's Awesome New API

Wed, 03 Jul 2013 17:00:00 +0000

Back in March, I emailed Magic Seaweed to ask them if they had a public API for their surf forecast data. They responded that they didn’t at the time, but that it was certainly on their to-do list. I am interested in the marine data for my Gower Tides application.

Yesterday, I visited their website to have a look at the surf reports and some photos, when I noticed the presence of a Developer link in the footer of the site. It linked to pages about their new API, with an overview describing exactly what I wanted.

Since the API is currently in beta, I emailed them requesting a key, which they were quick to respond with and helpfully included some further example request usages. They currently do not have any strict rate limits in place, but instead have a few fair practice terms to discourage developers from going a bit trigger happy on API requests. They also request that you use a hyperlinked logo to accredit the data back to them. Due to caching, I will not have to make too many requests (since the application will preserve ‘stale’ data for 30 minutes before refreshing from Magic Seaweed, when requested), so hopefully that will keep the app’s footprint down.

I have written the app’s new backend support for handling and caching the surf data ready for incorporating into the Android app soon. So far, the experience has been really good, with the API responding with lots of detailed information - almost matching the data behind their own surf forecasts. Hopefully they won’t remove any of the features when they properly release it!

Accidental Kernel Upgrades on Digital Ocean

Thu, 20 Jun 2013 17:00:00 +0000

I today issued a full upgrade of the server at flyingsparx.net, which is hosted by Digital Ocean. By default, on Arch, this will upgrade every currently-installed package (where there is a counterpart in the official repositories), including the Linux kernel and the kernel headers.

Digital Ocean maintain their own kernel versions and do not currently allow kernel switching, which is something I completely forgot. I rebooted the machine and tried re-connecting, but SSH couldn’t find the host. Digital Ocean’s website provides a console for connecting to the instance (or ‘droplet’) through VNC, which I used, through which I discovered that none of the network interfaces (except the loopback) were being brought up. I tried everything I could think of to fix this, but without being able to connect the droplet to the Internet, I was unable to download any other packages.

Eventually, I contacted DO’s support, who were super quick in replying. They pointed out that the upgrade may have also updated the kernel (which, of course, it had), and that therefore the modules for networking weren’t going to load properly. I restored the droplet from one of the automatic backups, swapped the kernel back using DO’s web console, rebooted and things were back to where they should be.

The fact that these things can be instantly fixed from their console and their quick customer support make Digital Ocean awesome! If they weren’t possible then this would have been a massive issue, since the downtime also took out this website and the backend for a couple of mobile apps. If you use an Arch instance, then there is a community article on their website explaining how to make pacman ignore kernel upgrades and to stop this from happening.

WekaPy

Wed, 12 Jun 2013 17:00:00 +0000

Over the last few months, I’ve started to use Weka more and more. Weka is a toolkit, written in Java, that I use to create models with which to make classifications on data sets.

It features a wide variety of different machine learning algorithms (although I’ve used the logistic regressions and Bayesian networks most) which can be trained on data in order to make classifications (or ‘predictions’) for sets of instances.

Weka comes as a GUI application and also as a library of classes for use from the command line or in Java applications. I needed to use it to create some large models and several smaller ones, and using the GUI version makes the process of training the model, testing it with data and parsing the classifications a bit clunky. I needed to automate the process a bit more.

Nearly all of the development work for my PhD has been in Python, and it’d be nice to just plug in some machine learning processes over my existing code. Whilst there are some wrappers for Weka written for Python (this project, PyWeka, etc.), most of them feel unfinished, are under-documented or are essentially just instructions on how to use Jython.

So, I started work on WekaPy, a simple wrapper that allows efficient and Python-friendly integration with Weka. It basically just involves subprocesses to execute Weka from the command line, but also includes several areas of functionality aimed to provide more of a seamless and simple experience to the user.

I haven’t got round to writing proper documentation yet, but most of the current functionality is explained and demo’d through examples here. Below is an example demonstrating its ease of use.

model = Model(classifier_type = "bayes.BayesNet")
model.train(training_file = "train.arff")
model.test(test_file = "test.arff")

All that is needed is to instantiate the model with your desired classifier, train it with some training data and then test it against your test data. The predictions can then be easily extracted from the model as shown in the documentation.

I hope to continue updating the library and improving the documentation when I get a chance! Please let me know if you have any ideas for functionality.

Gower Tides Open-Sourced

Sun, 26 May 2013 17:00:00 +0000

This is just a quick post to mention that I have made the source for the Gower Tides app on Google Play public.

The source repository is available on GitHub. From the repository I have excluded:

Images & icons - It is not my place to distribute graphics not owned or created by me. Authors are credited in the repo’s README and in the application.
External libraries - The app requires a graphing package and a class to help with handling locally-packaged SQLite databases. Links to both are also included in the repo’s README.
Tidal data - The tidal data displayed in the app has also been excluded. However, the format for the data stored by the app should be relatively obvious from its access in the source.

Contribution to Heroku Dev Center

Tue, 07 May 2013 17:00:00 +0000

The Heroku Dev Center is a repository of guides and articles to provide support for those writing applications to be run on the Heroku platform.

I recently contributed an article for carrying out Direct to S3 File Uploads in Python, as I have previously used a very similar approach to interface with Amazon’s Simple Storage Service in one of my apps running on Heroku.

The approach discussed in the article focuses on avoiding as much server-side processing as possible, with the aim of preventing the app’s web dynos from becoming too tied up and unable to respond to further requests. This is done by using client-side JavaScript to asynchronously carry out the upload directly to S3 from the web browser. The only necessary server-side processing involves the generation of a temporarily-signed (using existing AWS credentials) request, which is returned to the browser in order to allow the JavaScript to successfully make the final PUT request.

The guide’s companion git repository hopes to demonstrate a simple use-case for this system. As with all of the Heroku Dev Center articles, if you have any feedback (e.g. what could be improved, what helped you, etc.), then please do provide it!

eartub.es

Thu, 25 Apr 2013 17:00:00 +0000

Last weekend I went to CFHack Open Sauce Hackathon. I worked in a team with Chris, Ross and Matt.

We started work on eartub.es, which is a web application for suggesting movies based on their sound tracks. We had several ideas for requirements we wanted to meet but, due to the nature of hackathons, we didn’t do nearly as much as what we thought we would!

For now, eartubes allows you to search for a movie (from a 2.5 million movie database) and view other movies with similar soundtracks. This is currently based on cross matching the composer between movies, but more in-depth functionality is still in the works. We have nearly completed Last.fm integration, which would allow the app to suggest movies from your favourite and most listened-to music, and are working towards genre-matching and other, more complex, learning techniques. The registration functionality is disabled while we add this extra stuff.

The backend is written in Python and runs as a Flask application. Contrary to my usual preference, I worked on the front end of the application, but also wrote our internal API for Last.fm integration. It was a really fun experience, in which everyone got on with their own individual parts, and it was good to see the project come together at the end of the weekend.

The project’s source is on GitHub.

flyingsparx.net On Digital Ocean

Tue, 23 Apr 2013 17:00:00 +0000

My hosting for my website has nearly expired, so I have been looking for renewal options.

These days I tend to need to use servers for more than simple web-hosting, and most do not provide the flexibility that a VPS would. Having (mostly) full control over a properly-maintained virtual cloud server is so much more convenient, and allows you to do tonnes of stuff beyond simple web hosting.

I have some applications deployed on Heroku, which is definitely useful and easy for this purpose, but I decided to complement this for my needs by buying a ‘droplet’ from Digital Ocean.

Droplets are DO’s term for a server instance, and are super quick to set up (55 seconds from first landing at their site to a booted virtual server, they claim) and very reasonably priced. I started an Arch instance, quickly set up nginx, Python and uwsgi, and started this blog and site as a Python app running on the Flask microframework.

So far, I’ve had no issues, and everything seems to work quickly and smoothly. If all goes to plan, over the next few months I’ll migrate some more stuff over, including the backend for the Gower Tides app.

Trials of Eduroam

Tue, 16 Apr 2013 17:00:00 +0000

I’ve been having trouble connecting to Eduroam, at least reliably and persistently, without heavy desktop environments or complicated network managers. Eduroam is the wireless networking service used by many Universities in Europe, and whilst it would probably work fine using the tools provided by heavier DEs, I wanted something that could just run quickly and independently.

Many approaches require the editing of loads of config files (especially true for netcfg), which would need altering again after things like password changes. The approach I used (for Arch Linux) is actually really simple and involves the use of the user-contributed wicd-eduroam package available in the Arch User Repository.

Obviously, wicd-eduroam is related to, and depends on, wicd, a handy network connection manager, so install that first:

# pacman -S wicd
$ yaourt -S wicd-eduroam

(If you don’t use yaourt download the tarball and build it using the makepkg method.)

wicd can conflict with other network managers, so stop and disable them before starting and enabling wicd. This will allow it to startup at boot time. e.g.:

# systemctl stop NetworkManager
# systemctl disable NetworkManager
# systemctl start wicd
# systemctl enable wicd

Now start wicd-client (or set it to autostart), let it scan for networks, and edit the properties of the network eduroam Set the encryption type as eduroam in the list, enter the username and password, click OK and then allow it to connect.

Cardiff Open Sauce Hackathon

Thu, 11 Apr 2013 17:00:00 +0000

Next week I, along with others in a team, am taking part in Cardiff Open Sauce Hackathon.

If you’re in the area and feel like joining in for the weekend then sign up at the link above.

he hackathon is a two-day event in which teams work to ‘hack together’ smallish projects, which will be open-sourced at the end of the weekend. Whilst we have a few ideas already for potential projects, if anyone has any cool ideas for something relatively quick, but useful, to make, then please let me know!

AJAX + Python + Amazon S3

Fri, 05 Apr 2013 17:00:00 +0000

I wanted a way in which users can seamlessly upload images for use in the Heroku application discussed in previous posts.

Ideally, the image would be uploaded through AJAX as part of a data-entry form, but without having to refresh the page or anything else that would disrupt the user’s experience. As far as I know, barebones JQuery does not support AJAX uploads, but this handy plugin does.

Handling the upload (AJAX)

styled the file input nicely (in a similar way to this guy) and added the JS so that the upload is sent properly (and to the appropriate URL) when a change is detected to the input (i.e. the user does not need to click the ‘upload’ button to start the upload).

Receiving the upload (Python)

he backend, as previously mentioned, is written in Python as part of a Flask app. Since Heroku’s customer webspace is read-only, uploads would have to be stored elsewhere. Boto’s a cool library for interfacing with various AWS products (including S3) and can easily be installed with pip install boto. From this library, we’re going to need the S3Connection and Key classes:

from boto.s3.connection import S3Connection
from boto.s3.key import Key

Now we can easily handle the transfer using the request object exposed to Flask’s routing methods:

file = request.files['file_input_name']
con = S3Connection(<'AWS_KEY'>, <'AWS_SECRET'>)
key = Key(con.get_bucket(<'BUCKET_NAME'>))
key.set_contents_from_file(file)

Go to the next step for the AWS details and the bucket name. Depending on where you chose your AWS location as (e.g. US, Europe, etc.), then your file will be accessible as something like https://s3-eu-west-1.amazonaws.com//. If you want, you can also set, among other things, stuff like the file’s mime type and access type:

key.set_metadata('Content-Type', 'image/png')
key.set_acl('public-read')

Setting up the bucket (Amazon S3)

Finally you’ll need to create the bucket. Create or log into your AWS account, go to the AWS console, choose your region (if you’re in Europe, then the Ireland one is probably the best choice) and enter the S3 section. Here, create a bucket (the name needs to be globally unique). Now, go to your account settings page to find your AWS access key and secret and plug these, along with the bucket name, into the appropriate places in your Python file.

And that’s it. For large files, this may tie up your Heroku dynos a bit while they carry out the upload, so this technique is best for smaller files (especially if you’re only using the one web dyno). My example of a working implementation of this is available in this file.

Decking Building

Sat, 30 Mar 2013 17:00:00 +0000

I managed to turn about two tonnes of material into something vaguely resembling ‘decking’ in my back garden this weekend. It makes the area look much nicer, but whether it actually stays up is a completely different matter.

Gower Tides App Released

Thu, 07 Mar 2013 17:00:00 +0000

A few posts back, I talked about the development of an Android app for tide predictions for South Wales. This app is now on Google Play.

If you live in South Wales and are vaguely interested in tides/weather, then you should probably download it :)

The main advantage is that the app does not need any data connection to display the tidal data, which is useful in areas with low signal. In future, I hope to add further features, such as a more accurate tide graph (using a proper ‘wave’), surf reports, and just general UI updates.

ScriptSlide

Mon, 18 Feb 2013 17:00:00 +0000

I’ve taken to writing most of my recent presentations in plain HTML (rather than using third-party software or services). I used JavaScript to handle the appearance and ordering of slides.

I bundled the JS into a single script, js/scriptslide.js which can be configured using the js/config.js script.

There is a GitHub repo for the code, along with example usage and instructions.

Most configuration can be done by using the js/config.js script, which supports many features including:

Set the slide transition type (appear, fade, slide)
Set the logos, page title, etc.
Configure the colour scheme

Then simply create an HTML document, set some other styles (there is a template in css/styles.css), and put each slide inside

...

tags. The slide menu is then generated autmatically when the page is loaded.

Research Poster Day

Mon, 21 Jan 2013 17:00:00 +0000

Each January the School of Computer Science hosts a poster day in order for the research students to demonstrate their current work to other research students, research staff and undergraduates. The event lets members of the department see what other research is being done outside of their own group and gives researchers an opportunity to defend their research ideas.

This year, I focused on my current research area, which is to do with inferring how interesting a Tweet is based on a comparison between simulated retweet patterns and the propagation behaviour demonstrated by the Tweet in Twitter itself. The poster highlights recent work in the build-up to this, a general overview of how the research works, and finishes with where I want to take this research in the future.

Delving into Android

Tue, 13 Nov 2012 17:00:00 +0000

I’ve always been interested in the development of smartphone apps, but have never really had the opportunity to actually hava a go. Whilst I’m generally OK with development on platforms I feel comfortable with, I’ve always considered there to be no point in developing applications for wider use unless you have a good idea about first thinking about the direction for it to go.

My Dad is a keen surfer and has a watch which tells the tide changes as well as the time. It shows the next event (i.e. low- or high-tide) and the time until that event, but he always complains about how inaccurate it is and how it never correctly predicts the tide schedule for the places he likes to surf.

He uses an Android phone, and so I thought I’d try making an app for him that would be more accurate than his watch, and maybe provide more interesting features. The only tricky criterion, really, was that he needed it to predict the tides offline, since the data reception is very poor in his area.

I got to work on setting up a database of tidal data, based around the location he surfs in, and creating a basic UI in which to display it. When packaging the application with an existing SQLite database, this helper class was particularly useful.

A graphical UI seemed the best approach for displaying the data, so I tried [AndroidPlot]](http://androidplot.com/), a highly-customisable graphing library, to show the tidal patterns day-by-day. This seemed to work OK (though not entirely accurately - tidal patterns form more of a cosine wave rather than the zigzags my graph produced, but the general idea is there), so I added more features, such as a tide table (the more traditional approach) and a sunrise and sunset timer.

I showed him the app at this stage, and he decided it could be improved by adding weather forecasts. Obviously, preidcting the weather cannot be done offline, so having sourced a decent weather API, I added the weather forecast for his area too. Due to the rate-limiting of World Weather Online, a cache is stored in a database on the host for this website, which, when queried by the app, will make the request on the app’s behalf and store the data until it is stale.

I added a preferences activity for some general customisation, and that’s as far as I’ve currently got. In terms of development, I guess it’s been a good introduction to the ideas behind various methodologies and features, such as the manifest file, networking, local storage, preferences, and layout design. I’ll create a Github repository for it when I get round to it.

Seminar: Retweeting

Wed, 10 Oct 2012 17:00:00 +0000

I gave a seminar on my current research phase.

I summarised my work over the past few months; in particular, the work on the network structure of Twitter, the way in which tweets propagate through different network types, and the implications of this. I discussed the importance of precision and recall as metrics for determining a timeline's quality and how this is altered through retweeting in different network types.

I concluded by talking about my next area of research; how I may use the model used for the previous experimentation to determine if a tweet is particularly interesting based on its features. Essentially, this boils down to showing that tweets are siginificantly interesting (or uninteresting) by looking at how they compare to their predicted retweet behaviours as produced by the model.

DigiSocial Hackathon

Thu, 20 Sep 2012 09:15:00 +0000

We recently held our DigiSocial Hackathon. This was a collaboration between the Schools of Computer Science and Social Sciences and was organised by myself and a few others.

The website for the event is hosted here.

The idea of the event was to try and encourage further ties between the different Schools of the University. The University Graduate College (UGC) provide the funding for these events, which must be applied for, in the hope that good projects or results come out of it.

We had relatively good responses from the Schools of Maths, Social Sciences, Medicine, and ourselves, and had a turnout of around 10-15 for the event on the 15th and 16th September. Initially, we started to develop ideas for potential projects. Because of the nature of the event, we wanted to make sure they were as cross-disciplined as possible. A hackday, in itself, is pretty computer science-y so we needed to apply a social or medical spin on our ideas.

Eventually, we settled into two groups: one working on a social-themed project based on crimes in an area (both in terms of distribution and intensity) in relation to the food hygiene levels in nearby establishments; another focusing on hospital wait times and free beds in South Wales. Effectively, then, both projects are visualisations of publicly-available datasets.

I worked on the social project with Matt Williams, Wil Chivers and Martin Chorley, and it is viewable here.

Overall the event was a reasonable success; two projects were completed and we have now made links with the other Schools which will hopefully allow us to do similar events together in the future.

Will Webberley's Blog

Cross-publishing web content to Gemini

CI/CD with Woodpecker and Gitea

Thoughts on book logging

A question about encryption for self-hosting

Option 1: fscrypt PAM decryption

Option 2: Full-disk encryption with LUKS

Option 3: Resistance to power drops

My ask

Window management workflows on macOS

macOS window management

The window-hiding workflow

Addendum: Stage Manager

From Google and Apple Photos to Photoprism

A history of media storage solutions

Apple Photos

Photoprism

Tailscale: multi-service HTTPS on a single machine

HTTPS on Tailscale

Limitations in multi-service contexts

My experiences

Conclusion

Note-taking apps: Bear and Joplin

Some of my gripes with Bear (and CloudKit in general)

Why I prefer Joplin

Re-Building my Website with Hugo

GatsbyJS

Why consider a move?

Switching to Hugo

Things I love ❤️

Making Tax Digital and Plain-Text Accounting

HMRC’s Making Tax Digital

The impact on plain-text accounting

Switching-up my workouts 🏋️‍♀️

Starting to workout

Change of routine

A changed person? 🤔

My setup for coding on iPad

1. Provision a VPS

2. Configure Docker and initial server set-up

3. Configure Tailscale and firewalls

4. Run Code Server

5. Post-setup tasks

Optional next steps

Conclusion

Parcel to the rescue

Could Parcel help?

Alternatives

Simple Ledger visualisations using Python

Step 1: Outputting Ledger data

Step 2: Data processing

Step 3: Producing the diagram

Step 4: Add sub-accounts to the visualisation

Conclusion

On alcohol

Self-hosting apps and services using Traefik reverse proxy

My (until recent) setup

Why I wanted a change

Step 2: Next, write a new Docker Compose file

More complex services

Conclusion

Interacting with a Nextcloud instance deployed with Docker

Invoking occ on a running Nextcloud Docker container

The future of the decentralised web: thoughts on web0 and web3

Decentralisation and web3

How decentralised is web3, really?

Some closing thoughts

Completing the #100DaysToOffload challenge

Interacting with other challengers

Planning “when” to write

Planning “what” to write

Getting over the inertia of writing & some learnings

Takeaways

Browser push notifications in a Flutter web application

Initializing a Flutter app

Setting-up push for web

Caveats

Conclusion

Moving my Matrix identity to Element One

Why I chose to move

Invoking `occ` on a running Nextcloud Docker container