I love working with Claude using git worktrees. They allow me to work on multiple features concurrently on the same code base. In fact, worktrees are the recommended way of working on multiple things in parallel with code. But I quickly run into problems with persisting environment variables and allowing my code to find secrets, both of which I do not commit to git. My default response would have been to copy-and-paste the .env files into the worktree and to hard code a path to some secrets file stored separately (to wit you will start seeing ugly things like Users/binghao/…/secrets.yaml in the codebase. Yuck!). If I were feeling a bit fancier, I’d probably run a secrets vault or use a secrets manager on AWS for my secrets. But that’s just additional infrastructure I would have to run and can be a hassle for small hobby projects.

This is where Claude led me to direnv and sops which help me overcome both problems, respectively. In this post, I will talk about what the above two things are, how I use them and what are some of the gotchas.

Subscribe now

direnv

What it is

direnv is a tool that allows me to automatically load environment variables whenever I enter a directory. It also unloads the environment variables when I leave the directory so that I don’t get stuck with stray environment variables when I go and do other stuff within the same shell.

How I use it

Once direnv is installed, everything starts with a .envrc file in your repository root. In my case, I only have a very simple directive within the .envrc file, which is as follows,

dotenv_if_exists <dir_of_dotenv_file>/.env

This states that whenever I enter the directory, if the dotenv file exists at the stated path, load it.

Then of course, I will have my dotenv files with my environment variables such as database ports, etc. listed in there.

Now each time I enter my git worktree, the environment variables will be automatically loaded.

Gotchas

• When you first enter a newly created worktree, direnv won’t be able to load the environment variables until you type direnv allow to explicitly allow the loading of the .env file.

• In a deployment situation, you would still need to enable your CI/CD framework to find the environment variables and set it on deploy. The direnv documentation gives an example of how it can be done with GitHub Actions.

Other thoughts

• I really like the fact that the environment variables are loaded and unloaded automatically without me having to manually source certain files or add in code to load the .env file. It saves me the time and frustration needed to ensure that the environment variables are properly loaded with each new worktree.

• direnv includes many other directives that can enable much richer behaviour than what I’ve explained here.

sops

What it is

sops (aka Secrets OPerationS) is tool that is used to edit encrypted files (JSON, YAML, ENV, etc.). It supports encryption with “AWS KMS, GCP KMS, Azure Key Vault, HuaweiCloud KMS, age, and PGP”.

How I use it

Basically, sops allows me to store my secrets file in the git repo in a way such that the secret names are visible but the secrets themselves are encrypted. In this way, I can have quick reference of the secret key names while I’m working on the code but at the same time have the secret safe enough to commit to git. An example of the secrets YAML file that is encrypted by sops is shown below.

aws:
    bearer_token_bedrock: ENC[AES256_GCM,data:gIibPeeTGYY4npuXKl7Ogjz9+r//mNzu5u0GvAZm9bghFgn24+XTqud3wyz+76OB2NRgkYRm3BruMDFdC4tTwTWeiCg0zQ5k5Tl+4ZG+/rZsoqs1Xr110F7ZQ10STq0LxxIrRpQSKqgaOZURiDJKXNBXwyzqJNdSzu8WRDis3wyqDDw6,iv:lycm17nnQAQDCmv2ePV7sF6oF/Tg0y9GMUjhNJWrMDs=,tag:C7oHCfV+DxfhcCsUnZ1YpA==,type:str]
    region: ENC[AES256_GCM,data:PDulQJpkP3Ir9nw3aA0=,iv:PMiFy9KGAU0ofW7ilF20iRD1zHS8baUZU1ki81rvOY0=,tag:opBsmKCbKfB8lVhl8RykCQ==,type:str]
    bedrock_model_id: ENC[AES256_GCM,data:u8dnWmGXO2yrWzYALTJx5a4Cq/WqUX3ALD6sEcZ2DDa2CZQOUeRIfWpLrUxr4uU=,iv:M9uJJTxooRo7lAuhmpPkp5of6QzuiNC3n0i3kfVRH4I=,tag:qRt/d1HLgoXMjGIMApm8bw==,type:str]

As can be seen, I can still see how to reach the secrets as the variable names are still visible to me but the values are now encoded gibberish.

In my case, I use age to encrypt my secrets as it is a local development project. I start by creating a key file in the same location as my .env file, say ~/.config/projectyXYZ/age-key.txt, by issuing the following command. This step also outputs a public key which needs to be noted down.

age-keygen -o ~/.config/projectyXYZ/age-key.txt
chmod 600 ~/.config/projectyXYZ/age-key.txt

Then I set the environment variable SOPS_AGE_KEY_FILE in my .env file which gets loaded by direnv each time I enter the repository. This tells sops to use this key file to decrypt my secrets file.

SOPS_AGE_KEY_FILE=~/.config/projectXYZ/age-key.txt

In my repo root, I have another .sops.yaml file that looks like this.

creation_rules:
  - path_regex: secrets/.*\.enc\.yaml$
    age: <public key from key creation step>

This tells sops that any file with the path name according to the regex pattern should be decrypted with the age key using the SOPS_AGE_KEY_FILE and the public key.

To edit your secret values, you can’t just open the file and type. As mentioned, the values are encrypted. You need to use sops <path_to_secrets_file>.yaml, which will use sops to decrypt the file and open in the default editor. Once you save and exit, the values will be decrypted again.

To simply see the secrets, type sops -d <path_to_secrets_file>.yaml.

I also have a script (courtesy of Claude), called with-secrets, that would take the secrets file, decrypt it, reformat the key names (replace “.” with “_” and all uppercase) and exports the keys and values as environment variables, shown below.

#!/usr/bin/env bash
#
# Decrypt SOPS-encrypted secret files and exec a command with the
# decrypted values exported as environment variables.
#
# Usage: scripts/with-secrets <secret-file> [<secret-file>...] -- <command> [args...]
#
# Each YAML file is decrypted via `sops -d` (never touches disk) and
# flattened: nested keys become UPPER_SNAKE_CASE env vars.
#   aws.access_key_id  ->  AWS_ACCESS_KEY_ID
#   aws.region         ->  AWS_REGION
#
# Requirements: sops, yq, age key configured via SOPS_AGE_KEY_FILE

set -euo pipefail

die() { echo "error: $*" >&2; exit 1; }

# --- Preflight checks -------------------------------------------------------

command -v sops >/dev/null 2>&1 || die "sops is not installed (brew install sops)"
command -v yq   >/dev/null 2>&1 || die "yq is not installed (brew install yq)"

if [[ -z "${SOPS_AGE_KEY_FILE:-}" ]]; then
  die "SOPS_AGE_KEY_FILE is not set. Add it to your env vars."
fi

if [[ ! -f "$SOPS_AGE_KEY_FILE" ]]; then
  die "Age key file not found: $SOPS_AGE_KEY_FILE"
fi

# --- Parse arguments ---------------------------------------------------------

secret_files=()
while [[ $# -gt 0 ]]; do
  case "$1" in
    --)
      shift
      break
      ;;
    *)
      [[ -f "$1" ]] || die "Secret file not found: $1"
      secret_files+=("$1")
      shift
      ;;
  esac
done

[[ ${#secret_files[@]} -gt 0 ]] || die "No secret files specified"
[[ $# -gt 0 ]]                  || die "No command specified after --"

# --- Decrypt and export ------------------------------------------------------

for secret_file in "${secret_files[@]}"; do
  # Decrypt to stdout, flatten nested YAML to KEY=value lines.
  # yq outputs "section.key = value"; we convert dots to underscores
  # and uppercase everything.
  while IFS='=' read -r key value; do
    # Skip empty lines
    [[ -z "$key" ]] && continue
    # Trim whitespace
    key="$(echo "$key" | xargs)"
    value="$(echo "$value" | xargs)"
    # Convert dots to underscores, uppercase
    env_name="${key//./_}"
    env_name="${env_name^^}"
    export "$env_name=$value"
  done < <(sops -d "$secret_file" | yq -r 'to_entries | .[] | .key as $section | .value | to_entries[] | "\($section).\(.key)=\(.value)"')
done

# --- Exec the command (replaces this shell process) --------------------------

exec "$@"

I use it as such, with-secrets secrets/secrets_file.yaml -- scripts/app_you_want_to_run. The front half decrypts the secrets and exports the environment variables and the second part is taken as an argument and run in the last line of with-secrets.

Gotchas

• In my case, the age key file is the key to the secrets, if I lose it, then the secrets cannot be recovered. Although, if I do lose the key file, I should probably reinstantiate all the secrets anyway.

• You can use external key management systems like AWS KMS, but that will be an external infrastructure you have to manage. And if you are indeed using something like AWS KMS, you might as well use it directly. For a local project though, sops works well.

• There is no automatic rotation for the secrets values (as what AWS KMS does for you) as the secrets resides within a file, except that outsiders cannot view it.

Other thoughts

• I really like the fact that I can see the structure of the key file while coding without worrying that the secrets will be leaked. For example, I can know that my AWS Bedrock token can be reached at secrets[“aws”][“bearer_token_bedrock”] by looking at the encoded secrets file.

• I think it works will for a small-medium sized individual project as it provides the right level of security with minimal setup and infrastructure.

Both tools are of course much more flexible and powerful than what I give them credit for in this post. I hope you found this post useful and are encouraged to dive deeper on how direnv and sops can be integrated into your workflows to make your life easier.

Please feel free to post your comments/thoughts or any questions in the comments section of this blog post.

AI coding has shifted the emphasis in software development from the act of writing code to designing software applications. This means that the focus is on clear headed thinking and well crafted application design. Not that good design wasn’t important before, it’s just that with the execution of coding now largely taken out of our hands by the likes of Claude, the center of gravity when it comes to developing a “good” application lies in the thinking. In the past, even if you had a great design, your application might still turn out bad due to less-than-desirable execution from a less-than-skilled developer.

Now when using Claude Code, you will often hear people say, “Always start with the Plan Mode”. I don’t disagree. But I find that there is a /think step before the /plan step that helps me to understand my work a lot more and prevents me from giving up ownership of my project (ever so slightly) to Claude.

The thing is, Claude’s /plan Mode, assumes that there is a task at hand and once you plan, the next step is to execute. But sometimes you don’t want to do that. Sometimes you want to ruminate a bit. Sometimes you want to question whether what you are doing is right. Or just explore multiple possibilities.

In Claude’s /plan mode, after Claude asks you the questions it wants to ask, it just want to proceed to the execute phase. Even if you are not in /plan Mode, the default mode sometimes is still eager to do things.

This is why I have a /think skill (see below) in my .claude/skills folder, which I use mainly in three scenarios.

---
model: opus
---

# Thinking Partner Mode

You are now in thinking partner mode. This is a discussion session — optionally grounded in the actual codebase.

## Rules

- You MAY use read-only tools: Read, Glob, Grep, and Task (Explore agents only)
- DO NOT use write or execute tools (Edit, Write, Bash, NotebookEdit, etc.)
- DO NOT write or modify any code
- Explore relevant code when it would inform the discussion

## What to do

- Ask clarifying questions to understand the problem
- Offer frameworks, mental models, and structured thinking
- Challenge assumptions and explore tradeoffs
- When the discussion involves code, read relevant files to ground your thinking
- Help organize and refine ideas
- Suggest options without committing to implementation

## Exiting this mode

When the user says "let's implement", "now code", "start coding", or similar - acknowledge the mode switch and begin using tools as needed.

Scenario 1: Greenfield project ideation

This is the most obvious scenario. You have an idea that’s bubbling in your head and you want to make it real. You create a new repo and enter /think mode and just start talking and sculpting out your idea.

The thing is, this mode won’t keep asking you to create a CLAUDE.md or ask you whether it can edit files, etc.. It will just talk and reflect on your conversation and push back on your ideas to make you think twice or suggest ways of doing things that you haven’t thought of before.

Personally, this works very well for me as I’m the type who likes to “talk” out my thinking.

What happens after /think-ing: I’ll ask Claude to summarise our conversation and update CLAUDE.md or the documentation in the docs folder. Usually at the start of a greenfield project, I’ll just put everything in CLAUDE.md until it reaches around 500 lines or when I’m done and ready to start implementing, then I’ll split the information into separate documents in the docs folder.

Scenario 2: Speccing MVPs or Milestones

When I have the overall idea of the project sculpted out, I break it down into MVP stages or milestones. These are big chunks of work that bring the idea through various stages of maturity. I don’t do all the MVPs that take the project to finish all at once. I do them one stage a time. Giving myself time to reflect and improve after each stage.

In this scenario, Claude is armed with the high-level thinking from Scenario 1. /think mode then helps me identify parts of the MVP specs that are not in line with the overall design or whether I’m being too ambitious (or not ambitious enough).

What happens after /think-ing: I’ll ask Claude to create a GitHub issue that will contain the MVP specs and act as the parent issue to the later development tasks. Claude will also figure out the key phases of the MVP development. After which, I can continue to refine the MVP specs or ask Claude to divvy up the work into GitHub sub-issues.

Scenario 3: Understanding and refining development tasks

Once I have the fine-grained GitHub issues that specify individual development tasks, I still use /think to ask questions about the task and test my understanding of why certain things are done.

In this way, I don’t entirely lose control of the project (at least I don’t feel like I lose control) and it helps me understand the code that was written when I review it.

What happens after /think-ing: I proceed with /plan mode as recommended and implementation.

I know it seems like the things I do with /think mode seems like what can be done with the default Claude mode. But as I mentioned, in /think mode, Claude is not so eager to get to the doing. Also, as can be seen in the skill markdown, /think mode can be embedded with a personality that helps with ideation.

At first, I constructed the /think mode because I got irritated with Claude constantly asking me whether it can start implementing things. But over time, I found myself naturally starting in /think before I do anything. I really enjoy the way the /think mode pushes my thinking, challenges my assumption and allows me to fill up more details in my project design before batting down execution plans. In fact, even when I do have the task specifications, I find myself chatting for a little bit in /think before starting so that Claude can have slightly more context beyond the immediate task or make some last minute adjustments before starting.

/think —> /plan —> /implement…

Let me start by saying that I am using the term “AI” like most people would in 2025. To more specific, AI in this article refers to a mash-up of Large Language Models (LLMs), generative AI (GenAI) not just for language but also for images, videos and audio as well as Agentic AI mediated by transformer-based LLMs. (Read the last one as “Agentic AI mediated by transformer-based LLMs” not just “Agentic AI” since the term “Agentic AI” can mean a lot more that what we are seeing). Let me also start with the end, which is that the AI revolution to me, at this moment, feels more like a “Google Maps moment” rather than an “invention of the wheel” event.

For the most part of 2025, I've been consciously using AI in my work and daily life. I’ve used Gemini for research for work and holiday research. Nano Bananas to generate images for my blogs. ChatGPT to construct a learning companion for my daughter. NotebookLM to do deep dives into technical topics. Taking the agentic AI for research use case further, I’ve also designed a Claude Code agent to extract financial data to conduct market analysis. GitHub Copilot and Claude Code were my tools of choice which I used extensively for coding tasks, both personal and at work. I’ve even tried my hand at writing my own AI agent, just for kicks.

Throughout my experiments I was on a lookout for what makes this latest technology revolution special. I was interested in it because I wanted to find out what would be its impact on not only my career but also how it would impact my daughter’s future.

The first thing I noted is that AI, like any tool, needs to be learnt and practised with in order to derive benefit. I had to learn how to prompt it and how to release information to it step by step. At times, it felt like I was coaching an intern. This is also remarked by many others. Like any intern, they are also prone to missing details and steps. For example, my market analysis agent is able to performs most of the task but keeps missing minor steps here and there.

Next, I found AI a fantastic learning partner on topics that I have some knowledge on. I was trying to learn Active Inference and Variational Message Passing. Solely reading papers was not working very well for me as sifting through and deciphering information from various authors required holding a great deal of information in my head. I was able to use AI as a sparring partner and a knowledgeable tutor. It’s quite another story for things that I’m not too familiar with though. An example is the financial market analysis agent that I was trying to come up with. I was not able to ask questions about whether the analysis that AI was doing is correct. When AI came up with a suggestion, I can’t help but feel suspect.

Another related observation is that AI is great at boilerplate work, especially for coding work. AI saved me a lot of work in navigating the peculiarities of software libraries and frameworks. It got me to a starting point for a new software project really quickly, like setting up the project scaffold and writing tedious API specs and schemas. It was also tremendously helpful in helping me debug infrastructure issues like cloud configurations or deployment script errors. One thing that slightly caught me off guard is how much additional context engineering that is required to get consistent results from AI for software development.

However, when it came to work that required creative thinking or iterative experimentation such as data analysis, I found AI a little lacking. Not lacking in the sense that AI can’t generate outputs. But “lacking” in the sense that, 1) I don’t feel like I’m any wiser after the fact since AI did the thinking for me and 2) for tasks where I didn’t have any “mechanical issues” like for data analysis where I know the tooling and syntax quite well, I felt like I had to do a lot to get the results I would have gotten really quickly. In short, for tasks that the main obstacle was realising the idea I have in my head with tooling that I’m decently well-versed in, AI was not much help for me.

Lastly, the latest AI revolution has changed the man-machine interface and has automated a lot of work that was previously impossible, this has shifted the emphasis of value creation heavily to critical thinking. To paraphrase a big-data age saying, garbage thinking, garbage output. In the software development space, it means more effort is spent designing the infrastructure architecture, application functional modules and evolving the software development lifecycle to help product owners come up with task specifications faster and incorporating a more rigorous review process. AI doesn’t take away effort in thinking and learning fundamental concepts, it emphasises it since the effort of execution has been removed from you.

As I reflect on the points above, the closest historical analogy to the current AI that came to my mind was Google Maps. For those who remember the days pre-Google Maps, people had to call, ask around or fiddle around with paper maps, if they could get their hands on one. Even then, people got lost a lot and waited around for others a lot. Google Maps changed that. Nowadays, I can drop into any decently developed city and navigate my way around. Google Maps also enabled a whole slew of new economic activities like deliveries and small business advertising. It became my go-to tool for finding out where things are and how to get to places. It brought me a great deal of convenience and it would be hard to go back to a state where I don’t have Google Maps. However, it did not simply automate my life away or make the world come crashing down. It did however diminish my ability to navigate without preplanned routes and instructions.

Now, I’m not saying AI is the same as Google Maps (I said “closest analogy”). However I do believe that when the dust settles, people might think of 2025’s AI like how I think of Google Map in 2025. I know it doesn’t feel like it given the frenzy surrounding AI. But like what Howard Marks says, people often alternate between perceiving reality as being extremely hot or cold when it’s more often than not somewhere in between. The current AI is not fully autonomous or deterministic. Neither is it creative without humans. It is however very good at doing well-scoped tasks that were previously “un-automatable” due to pattern recognition requirements that used to be considered “human-level”. Lots of new economy will be built on top of this. Lots of old economy might be destroyed. It will in one shape or another become an indispensable part of everyone’s daily toolkit (though how long this will take is another important question, especially for the markets) and it will shape our behaviour.

But it does not make us less human. In fact, it emphasises the part of us that is uniquely human, our ability to think and create. Yes, there inherently a danger that AI in its current form, wielded by big Tech, might take away people’s ability to think for themselves, case in point, social media. That will be true. But it will also be true that it will force the conversation on the meaning of work and the meaning of creativity on society. And as always, we end up somewhere in the middle.

Let me first state up front that this is not a post to make a case for or against vibe coding. In my mind, the cat is already out of the bag. Vibe-coding is here to stay, like it or not. The question is, where will things end up? In this post, I conduct a little thought experiment of a potential future world where vibe-coding continues to exist and what that might signify for personal computing.

As with all thought experiments, this post says nothing about the plausibility of such a world materialising within any given time horizon (although I think that the future imagined here is actually not that far away) and it will emphasis certain aspects of the dynamics while downplaying others. If you are still interested, read on…

Let’s start with some definitions…

I asked Gemini for a concise definition of vibe-coding and personal computing. Its responses are quite reasonable from my point of view. So here they are…

Personal computing is the use of a general-purpose computer designed for direct, individual control by one person. The devices, such as desktops, laptops, tablets, and smartphones, allow a single end-user to execute various tasks and applications without requiring a centralised operator or time-sharing with other users.

— Google Gemini 2.5 Flash

Vibe coding is an AI-assisted software development approach where a user generates functional code by providing natural language prompts and high-level descriptions to a Large Language Model (LLM) or AI agent, often with minimal or no direct review of the generated code.

— Google Gemini 2.5 Flash

These are the definitions that I will be going with in this post. Note the bolded parts, we’ll come back to them later.

The incompatibility of vibe-coding with the current software delivery model

There has been a myriad of complaints about vibe-coding. Many of them revolve around the consistency and robustness of AI generated code. Some of them are true. Some are due to the injured pride of software developers. Some of them will be remediated as AI gets better. Some are due to the fundamental flaws of Large Language Models (LLMs) as a “next-token generation engine”.

However, if we look beyond the surface of the complaints, one of the key source of these complaints about robustness are due to our current model of software delivery, which takes two main forms: Software-as-a-Service (SaaS) and Software-as-a-Product (SaaP).

In SaaS, a software provider builds the software and users access the software via a common portal. An example is Gmail. In SaaP, a software provider builds teh software and delivers it to user devices. Examples include Adobe Photoshop or Microsoft Office. Of course, these are two end of a spectrum. There are a million shades of grey in-between, like how Google develops a Gmail app and it is being used on users’ phones or how Microsoft allows for on-premise deployment of its SaaS products.

Either way, users interact directly with the software that is being built by the software provider. The users conform to the way the software provider has decided how the software functionality is to be delivered. Moreover, each software functionality is confined within the software product built.

In this way, the complaints around the lack of robustness of code generated by AI is well-justified (for now). After all, if the software provider vibe-coded and delivered the software with massive flaws to users, the blast-radius would be huge. Any security flaw, once exploited, could expose millions of users.

But what if the paradigm for software delivery changes?

A potential future: Software-as-Tools and N=1 Software

Let us now imagine that all user interactions with software is now intermediated by autonomous agents. After all, as I mentioned in my previous post, the real change that the GenAI revolution has brought about is a paradigm shift in the man-machine interface.

Users will all vibe-code. They will simply tell the autonomous agents what they want, and the agents will flexibly select the specific functionality they require to meet their users’ demands — meeting the definition of vibe-coding. Recall that currently, software functionality from a user’s perspective is “boxed” up in particular software applications. Yes, software providers have provided APIs for other developers to access fine-grained functionalities and compose those functions into other applications. But this fine-grained access is currently mostly limited to software developers — most of the world population aren’t software developers. If we look again at the definition of personal computing, we will see that being able to access fine-grained functions via API would not constitute “direct, individual control by one person” or personal computing.

The agents will develop N=1 software to do what the users need at that moment. The agents will take the fine-grained functionalities provided by software providers to construct dashboards, infographics, websites, micro-apps, etc. for the user on the spot. This presentation of information and performance of action will be for the user only, hence “N=1 software”. From this vantage point, the lack of robustness of software becomes less of a concern, since there is only 1 user. What we now think of as software prototypes/demos will in fact become the user application (if the word “application” still means anything) in the future.

Software providers will no longer take care of end-to-end user journeys. Software providers will simply provide small piece-meal functions, which in today’s agentic AI parlance are called tools, to be consumed by autonomous agents. Software providers as we know them today will retreat into the background just like how IT networking has retreated into the background of web-application development. This is where software development practices will continue to live on.

Requisite Conditions

I know that even at the time of writing, we are already able to catch glimpses of the future proposed above. More and more software providers are providing Model Context Protocol (MCP) interfaces to be used by AI agents. However, from what I can see, a lot of these functionalities are provided by a small set of companies and to a small set of users (usually software developers or very tech savvy people). In order for the above scenario to materialise in the mainstream, I believe a set of additional conditions need to be satisfied.

A trust mechanism for software

In the new world described, one can no longer say, “XYZ company was at fault for not taking care of this or that thing that caused me to lose money/get scammed/not be able to get my stuff on time, etc.”. Everyone in fact becomes a producer of software of sorts. Everyone is like a chef (or rather everyone has a chef) that takes raw ingredients (tools) and makes a dish to the person’s liking (N=1 software). If something goes wrong, then do we know if it is the ingredient that is at fault? Or is the chef (AI agent) a bad chef?

One way to circumvent this problem is to have all software tools openly reviewed and scanned reviewed by an independent (or maybe decentralised) body and the results published on a decentralised ledger. The execution of the code should then reference a repository where the code version/provenance could be transparently reviewed. On the point of transparent execution of code, the reader might want to check out the Unison programming language on content-addressed code.

Smaller models capable of running on personal devices

Going back to the analogy of AI agents being personal chefs, an underlying point about that is that the personal chef knows your personal likes/dislikes. In the case of N=1 software, it would be knowing what’s the best way to present information so that you can consume it easily, knowing your intellectual blind spots and taking the effort to help you cover them or understanding what you might be interested it given what your recent experiences. This cannot happen if everyone is using AI models that are designed for everyone else as well.

My personal take is that the current transformer-based large AI models will not be the paradigm that takes us into the age where AI is ubiquitous. There should be a different AI approach, perhaps something like the Tiny Recursive Model (TRM) combined with modular AI (see below).

Modular AI

Just like how software development went from monolithic applications to modularised plug-and-play libraries, the same could happen to AI. In the new world of personal computing, there is a need for access to knowledge (via large models, like how the internet has Wikipedia), personalisation (via small local models) and actuation (via tools). All three can be built and owned by different parties. All three are required for AI to fulfil its full promise for individuals, in my opinion.

For that to happen, the current way of running AI models has to change. AI models have to be reconfigurable and be able to “hot-swap” modules. Say I have an LLM that has general knowledge and the ability to “speak”. Now I want to inject embeddings that encodes my preferences or add in a new sensing modality like the ability to “see”, there should be a way to do it. Or it may even be the other way around where the LLM simply provides the embeddings and everything is orchestrated locally by my personal model.

Now, if any of the models change, for example the embedding size changes, how will the other models cope with it? Currently, the way this is overcome, by virtue of the advent of LLMs, is via natural text. Like how LLMs choose which tools to use. But natural language is a very low bandwidth way of communicating information and what happens if it is visual information? If the information to be conveyed cannot be conveyed as text, then the current way of communicating is via actual neural network layers that are run on the same chip (or in the same AI model).

To this, I do not have an answer yet.

Decentralised ownership of AI models and storage of personal data

Together with the point above, there will be a need to store the personalised models and data in a decentralised manner. This could perhaps again be achieved via a decentralised ledger, where everyone owns their own data which is used to train their own model (the personal “chef” or AI agent) and is only accessible by the individual.

Of course, there is no guarantee that such a decentralised ownership model will materialise. If we look at what is happening now, the large AI models, by virtue of their size, are owned by large corporations. Those same corporations also happen to be in control of our data. However, I cannot imagine this to be good if AI were to become ubiquitous and the power these companies will wield over society.

Widespread support for provision of services via agents

This goes without saying, for the “N=1 software” future to become reality, more service providers need to provide their services as tools. I’m not just talking about those names that you wouldn’t be surprised to see in Silicon Valley or tech startups claiming that they would revolutionise the world. I’m talking about everyday services like restaurants, plumbers, etc.. Because having to jump between the old (phones, websites, apps) and new way (AI agents) of doing things simply won’t lower the inertia of change that people have. This tells me that that future may not come as soon as tech billionaires want us to think.

What does it mean?

General public

For the general public, as I’ve said before, this means that the way humans interact with machines/software will complete change. Everyone will have their own personalised view of things (whether that is socially desirable is another question). On top of that, everyone will need “people” management/communication skills. The reason why I say this is because anyone who is using AI agents extensively now will realise that dealing with AI agents is very much like dealing with a person. And the better your communication skills are, the better you can get things done with AI agents.

Software developers

As for software developers, I actually don’t think that this will be the end of software development. Just like how many computing-related activities such as networking, sort of retreated into the background, so will software development as we currently know it. Software developers will still be required to develop tools, which are the fine-grained functionality that probably requires software development rigour. Of course, the skillset involved will be very different from today and it will also be AI-assisted.

There will be also be other areas that require software engineering that will grow, such as in decentralised ledgers (see later section).

Service providers (Sellers)

Service providers (or software providers) will also have to get used to a very different world. Think Uber which takes care of the end-to-end user journey of getting people from point A to point B. You book a ride, see when your ride is arriving, pay, lodge complaints, etc., all in one app. And that user journey is designed and engineered by one company with a particular philosophy or approach.

In the new world, service providers not only need to deal with the fact that there will no longer be a consistent representation of their services to the end-user, they also need to contend with the fact that there is an additional layer of mediation which is the AI agents between them and their users. This is true regardless of whether the AI agents are owned by large corporations or individuals. The result of this is that service providers will be “out of sight-out of mind” for their customers. Just like how when AliPay or WeChat became the default mode of payment in China, Chinese banks only saw 2 transactions each month from their customers: one when the salary came in and another when the salary got transferred to AliPay/WeChat. This means that they will no longer be able to influence their customers or lock them in like they can now.

Decentralised ledgers / Blockchains

With the advent of generative and Agentic AI, the need for proof of authenticity is never higher. And as mentioned before, decentralised ownership and storage of personal data might require a mechanism like a decentralised ledger. Maybe there are other ways to do this, but for now blockchains seem like a good candidate to store model certifications and ownership proofs.

New economy of tool aggregators and AI trainers

This new world of personal computing will also spawn new economies of tool aggregators and AI trainers. Just like how the internet age brought us AirBnB and Bookings.com, the new world will have its own aggregators for AI agents to discover new tools.

There will also be AI trainers to help people train their personal AI. Or maybe even AI trainers that train specialised AI agents that can be replicated infinitely to anyone who needs a specialised task done. For example, a 10x software developer in Rust could train an AI agent that codes as well as he/she does. Put it on the blockchain and market it via one of the tool aggregators.

Taking into account radical uncertainty

The impact of AI on personal computing expressed here is by no means exhaustive and certain (remember, this is only a thought experiment). To borrow the term radical uncertainty by Mervyn King, there are many unknown unknowns for which we cannot “price” in the effects that they might have.

Conclusion

This is a thought experiment on how the future of computing will look like in the age of AI. I do not purport that the world proposed is a plausible one, only a possible one. And for sure, there are certain elements that are far-fetched, certain elements oversimplified and certain elements that would require fundamental changes that are harder to come by than I can imagine. But I do believe that some aspects of this new world will materialise in the very near future and it is rather exciting to think about it.

I was inspired by Gemini Live, which allows you to speak to an AI agent, and Claude Code, where user interactions takes place via a terminal, and I thought to myself, “hmm… Shouldn’t be too hard to build something simple which can do both myself, right?” As you can probably guess by now, the answer is “wrong!” But you can also guess that I was foolhardy and naive enough to not realise this and to go along with the idea of building my own AI assistant, Dexter. Otherwise I would not be writing this post. So here goes…

Before we go further…

In this post, I will be going through what Dexter can and cannot do from a user’s perspective as well as some of the key learnings and challenges encountered while writing Dexter.

For more technical details, see the code repo here.

Regarding the use of AI, this post is written entirely “by hand”. During the process of building Dexter, AI (Gemini 2.5 and Claude Sonnet 4) were consulted. I would say >95% of the code was “hand-written” (see here for more details). You might ask why not use AI to write the code? Isn’t that what everyone is doing? Well, this is a hobby project for me to learn and sharpen my skills. Where’s the fun if everything is done my “someone” else?

Who is “Dexter”?

Agent Dexter, is an AI agent that runs in my own computer. It has a terminal frontend so that I can type my questions and supports voice input so that I can wake it up with the wake word “Dexter”, ask my question verbally. My queries are then be handled by an AI agent which uses a Large Language Model (LLM) to decide whether to answer my question directly or use tools available to it to answer my queries.

While Dexter runs on my computer, the key intelligence comes from the LLM which resides on AWS’s servers that are queried using AWS Bedrock. So, if you intend to play around with it, know that not everything takes place on your computer.

Another thing to note is that, at present, Dexter only runs on a Mac. To make “him”1 run on a Windows or Linux machine, you’d probably have to make some changes to the code or accept that you cannot interact with Dexter via voice commands.

What can Dexter do?

Dexter has two operation modes: voice and full. See instructions on invoking the two modes here.

Voice mode

In voice mode, you can wake up “him” up by saying “Dexter” and then asking your question.

Full mode

In Full mode, you can either wake Dexter up as in voice mode or you can directly type your queries into the terminal.

The user experience in full mode is still a little laggy as I’ve not implemented streaming inputs and outputs.

Key Learnings & Challenges

AI agents enable flexibility in user interactions

One thing that was obvious to me is that AI agents enable user interactions that can be quite fluid and flexible. I could sort of imagine how if I were to build functionalities like searching for news and performing general tasks like adding 3 to 5 in the past, I’d have to do so with some buttons or drop-downs or tabs. Now, AI agents powered by language understanding from LLMs allow me to just speak and get things done.

Of course whether this flexibility is desired in the first place, depends on what you are building. I can imagine cases where you don’t want too many free flow conversations, and you just want to guide the user down a “narrow path”.

Tools make agents useful but effort required in orchestration and prompting

Another nice thing with AI agents is that you can put tools in “their hands” and it would immediately be “integrated”. For example, in this little project, I wrote a simple search_news tool that queries Perplexity. All I had to was to tell Dexter that this tool is now available and it would be used the next time I start Dexter up.

In a separate project that I did concurrently, I also tried using MCP servers integrated to the Strands Agent framework that I used for Dexter. The integration process was rather painless too.2

That being said, again speaking with experience from other projects involving AI-assisted coding, while effort maybe reduced when it comes to modifying code to integrate new functionalities, effort is need to prompt the AI agent properly so as to ensure that the agent’s behaviour and way of doing things is consistent over time.

Conventions are starting to emerge in the AI agent world

I’m also noticing that conventions are starting to emerge in the AI agent world over time. For example, dictionaries with a more or less standardised format (such as {“role”: “user”, “content”: “…”}) in keeping a record of the conversation between the AI agent and the user are being used by different frameworks.

Though I wouldn’t call this phenomenon a “standard” yet as I feel that things will change very quickly in the AI world.

Building Dexter’s speech-to-text capabilities was fun

I started out wanting to use RealtimeSTT for the STT engine, as it included wakeword detection and voice activity detection (VAD) out of the box. However, it does not support Mac GPUs and running it on CPU was just too slow. That’s why I had to go with MLX Whisper and implement wakeword detection and VAD myself.

The experience of going through researching different frameworks and working it into an application was rewarding.

Overcoming terminal race conditions

If you look at the way voice mode is run, you would see that everything is run in a single Python program with queues handling data sharing.

However, when I tried to do the same for full mode, where the user can also type into the terminal (like Claud Code), I ran into a whole lot of trouble as a result of race conditions coming from the various running processes competing for the terminal’s “attention”.

In fact, this was the key challenge that took up half the development time, by my estimation. I had to give up in the end and go for the approach where I have 3 separate processes communicating over HTTP.

This is the key learning for me on the coding front this time round.

Final Reflections

Dexter was a fun little hobbyist project to dip my toes into the world of writing AI agent-enabled applications. While I have to admit it took more time than I thought it would and struggling with the terminal race conditions was a huge pain, it was fun figuring out how to integrate STT with Dexter. And maybe Dexter will become a platform for me to explore other Agentic AI functions such as integration with a knowledge graph or a persistent memory layer for AI agents.

NotebookLM burst onto the GenAI-enabled learning scene with its AI-generated podcast that summarises any topic for you based on digital materials you have on hand. For those who are not familiar, NotebookLM is a web application by Google that allows you to upload text sources like PDFs or websites and then allows you to ask questions about the sources, which it then proceeds to “read” through all of them to find the answers you are looking for. As the name suggests, it’s a tool that is touted for learning.

I’ve been using it for a while for my own learning on areas that are technical, such as Active Inference and more general, like Universal Basic Income. In this post, I summarise my reflections on my use of NotebookLM, tips on using it better and things I wished NotebookLM had.

General reflections

If I were to summarise what it felt like using NotebookLM, I would say this:

Of course, the AI-generated podcast wowed me initially. But it was the ability to peruse a curated set of information, hold it in memory and answer your question coherently that made me feel that NotebookLM was really a useful tool for learning.

I was struggling with learning Active Learning which required a very different way of thinking about machine learning that I was used to and involved knowledge in highly mathematical topics like variational learning or Partially Observed Markov Decision Processes (POMDPs). On top of that, the papers were written by neuroscientists and biologists which have a different way of expression compared to mathematicians and computer scientists and were writing about the application of Active Inference in neuroscience, an area which I had very little knowledge or intuition about. There was no way I could hold multiple papers in my head and simultaneously summarise and synthesize the information in them.

That’s when NotebookLM came to the rescue. I uploaded all the papers I could find on Active Inference and I was able to ask questions that perplexed me in some papers to which answers could be found in other papers. The Active Inference community is not very big and the author sometimes assumed that you knew what they were talking about. A memorable moment for me was when NotebookLM demystified the difference between state, policy and parameter inference for me in the Active Inference framework. NotebookLM synthesised information from multiple papers, filled in the gaps from one paper with information from another and presented the information in a coherent manner.

Another really nice feature of NotebookLM is that it is able to generate mind maps for you. Take the mind map generated from my “Universal Basic Income” notebook. It organises the main threads of information for you to explore across all your sources. It is also able to generate other documents like FAQs, briefing docs and timelines. I can imagine a feature like timelines to be useful to people like lawyers.

The latest feature includes generating a video summary. Something I believe visual learners would find useful.

I also didn’t feel like I was a passive recipient of knowledge while I was using NotebookLM, you know, like how you feel like you’re not really using your brain if ChatGPT is simply giving you all the answers. I was an active participant. I had to find the sources of information and I had to think of the questions to ask. It is a product which lives up to its purpose as a study companion.

Usage Tips

Include a “foundational” source for technical topics

NotebookLM can only give you answers from the sources you upload. That’s fine for many cases. However, when you are trying to learn a technical subject and all you have are academic papers, it may not be able to explain things as well to you as you would want it to, simply because it does not have the knowledge. Academic papers often assume understanding in certain foundational topics that you might not have.

For example, I uploaded a copy of Bishop’s seminal book “Pattern Recognition and Machine Learning” when learning Active Inference, and as a result, NotebookLM was able to explain the use of factor graphs in Active Inference better. This is because it had access to the foundational knowledge in Bishop’s book.

Be balanced with your input sources

Again, stemming from the fact that NotebookLM can only know what you give it, take extra care to be balanced with your sources. If all your sources portray a particular topic negatively, then you will only get answers that paint the topic in a negative light, vice versa.

This is especially important when using the “Discover” feature to collect input sources, where you can ask NotebookLM to find information sources for you with a text prompt. For example, when I was using NotebookLM to discover sources on Universal Basic Income, I said “I want to learn about the pros and cons of Universal Basic Income”, instead of “I want to learn what is good about Universal Basic Income”.

Understand that it’s ability to answer you is limited by the LLM that supports it

At the end of the day, the responses you get are still powered by the LLM sitting behind NotebookLM. So do note that there will be some things it cannot do. I realised this when I tried to ask it to perform a step-by-step derivation of an equation. I guess I was thinking too highly of NotebookLM at that point in time.

Feature Wish List

Live data or API access

I wish there were some way to update my sources automatically either by subscribing within NotebookLM or API access. I use Perplexity to keep up with news and I just kept thinking that it would be great if I could pipe some of these articles into notebooks in NotebookLM for me to query later. This is useful if you are keeping track of an on-going event like the Russia-Ukraine war (and you want to use the timeline feature).

Video or image sources

Currently NotebookLM deals predominantly with text sources. Given the progress in multimodal LLMs, it would be great if it could also parse images or videos. In this way, if I see a great talk or lecture, I could upload it and query it.

Better math formatting

Currently when NotebookLM saves its replies as a note, the math is shown as raw LaTeX. I don’t think it’s too much to ask for the math to be properly displayed, is it?

Handwriting support on tablets

I’m someone who learns better by writing. It’d be great if I could write notes on my IPad in NotebookLM and the LLM can parse my handwritings as well. I know it’s a tall order, but hey, it is a wish list after all…

Conclusion

I thoroughly enjoyed using NotebookLM. Nowadays, when I find myself having too much information to process about a particular topic, NotebookLM is my go-to tool. I hope it keeps on improving as I honestly think that it is useful not just for students but also for professionals.

I was just reading an old 2018 paper on computer vision and a thought suddenly struck me.

Objectivity is lost in the age of GenAI.

Or at least, the facade of objectivity is lost. Depending on your personal philosophical leanings.

Take for example, Structure from motion (SfM). It used to be that people had to calculate homography matrices and perform triangulation. It was cumbersome and complicated. But if you had the right information like the camera’s focal length and all, you’d arrive at a physically sensible solution. In other words, you would know the camera’s pose (rotation, position,etc.) and you’d have a measurement of the object’s distance based on hard math. Nowadays with segment anything and depth anything AI models, you’d be able to get a depth map of the objects in the picture with little to no work. The catch is, everything is now relative. The depth map (or the distance each image pixel is from the camera) is what the AI model thinks it is. The change in depth distance between 2 pixels is what the AI model estimate it to be based on the images it has seen before.

The origin of this thought was the Deep Structured Generative Models paper, which led me to think about how I would do the same with GenAI (perhaps more on this in a later post). It then led me to think about how I would use a combination of image generation with background removal to generate assets, use LLMs to generate semantic information and video generation to generate different views of the digital assets, etc..

💡

Actually I started reading the paper because I had problems with pure image generation via ChatGPT when creating thoughtsre’s logo.

Anyway, it then occurred to me that each of these steps feels like I am asking a person to do something for me. I didn’t have to know how to calculate homography matrices anymore. Each step was essentially a black box.

This may be convenient for me. Just like how a company outsources certain functionality so that it can focus on its core business. We can take this analogy further and think about what happens when a company unwittingly outsources its core business functions in pursuit of efficiency. History is replete with bad examples. Another story for another day…

Coming back to the thesis of this article, when one asks another person to do things, the outcome is at least partially determined by the inherent biases and capabilities of the person. This is why we keep comparing between models of different companies and how they perform on different benchmarks. It’s like how we compare the design proposals of various interior designers before we decide to engage one.

Setting success criteria and measurement metrics help. But think about it, if it were a human, there is always some element of subjectivity that is due to the human. Increasingly this will be true for machines as well. The output will be colored by its training data (or experiences). If we blindly accept its outputs, then we are also accepting its subjectivity, so to speak. Just like how if you employ a director to make a movie, you are buying into his artistic direction and vision.

Going back to the analogy of interior designers, it’s like judging the designer’s capabilities by their use of space or adherence to your requirements. Two designers might use space equally efficiently but one is just better at understanding your asks, perhaps because his life experiences and tastes are just more similar to yours.

To try and define success exhaustively would be to make the work of metrics definition an industry unto itself. Then where’s the productivity gain? In any case, it’s a fool’s errand. AI’s capability will far exceed human capability of monitoring. See AI 2027 or recent incidents on AI models employing blackmail to avoid shutdown.

One might argue that this is human society works. We deal with this kind of subjectivity all the time. All humans hold subjective opinions. It can work with machines as well. To wit, I say maybe.

Note that, despite the progress made, human society is still filled with suffering and unfairness. Can we assume that a human society mixed with machines will be better? Moreover, there are many more humans each developed with their unique experiences. The diversity averages things out. And human society has had a lot of time to derive common ground, albeit through painful experiences.

Can we say the same of large AI models in the hands of a few?

Others might also argue that objectivity has never existed. That is not true in select fields like mathematics. But to be honest, mostly true in everyday lives. We have, however, found ways in human society to deal with it. We have laws and cultural norms that are distilled over time.

Importantly, we have, as a species, valued objectivity and at least tried our best to uphold it. This is why we have laws that require proof beyond doubt and chain of evidence to convict crimes. That is why we pursue knowledge in hard science. That is why it’s different when we meet face to face, instead of via Zoom.

With GenAI’s ability to create realistic content, it is hard and mostly too tiring to find out what is true anymore. We will be inundated by generated content delivered to us via the well-hone machinery of social media that constantly sways our thinking and judgement.

At the end, what makes humans want to work with others is trust, which comes from cultural similarity or some other binding force like the law or familial bonds. And I think there are a few ways where we might be able to salvage objectivity.

One, devise a legal framework to treat AI as persons. We urgently need to adapt our current legal framework to treat AI as persons with legal liabilities. Fortunately, we already have what we need. A company is a legal person that is not a human. Hence, just like a company needs to be capitalized and there may be humans (up to a certain point) that are liable for its actions, the same goes for AI.

Two, adapt the use of blockchain as a record of AI actions and human outputs. With the proliferation of GenAI, authenticity becomes a prized commodity. This is where AI actions and human outputs need to be recorded in a globally immutable records like a blockchain. On top of that, with the help of the legal framework, the outputs of AI have to be watermarked, like SynthID.

Three, use small models but lots of them. One of the key dangers in the current GenAI revolution lies in the fact that only a handful of operators own and are responsible for training these models. The reason for that is that these models (even the so-called resource efficient DeepSeek models) are so expensive to run and train. There are some that would advocate stopping or slowing the development of AI. I personally don’t think that will work. The cat’s already out of the bag. In fact, I feel that the research and engineering community needs to urgently work out a way to have the same capabilities but in a much much smaller form factor. In other words, find a way to enable AI models to have high intelligence but lightweight enough to be deployed on mobile devices. Importantly, these AI models should be able to learn quickly, easily and independently, such that they are truly personal assistants to people. There should be at least as many independent AI models as there are people. To do this, we would need to shift the current AI paradigm away from the transformer-based models that are super heavy and achieve quantum leaps in AI hardware. In this way, we achieve a parallel “machine society”, where hopefully diversity of independent AI agents helps to average things out and we can seek some form of objectivity within.

That being said, it is unclear to me whether any of these methods will work in the long run. Maybe it will only serve to delay the inevitable. Take for example the third suggestion of creating a parallel machine society. How we would ensure that the resultant AI agents would really be diverse? Even if they were, would humans be seen by the machines as a cohabitant or a parasite?

I’d love to hear what you think. Leave a comment and let me know your thoughts.

Unfortunately, I am short on patience. Also, I believe the situation is aggravated when teaching one’s child because you inadvertently bring in your expectations for the child in your communication. That creates extra pressure during the tutoring sessions. Every frown, every micro-expression, is taken in by the child and can be construed as disappointment. Over time, it can generate a lot of stress.

I was then inspired by a conversation I had with a friend and I started to think about ways to alleviate this using AI. Perhaps there is a way for me to help my daughter learn, but taking my lack of communication skills and impatience out of the picture.

A ChatGPT Chinese Learning Companion

What I did was to construct a custom GPT on ChatGPT to help my daughter revise her Chinese spelling. The prompt used to generate the GPT, which I call 熙师, is shared here.

Basically, I uploaded a copy of my daughter’s primary school Chinese curriculum and asked chatGPT to generate explanations, example sentences and simple multiple choice questions to test her understanding.

As this GPT is targeted for use by young children, I can’t share it publicly via ChatGPT’s marketplace as per OpenAI’s policy. You will have to set it up in your own chatGPT subscription. Instructions are shared in the repository as well.

My daughter’s initial reactions

My daughter was really amused when I told her that I had “coded” a Chinese spelling revision spelling app for her. When she used it, she took to the interface quite naturally. The ChatGPT function where the text is read out by a friendly voice helped ease the initial adoption challenges as well.

The fact that my daughter can ask ChatGPT to keep repeating till she understood also made it easier for my daughter to learn. ChatGPT has infinite patience.

Also, she really appreciated the ability for ChatGPT to explain things to her in English when she had a hard time understanding. Of course, I had prompted ChatGPT to only give explanations in English if needed and not by default. And the questions are not in English so she still had to read the questions in Chinese.

What I liked about it

The first thing I was surprised by was how well ChatGPT read the Chinese curriculum PDF that I uploaded. All the Chinese spelling words were read accurately and segmented properly by the lesson of the week. No formatting or special instructions on how to read the PDF needed. This goes to show how far AI has gotten in document reading in the last few years. See the image below to see how all the lessons are properly enumerated (I asked ChatGPT to enquire which lesson to revise before starting).

On the whole, ChatGPT was able to generate relevant content for each of the Chinese spelling phrases. It’s not perfect but much better than I expected.

As instructed in the prompt, the ChatGPT learning companion always took a very encouraging tone. It was pretty encouraging for me to see the positive response my daughter had to ChatGPT’s encouragement.

The text-to-speech feature made the encouragement feel more real as well. The voice was personable and gave my daughter another medium to receive information other than reading the words on screen.

What I didn’t like about it

I guess ChatGPT’s model isn’t too well trained on the Chinese language. (When I say “isn’t too well trained”, it probably is still better in Mandarin than the average Singaporean Chinese person.) some of the sentences constructed by ChatGPT for the multiple choice questions didn’t quite make sense. I guess it could be an artifact of ChatGPT trying to construct sentences with incorrect usage of the spelling phrases. After all, it’s still an autoregressive model that predicts the next token given the previous ones.

My next gripe is with the text-to-speech function. While it provided a higher level of personal interactivity, the voice generation took quite a bit of time and as such the lesson didn’t flow too well if we used the speech generation at every step due to the waiting. Also there was limited control over the speech generation in ChatGPT. For example, I found the speech generated to be quite fast and there was no way for me to slow it down.

Lastly, because the behaviour of the custom GPT is entirely controlled via the instructions given to it during its construction (see “Instructions” section in the repo), changing the prompt instructions can result in unexpected change in the flow of the lesson.

For example, I tried to support two lesson modes. One for revision of the entire curriculum where only 5 questions are asked for a short exercise and another just for the spelling test of the week where all 10 phrases should be revised. I found that changing the instructions in one would result in changes in the way the other is presented.

That being said, I see the above difficulties/shortcomings as engineering challenges that can be overcome by current techniques and by OpenAI deciding to open up more features. They are not insurmountable. It’s simply a matter of enable more features for the public.

What I thought would have liked to have

Following from the last shortcoming mentioned, I would have liked to be able to split the lesson into small independent parts that I can prompt engineer separately but yet bring together under one coherent agent. This is will overcome the problem of changing the prompt in one part affecting behaviour in other parts and is possible using techniques like Model Context Protocols (MCP) and Agent2Agent (A2A) for agents to communicate with each other. And of course, OpenAI to open up these features to the public.

I would also have loved the ability the specify buttons for standard responses, such as “Can you explain it to me again?”. My daughter was spending quite a bit of time typing.

Lastly, I would have loved the ability to introduce other medium of communication like writing and drawing because learning Chinese requires the student to learn how to write the Chinese characters. This would of course be more involved in terms of software engineering and would require ChatGPT to quickly be able to read my daughter’s handwriting.

Looking into the future and my personal reflections

As I reflect on this short foray into using ChatGPT for education, the first thing I realise as I was committing my “code” to the repo is that the “code” is just the prompt. Of course, I understand that this is just a hobbyist level prototype and nowhere near production grade. But heck, this would have been impossible a few years ago. The ability to generate natural sounding content with high-level instructions is quite phenomenal.

Combined with non-text content generation like videos, I think AI really has a chance to make a difference in education. There will be debate on whether it makes our children lazy or take away their ability to think critically. To that, I’m still not quite sure yet personally. Maybe that will be true. Maybe the effect will be different at different levels of learning.

However, in my case, my personal take is that I’m not asking AI to solve problems for my daughter. I’m not asking ChatGPT to think on her behalf. What I’m asking ChatGPT to do is to be a companion to my daughter, helping her think and learn while at the same time not having to deal with my shortcomings as a teacher.

I can imagine how this might be helpful for students who may not have access to private tuition. Or how AI learning companions can be equipped with the latest/best teaching methodologies that achieve the best learning outcomes for our students.

For now, some of the shortcomings such as not being able to practice writing Chinese, will stop me from using this for my daughter’s Chinese learning. Also, she has enough homework already.

Maybe I’ll use it when we are on a holiday, just to keep up a low level of engagement with schoolwork and prevent too much backsliding.

Just maybe…

Here’s how I think about AI land and how it looks like.

Broadly speaking, I see AI land being made up of three interconnected realms: algorithms, hardware and applications, within which developments in what we generically call AI land exist.

Each of these realms have unique characteristics. But the thing is, activities within each realm affect each other, sometimes constructively and sometimes destructively.

Realm of Algorithms

The realm of algorithms is inhabited by citizens such as the Convolutional Neural Network (CNN), Long Short-term Memory (LSTM), Graph Neural Networks, Bayesian Optimization, the Transformer (who in the last few years has become an all important inhabitant of the algo-realm), etc..

The work carried out here is in discovering new algorithms, improving existing algorithms and constructing the next AI paradigm. The publication of the Transformer architecture and attention is a work in creating new algorithms. Extending the context window of large language models (LLMs) using YaRN is about improving an existing algorithm. Work on Singular Learning Theory and Active Inference might be considered as constructing the next AI paradigm.

Those seeking to enter the realm of algorithms need to have enough data to train their models and the requisite mathematical prowess. The journey into the realm is not long but also not for the faint-hearted.

Realm of Hardware

The realm of hardware is a strongly fortified one which is extremely hard to enter. The entrance fees are high in terms of capital expense and chip making expertise and capabilities. Unless you come with deep pockets to play the game over a long time, I suggest you do not make the journey.

Its main output are computing infrastructure and chips on which AI models are run. Examples include the GroqChip, AWS Inferentia and, who could forget, Nvidia GPUs.

That being said, although the journey is arduous, great rewards are promised to those who dare and can. The barrier to entry protects those who are already in. Case in point, Nvidia’s dominating position in AI.

Realm of Applications

The realm of applications is like a trade and business hub where money, people and resources flow through freely. It doesn’t take much to enter but it is also potentially very lucrative to those who can gain enough scale and volume.

Like a trade hub that is responsible for combining multiple semi-finished inputs into final products to be delivered to end-users, the realm of applications seek to take existing AI techniques and apply them to solve real-world problems. Examples include Google’s landmark detection and v0.dev for AI-assisted frontend development.

The key to being successful in this realm is to have access to customers who are willing to buy your product, domain expertise to adapt AI models to become truly useful for industry participants and the engineering excellence to deliver products quickly.

Interactions of the three realms

The characteristics of the three realms are summarised below. The thing to note is that the realms are not isolated and are constantly interacting with each other.

Better hardware enables scale for AI applications. While innovative application of AI in real-world problems drives the need for hardware to keep up. For example, the rise of AI fuelled the creation of specialised chips like GroqChip.

The desire to embed AI into end-user applications will drive the need for better algorithms while better algorithms allows developers to be more innovative with the applications they develop. Using the LLM revolution as an example, the rise of LLMs engendered a slew of innovation in the way people do graphic design (Adobe AI), coding (GitHub CoPilot), consume news (Perplexity) and many more.

Algorithms in turn drive demand for AI hardware. Just look at how the demand of Nvidia’s GPU drove the meteoric rise of Nvidia’s stock price. And better AI hardware of course allows faster or more effective training of AI models.

Of course, it doesn’t mean the inhabitants of each land are necessarily friends, while they often are. For example, DeepSeek, an upcoming citizen of the the realm of Algorithms, has negatively affected Nvidia, a Hardware realm royalty, by making LLMs extremely cost and resource efficient.

Realm overlaps and their prominent inhabitants

The three realms not only interact, they also overlap. See the diagram below which shows the prominent citizens of each realm. Note that the list is by no means exhaustive.

I call the space where algorithms and applications overlap the land of Innovative Players. These are inhabitants who take new algorithms and apply them to solve new problems. Like how OpenAI took the world by storm with LLMs. Of course, as the novel applications become mainstream, innovative players move to become rooted residents of the realm of Applications.

The overlap between algorithms and hardware is where Demonstrators live. Often these are hardware vendors seeking to demonstrate the relevance of their products to the new algorithms. An example is how Nvidia optimized Flash Attention on CUDA.

Then you have the space where residents of the realm of Applications are forced into the hardware realm because of the need to deploy AI applications (or help others deploy) at scale. These are Large-scale Operators like AWS and Google.

Lastly, there is a space where all three realms meet and I don’t have a name for. The only resident I can think of is Google Deepmind. I must stress that this is probably due to my limited knowledge rather than an objective truth.

Conclusion

In conclusion, I see the developments in AI broadly fall into the three categories of algorithms, hardware and applications. While there are significant overlaps in how AI industry participants play in each of the three categories, there are unique characteristics of each category that should inform how we play in the AI space.

When I think about changes that are considered “revolutionary” or “phase-changes” from the past, I often find it useful to frame my thinking in terms of interfaces. It is important to notice such changes because when they occur, it often ignites a series of cascading changes that opens up new, not to mention unexpected, ways of working and living and the world is changed beyond recognition at the end of it.

Interfaces…

For example, the mechanization of agriculture was a phase change at the interface of man and his environment. The result was that man can produce much more food with much less effort. What used to be an all-consuming, back-breaking, do-or-die activity for all humankind became an activity which produces so much that in much of the rich world there is an excess of food. Of course the spoils of production is obviously not evenly distributed. But that is not what I am talking about. The release of manpower to other activities drove innovation not only in agriculture but in other areas as well. The ready provision of food allowed man to focus on higher order needs and wants.

Another interface is that of man and his fellow man, the man-man interface. The rise of social networks changed the way people interacted with each other. But it also gave rise to new business models like online advertising and allowed people to participate in the lives of others. Not to mention, along with social networks also came social engineering and other more insidious developments. Safe to say, the post-social network world is also a very different one from before, for those old enough to remember a time when the best one could do to stay in touch is to pick up the phone.

Focusing on the man-machine interface

There are many of such interface changes in history if we care to look but last interface, which is the one I’d like to draw your attention to, is the man-machine interface. The internet represents a change in the man-machine interface. A person can now access information and communicate digitally and remotely via computers. And talking about cascading change, the internet also spawned social networks (a change in the man-man interface).

The iPhone also represents a change in the man-machine interface. Instead of typing on keyboards, people started to interact with machines in a tactile manner. Now, almost all personal electronic devices follow the same interaction model.

In my mind, another sea change in the man-machine interface is currently getting started and LLMs/GenAI is but the opening salvo. For the first time, humans can interact with machines in one of the most natural medium possible, speech. I’m not talking about speech recognition, but machines that can understand the nuances of our language.

Of course, whether the future of this man-machine interface will still be transformer-based LLMs or Large Concept Models (LCMs) or “Agentic AI” or whatever new-fangled names people come up with, I don’t know. I personally doubt it will be what we see today. But what I am watching out for is what this change will mean in the longer term and how it might change the way we live and work. I’m pretty sure that this will engender new ways of working, interacting with each other (man-man interface) and novel business models that we cannot even begin to imagine.

If we consider also how the fields of AR/VR and wearables are advancing, does it mean that we will be able to speak our minds and let machines take on a lot of the more mechanical work? When we think about this, then we realize how the Google Glass was ahead of its time. While it was an innovation in the hardware space, it lacked the power of understanding that LLMs are now beginning to demonstrate and the interaction with the Google Glass was still unnatural. Now that the software side has caught up, we are seeing a resurgence in interest in AR/VR devices like the Meta Orion. I believe this is an iPhone-like moment where in 10-20 years time, the devices will be somehow wearable. The iPhone-like form-factor of mobile devices will go the way of the Nokia 3210.

The combination of increasingly competent (I won’t use the word “intelligent”) machine learning models with robotics also offers an exciting outlook. At present, the recent explosion of interest in LLMs/GenAI is constrained to the virtual world. But make not mistake, the field is constantly pushing the boundaries and trying to realize itself in the physical world. That will be through robots. Increasingly, we will be seeing robots, aided by flexible and powerful models, which can function autonomously in complex environment. On top of that, building on the foundations of telecommunications, machines can be almost omnipresent (think the Internet-of-Things). Brings to mind Skynet. True. But let’s not be too hasty in judging. Remember, it is almost impossible to foresee the changes that will be brought about by such phase changes. Oftentimes, we can only live it and then discuss it after the fact.

What I’ve discussed is just on the capability side of things. What about new business models applying these new capabilities that might be spawned out of this? After all, this is where rubber meets the road, where actual change in our lives really happen.

Already we are seeing the creative industry being disrupted. Copywriters, website designers, digital artists and many more creative vocations are being revolutionized as we speak. The shock that I had 6-9 months ago was that it was an industry that I had not foreseen been disrupted by AI until LLMs/GenAI. I had always thought, like many others, that it was the more mundane and repetitive work that will be disrupted. It was a lesson to me never to be complacent.

Manufacturing and logistics are also not spared as machines become more competent.

And what about the healthcare or home care industry? What used to be a high touch industry that is expensive for consumers may become increasingly accessible via personal assistants that people can talk to, remote assistance by human operators who can take care of multiple clients at once via AR/VR or even autonomous general purpose robots. Of course, we’d have to tune our definition of “touch” and “contact”. But that could lead to greater accessibility of crucial services.

Not all is well and good, no doubt. Man has the unique ability of using tools for the noblest means and his basest desires. Like how the internet drove the rise of online pornography. I’m sure that the recent advances in LLMs/GenAI will lead to nefarious use-cases that we cannot even begin to imagine. An example is the rise of deepfake scams, made almost indistinguishable from the “real” thing in the virtual world.

Let’s be careful about this…

This is why I believe regulation is important here. Just like how we have regulation to protect our children on the internet. Regulation, constructed by an inclusive and mature community, is an avenue via which the world can take a stand against unbridled innovation. There are a lot of nuances and caveats on this point alone and it is a discussion for another time.

Those who know me would know that I am generally an advocate for innovation. But this is different.

We stand at a point where the minimum capability requirements are being satisfied across various technological (and non-tech) domains at the man-machine interface that would unleash a torrent of change. We don’t need to look far for an example of how such a thing has happened before. The whole big data revolution occurred because minimum requirements in data production (the internet) and data processing capabilities (data centers, efficient CPUs/GPUs and big data frameworks like Apache Hadoop and Apache Spark) were satisfied allowing people to process data on an unprecedented scale. I believe stand at that point right now. We need to be prepared. This is what Bezos would call a Type 1-like decision point.

Overlapping changes

As Vaclav Smil described in his book “Growth”, changes don’t happen linearly and often overlap. They also have a fractal nature to them. I.e. you will have growth cycles within growth cycles. It’s just like how the iPhone man-machine interface change is somehow embedded in the internet revolution (another man-machine interface change) and is an aider and abettor to the social network man-man interface change.

So if you’ve read up to this point, please do not think I’m saying that the world is a clean order sequence of changes. They are all interlinked and embedded within each other. Where the world ends up depends on a balance of forces between these changes. This is why such changes always changes the world in a way we cannot foresee.

Along the way, there are still many technical challenges to overcome. There will be many hiccups. It will seem as if a lot of resources are wasted. What is clear that the recent LLM/GenAI developments have shown that it is possible for machines to understand us. Just like how once man landed on the moon, going to outer space and colonizing other planets is no longer a pipe dream. The next phase of development will go beyond what we see now.

Cascading changes

This change in man-machine interface kicked off by LLMs/GenAI will trigger other changes. One of the changes is advancement in computational efficiency, as witnessed by LLM developments from DeepSeek and also novel computing architectures like those from Cortical Labs. Another change, when we think of how general purpose robots now have a viable market, is in battery technology where you can have high energy density and safe batteries to support home use robots. These are what Christensen would call sustaining innovation that would reinforce the LLM man-machine interface change.

There will also other related changes. One that I can think of is the need for authenticity guarantees with the proliferation of machine generated content. This is where distributed ledgers and NFTs might make a comeback. I can imagine a future where human generated content will be signed on a global blockchain. It will also result in human interaction and performances becoming highly valued.

As machines acquire more and more “thinking” abilities and are able to interface with our lives in a very physical or real way, we might also need to start considering them as “persons” with liabilities. This might spur a change in our legal framework which is a core piece of the man-man interface.

Look beyond LLMs, GenAI or Agentic AI

It might seem easy to dismiss LLMs as less then useful, overwhelmed by all the new models and benchmarks coming out on a daily basis or think that agentic AI’s lack of robustness makes it useless. But that is to see things from today’s perspective and to focus on the wrong things, in my opinion.

The way I see it, LLMs, GenAI and Agentic AI are only the opening act to a broader change in the man-machine interface. And that is what we need to pay attention to rather than specific developments in specific technologies.

In general, the greater number of matches that can be discovered between key points of two images, the better we can calculate the transformation matrices, such as the homography matrix or the fundamental matrix, that are needed to understand the perspective changes between the images.

This is why I was excited to try out SuperGlue, an AI model that helps you do feature matching, when I first heard about it. I was curious to find out whether it would do better than the traditional OpenCV methods I was using in my previous experiments.

The code for this experiment is here. It’s not very complicated. Most of the code using SuperGlue is that given on the HuggingFace website.

Overview

In this experiment, I compare the feature matching results from SuperGlue with that from an ORB feature detector + brute force matcher. As a demonstration, I have three simple test cases which I ran through both approaches.

1. A scene of an outdoor building. This is the demo picture from HuggingFace.

2. Two overlapping pictures of a mountain. This is taken from the University of Maryland CMSC426 course. I chose this as the mountain image contained rough textures that might be hard to match uniquely if you only look at local pixel features.

3. Cropped pictures of a pagoda at Mount Fuji. The pagoda has rich features from a human perspective and contains certain background features that might be useful to a human in matching the images. So I was wondering whether a machine can do the same.

For the OpenCV approach, I used sensible parameters that I found effective in my previous image stitching experiment.

Results

The matching results for the demo scene from Hugging Face is shown below. Not only were the number of matches found by SuperGlue is much greater than that by OpenCV, it also seems like the results by SuperGlue were more accurate. For example, the tip of the dome in the image on right was matched to the base of the dome in the image on the left by OpenCV.

For the SuperGlue results, green, yellow and red lines indicate a good, OK and not-so-good match, respectively.

The results for the mountain were similar. I was surprised by the number of matches found by SuperGlue. For a human it might be easy because we take in the entire picture and match points based on its relative position to key contextual features such as the peak of the mountain.

Again, the same for the mount Fuji test case. One thing that I found quite interesting as well is that SuperGlue managed to match the corners of the pagoda roof (indicated by the red squares). ORB was not able to match those features.

Reason behind SuperGlue’s effectiveness

Traditional methods such as ORB uses local features as key point descriptors. One can think of it as embeddings in today’s LLM parlance. These descriptors are often a simple function of the raw pixel values. For example, ORB uses the BRIEF descriptors, albeit with some additional machinery to account for feature rotation. Nonetheless, it is very much a description of the local environment of the keypoint in the image. And the matching of key points are based off these local descriptors.

SuperGlue on the other hand, uses an attention graph neural network. This graph neural network not only connects the key points within each image but also key points from the other image. This allows contextual information from the rest of the image to be taken into account. And I believe this makes all the difference in performance. This is why the corners of the pagoda tower can be detected as it is able to understand how that key point is related to all the other key points in the images.

Final remarks

I was pleasantly surprised by the performance of SuperGlue and I can see how it can help boost the quality of the results of Computer Vision processes that depend on feature matching. I recall having issues calculating the homography matrix accurately due to a lack of matched keypoints, which then resulted in wildly distorted stitched images.

It is clear that using AI models in this way or maybe even multiple AI models in concert to replace previous methods of math/optimization-based calculations can potentially speed up many computationally heavy processes and can be very beneficial to many everyday processes.

This post is part of my journey to learn about SfM. See related posts here.

Overall process

COLMAP uses a technique that is probably state-of-the-art around 6-7 years ago. I will be experimenting with new techniques such as 3D Gaussian Splatting that have come up in the last couple of years in later posts.

The purpose of going through the use of COLMAP is to how the field came to be and also the fundamental concepts involved such as epipolar geometry.

The first step is to identify good image features that would be easily identifiable from different perspectives. This is done usually via the SIFT or ORB algorithms. For an example of how this is used, see my previous post on image stitching.

The features are then matched across images. The OpenCV website has a great series of tutorials on this. The difficulty here is finding good image pairs to match each other’s features with. A series of techniques/optimizations such as using robust image descriptors are applied here. This is because, the better the matching the better the results in the later steps.

The next step is to find out the camera poses of each image. Basically, what rotation and position relative to the object (or other cameras) is the camera at in the picture. This is done by solving for the fundamental matrix, which is kind of like the homography matrix, using the matched features in the previous step and epipolar geometry. In essence, you figure out what transformations are needed to move the features in one image to the matched feature positions in another. This then gives access to fundamental quantities such as the rotation and translation matrices as well as estimated intrinsic camera parameters (such as focal length).

The depth of the matched features are then found via triangulation. This gives a sparse representation of the 3D structure. Why “sparse”? This is because only the matched feature depths are estimated and not for every pixel in the image. The difficulty here is that the depth of the features have to be globally consistent. All the images must agree.

The last step is dense reconstruction. In COLMAP, an algorithm called PatchMatch to find correspondences between different image patches. Then armed with the rotation and translation matrices and the camera intinsic parameters, the depth and normals of each patch is then figured out. For an explanation of the process see this paper. The normals are important as you can’t just know where the surface is if you want to construct a 3D mesh, you also need to know which direction it is facing. Then with both depth and normal a 3D mesh can be constructed using a process like Poisson Surface Reconstruction.

For detailed explanations, I have found Schonberger’s thesis to be a great resource. I dare not expound in details the inner workings of each step as I am not an expert. My understanding here may not be fully accurate / complete. The description above is simply an attempt to outline the overall process and break a big black box into a series of smaller black boxes according to my understanding.

There are also many variations to the above-mentioned process. Computer vision is a vast field.

As an aside, I had naively wanted to code SfM from scratch initially. That though was quickly vanquished once I saw how complicated the SfM pipeline was and realised that it would months of work before I saw any result. Something I might do if I were doing a postgraduate in Computer Vision (CV) but not as a CV hobbyist.

Experiment Steps

1. Install COLMAP

   1. I built COLMAP from source so that it can utilise my GPU

2. Download test images

   1. In this experiment, I used the “South Building” images

3. Follow instructions in “Quickstart”

   1. The auto-reconstruction mode is pretty straightforward. Simply point COLMAP to your project and image folders

   2. There are more features that can be explored if you want fine-grained control or only want to use certain steps. I didn’t explore too much as I want to move on quickly to Gaussian Splatting.

4. Visualise the results using MeshLab

5. [Optional] Visualise using Blender

   1. The .ply files can be readily imported into Blender.

Results

Processing Time

I didn’t keep exact time but the total processing time for the 128 images in the “South building” dataset took many hours. Most of the time was spent in the dense reconstruction stage. This is one of the reasons why I want to move on quickly to Gaussian Splatting, which only requires the results from the sparse reconstruction and promises quick rendering.

Final output

The point cloud from the sparse reconstruction is shown below. It’s really just a collection of points and doesn’t cover every single part of the building.

The red squares are the recovered camera orientations for each of the input images.

After dense reconstruction and Poisson mesh generation, a 3D surface model is generated. This starts to look like something that you would see on Google Maps or a video game in the 2000’s. Note that not all parts of teh building is generated. Parts of the roof, where there isn’t enough information is missing from the generated mesh.

Image requirements

I had initially tried to do the reconstruction with my own images but the generation failed miserably. This is because the object of reconstruction, at least using COLMAP’s SfM pipeline, should be dominant in the images. Just like how the South Building takes up most of the pixel real estate in the images. Otherwise, most of the information will be coming from peripheral objects such as the table the object is on.

One way to circumvent this might be to segment the object out from the background before performing SfM.

Final thoughts

I honestly had no idea what I was getting into when I said I wanted to attempt SfM. If I had, I probably would have picked another topic to explore. Nonetheless, it was refreshing to learn about topics like epipolar geometry and how that could be used to solve for real-world parameters.

In digging deeper about SfM, I’ve also realised that traditional methods, like those demonstrated in OpenCV tutorials, have very high information requirements to begin. For example, the depth estimation tutorial would require that you know the camera’s intrinsic parameters which have to be obtained using a separate calibration procedure.

While the geometric approach can allow for very precise real-world results, I sometimes wonder whether the use of foundation vision models, such as DepthAnything or SegmentAnything can be used in a pipeline to generate something that is useful and quick. That might be something I’ll explore after I’m done with Gaussian Splatting.

Image stitching or photo stitching is the process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image.

Wikipedia

In this post, I’ll be going through my attempt at image stitching using Python and OpenCV. I also used the University of Maryland CMSC246 Computer Vision course and various tutorials (most notably here and here) to guide my learning.

Since my aim is to learn the concepts, I did not use ready-made functions in OpenCV for this work. Neither did I go crazy and started to code everything using Numpy. Instead, I tried to strike a middle ground by using helper functions in OpenCV that would calculate key quantities like key features and homography matrices that would be used to build towards a stitched image.

As usual, I encourage the reader to look through the code repo to follow along. For the impatient reader, the key Jupyter notebook that we’ll be looking through is here.

Image stitching process

The image stitching process involves at least 2 images. For every pair of images, I1 and I2, the following steps are performed to create a single image.

Feature Detection

The first step is to detect notable features in both images. A feature in an image are things like corners and edges. The usefulness of the image feature would also depend on its surrounding pixels. Like the red dots shown in the picture below.

These are points which would be informative when determining whether an object exists in pictures taken from different perspectives.

In this work, I used the ORB feature detector in the match_features function. The ORB feature detector calculates the coordinates of the key features as well as descriptors for each feature. Descriptors are arrays of numbers that encode key characteristics of the feature, much like embeddings in the LLM-space.

Feature Matching

The next step is to match the features in both images to each other. That is to say for example, the corner of the table in picture 1 is the same as that in picture 2.

Take the picture of Lenna below. On the left, I have a cropped and rotated image. On the right I have the full image. The red dots are the features detected individually in the previous step. The green lines are the features that are matched to each other. As can be seen, the eyes are matched to each other and there’s a single point on the nose that is matched in both images.

I used the FLANN feature matcher in this work. There is also a small filtering step to filter away weak matches.

Homography Matrix Calculation

Now we reach the crucial step, figuring how to transform one image to match the other by calculating the homography matrix, which describes how an image transforms from one camera’s perspective to another.

Seen below, the homography matrix H describes how a feature coordinate in I1, (x, y), transforms to the coordinates in the I2, (x’, y’).

Using the findHomography function in OpenCV, we find a homography matrix that best transforms all the matched features in I2 to the same coordinates in I1.

Say we have both images shown below, with the features matched. The mountain images are taken from the CMSC246 course.

The homography matrix will help transform I2 to I1 as shown below. The steps to do so are in the next section.

Something to note here is that the matched key features should be well distributed across the entire image in order for the homography matrix to be well calculated. If the matched key features are concentrated in a small area in the image, then only the area where the features are found will be well transformed and the rest of the image might not.

Image Merge

Final step. Merge I1 and I2 into one image.

First, transform the corners of I2 using the homography matrix with the perspectiveTransform function. This would help figure out the final dimensions the merged image. As can be seen in the example above, the final image will be bigger than the individual images and I2 can end up higher/lower or to the left/right of I1, depending on the matched features. By transforming the corners of I2, the corners will end up with a coordinate relative to I1. See merge_images function.

We then figure out the size of the final image.

The logic to get the final width, w and height, h, is shown below.

I2 is then transformed using the warpPerspective function, where M is the homography matrix.

Note here that there is another matrix Mt that is multiplied to M. This is a translation homography matrix that shifts I2 without rotation to account for the fact that the top left corner of I2 is no longer (0, 0). To see the effect of Mt, look at this notebook.

The 2 images are then added together using a mask matrix.

Overall algorithm

The overall process to merge a set of images is as follows:

1. Given a set of images, set one image as I1 and another as I2.

2. Merge I1 and I2 using the above process to result in I3.

3. Set I3 as I1.

4. Select another image as I2 and repeat steps 1 to 3 till all the images are merged

Final output

I made two examples. One of a mountain panorama (taken from the CMSC246 course).

And another one of a picture of Mount Fuji by Danill K from Unsplash.

Final remarks

I was stumped for a while first by the step involving the calculation of the final dimensions of the merged image and the need to include Mt in the homography matrix before transforming I2. Took a while to visualise the operations and figure out why all my images were turning out gibberish.

The final outputs were also not perfect. You can see the edges of the constituents images and some edges are not as well aligned. The exposure across images is also not consistent. However to get all the details perfect would take a lot more. See the example of the openCV pipeline here.

OpenCV provides a convenient class that can perform image stitching easily. The output can be seen below. The output is much better.

Nonetheless, this exercise has taught me about the key concepts behind image stitching. That’s enough for now.

Time to move on to structure-from-motion!

Large Language Models (LLMs) are changing the way people interact with information in an unprecedented fashion. The most famous example being ChatGPT by OpenAI. The ability to interact with information using natural language, to simply ask and get answers without having to learn an additional interface (keyboard, phone screen, website search bar) is almost magical at first contact.

This is also why big Tech like Google and Microsoft have both tried to capitalise on this by releasing their own LLMs. Countless startups have also sprung up in this Cambrian explosion of LLM-related innovation. Although the technology still has its problems such as hallucination and being extremely power hungry, innovation in AI and engineering is continually coming up with solutions to those challenges. It is quite clear, at least to me, that LLMs, or Generative AI in general, will have a huge impact in the way we live and work in the long run.

However, one thing still bothers me. And that is a lot of the mainstream LLMs (ChatGPT or Gemini) are developed and hosted by big tech on their servers. Which means that everytime we interact with one of the big tech LLMs, we are sending data to the big tech servers. Regardless of what the companies say about protecting user data, I still feel that we should not just leave it to the big tech companies to protect our privacy. Especially as LLMs evolve to become part of a digital personal assistant that is with us 24/7 and knows every thing about our daily lives. Just think about how Google knows the whereabouts of all the people using Google Maps. Think about the power it gives Google.

The open-source community has risen up to this challenge and has released a plethora of very capable LLMs that are free to use. An example is Llama, almost ironically released by Meta. And recently, I came across LMStudio, an application that supports Windows (yuck), Mac and Linux (wooohoo!) and allows people to easily download open source LLMs for use locally, without sending data out of their computers.

So I decided to give it a go and share my experience here.

Sorry for the long intro. Let’s get on with the show.

First impressions

The user interace is very slick. And very familiar to any one who has used ChatGPT before. For those using Ubuntu, the app binary is in .AppImage format. If you find that you can’t run it by double-clicking on the binary, follow the instructions here to install fuse.

Model download

This application even makes model download easy. Simply click on the magnifying glass on the left sidebar and you can select from a range of open-source models from HuggingFace.

For this experiment, I chose the Llama 3.2 1B model that is 8-bit quantized. LMStudio uses llama.cpp as its underlying runtime which means it can run models in the gguf format and it does it really fast. Also, as the LLM community pushes performance limits, smaller models have actually become very capable for many tasks.

I will be running most of the examples below (save for the one where I run it on my 4 year old laptop) using my workstation that comes with an Nvidia RTX A4500 GPU. It automatically detected my GPU and used it. No fuss.

Test 1: Simple conversation

In this test, I simply conversed with the model by asking it a hypothetical question about Romeo and Juliet. In this case, the model is relying on its built-in knowledge. Note here that I am not evaluating the model performance, just the interface.

See a short recording of the interaction below. The speed you see is the actual speed of generation. I did not speed things up.

Here’s an excerpt of the answer given to the question, “Do you think Romeo and Juliet would end up together if their families weren't at war with each other?”

While the story as we know it is set in a time of conflict between the Montagues and Capulets, I think it's possible to imagine a scenario where Romeo and Juliet might still end up together even without the feud between their families.

In fact, there are many adaptations and reinterpretations of Romeo and Juliet that explore alternative scenarios, such as:

1. Romeo and Juliet 2.0: In this version, the families are not at war with each other, but they're more in harmony and respect each other's differences. The story explores how their families would adjust to living together peacefully.

2. A utopian Romeo and Juliet: This interpretation takes place in a futuristic or alternate universe where the feud between the families is non-existent. In this world, Romeo and Juliet could still fall in love without any obstacles.

3. Romeo and Juliet: A historical reimagining: This version might take place in ancient Rome, where the social norms are different from our own. Perhaps the families would have more flexible views on relationships and love, allowing for a peaceful coexistence between them.

Even in these alternate scenarios, I think it's possible to imagine that Romeo and Juliet could still end up together if they chose to be in a relationship rather than being forced into one by societal pressures or family expectations. After all, their love is strong and true, and they're willing to overcome obstacles to be with each other.

…

Ultimately, the story of Romeo and Juliet is timeless, and its themes of love, passion, and loyalty continue to resonate with audiences today. While it's impossible to know for certain what would happen if their families weren't at war, I think it's possible that their love could thrive even without the obstacles they faced in the original version.

Test 2: Ask for up-to-date facts

In this test, I asked for the the tourist attractions in Osaka as well as their websites. The prompt is as follows:

Here’s the response:

Osaka is a vibrant city with a rich cultural heritage, offering many exciting tourist attractions for visitors. Here are some of the most famous ones:

Traditional Attractions:

1. Osaka Castle (http://www.osakacastle.jp/)

   • A 150-year-old castle that played a significant role in Japan's unification and is now a UNESCO World Heritage Site.

2. Dotonbori (http://dotonbori.info)

   • A famous entertainment district known for its neon lights, street food, and unique attractions like the Glico Man sign.

3. Umeda Sky Building (http://www.umedaskybuilding.jp/en/index.php)

   • The tallest building in Osaka, offering great views of the city from its observation deck.

Food and Drink:

1. Tsutenkaku Tower (http://www.tsutenkakutower.jp/)

   • A 150-foot-tall tower with a large statue of the sun goddess Amaterasu, offering panoramic views of Osaka.

2. Dotonbori Riverwalk (http://www.dotenboririverwalk.com/)

   • A scenic walking path along the Dotonbori River, featuring street food stalls and shops selling local specialties like takoyaki and okonomiyaki.

Museums:

1. Osaka Museum of History (http://museum.osaka.jp/eng/index.php)

   • A museum showcasing Osaka's history, from prehistoric times to the present day.

2. National Museum of Art, Osaka (http://www.mhokuboku.go.jp/eng/)

   • A world-class art museum featuring over 6,000 works of art.

Other Attractions:

1. Universal Studios Japan (http://usj-osaka.en.universal-studios.com/en/)

   • An amusement park based on popular movies and TV shows like Harry Potter, Jurassic Park, and The Fast and the Furious.

2. Osaka Aquarium Kaiyukan (http://www.osakaoakari.kaiyukan.jp/en/index.php)

   • A large aquarium featuring a variety of marine life from around the world.

Please note that some attractions may have restricted access due to maintenance or special events, so it's always best to check their websites before visiting. Additionally, Osaka is a bustling city with many other interesting attractions and experiences to explore beyond these famous ones!

This is where the lack of an agentic approach where the LLM can invoke functions to crawl up-to-date information prevents an accurate response. For example, the National Museum of Art, Osaka, has more than 8000 art pieces, not 6000. Also you would find that the URLs are hallucinations.

For thos not familiar with LLMs, all this is part and parcel of using LLMs and techniques like proper prompt engineering and retrieval augmented generation (RAG) have come a long way in overcoming such drawbacks.

Also, I’m using a very small model (by today’s standards), a larger model might work better but would not be a fair comparison when I try to run LMStudio on my old laptop later.

Test 3: Read a document

In this test, I uploaded my CV in PDF (yes, LMStudio supports document uploads) and asked the LLM the following question,

If the document can fit into the context window, LMStudio will parse it and place it in the LLM context.

Pretty cool…

Test 4: RAG. Yes, RAG.

LMStudio also supports Retrieval-augmented Generation (RAG). This means that you can upload large amounts of text that don’t fit into an LLM’s context window and have the LLM look up and summarise information from all the text data you upload.

To test this, I uploaded 13 of Shakespeare’s plays that I downloaded from the Gutenberg Project.

The thing is, LMStudio only supports uploading 5 documents at a time. So I had to batch my uploads. But once I’m done, I can ask questions across the stories I’ve uploaded.

It even gives me the citations.

Final test: Running it on my 4 year old laptop

So now I’m running it on my 4 year old laptop (Lenovo X13 Gen1, Ryzen 7, 16GB RAM). This is to test out how it feels like on a developer laptop rather than a workstation.

As you can see in the screen recording below, the generation is a lot slower. 8.24 tokens/sec versus 200+ tokens/sec on my workstation. But the over feel is still OK, given the light-weight Llama3 1b model. Not too bad for a 4 year old laptop.

Of course, it might struggle with larger models. But with lighter models getting more capable each day, I’m staying hopeful.

Other functionalities

An interesting thing I found is that LMStudio also supports serving out LLMs via an OpenAI-like REST API. I didn’t fully explore this. But it’s interesting.

Final thoughts

The entire experience of using LMStudio was extremely pleasant. The user interface was slick. Model download was seamless. Runtime was fast thanks to llama.cpp.

The setup was also painfree. I didn’t have to compile anything. I didn’t have to setup llama.cpp on my own, host a REST API and then code up a half-assed interface using Gradio (not that Gradio itself is half-assed, just my UI skills are not great). Everything just worked.

More importantly, none of the data left my workstation.

True, the fact that I can’t upload hundreds of documents can be a pain. And the lack of agentic approach can limit the use cases to summarisation, re-writing or ideation. But that’s OK. I mostly use LLMs at this moment for those use cases anyway. Maybe LLM agents that can query websites can be made available on LMStudio. Most people use LLMs in the chat format anyway.

On top of that, I have the whole open-source ecosystem to draw on. So for example, I can use a coding specific LLM to answer my coding questions. Maybe there will be a medical LLM to which I can direct my healthcare questions or ask to summarise my health check reports.

All-in-all, I’m very impressed with LMStudio and will continue to incorporate it into my daily work.

Such an effect can be achieved in Python using the context manager. In fact, many objects in Python already implement this. An example is the file handle that you get when you open a file. This is why you can write with open(‘filename.txt’, ‘r’) as f, and be sure that the file will be closed when you exit the with context.

More importantly, there will be times when you want to ensure certain custom finalizer action that is pertinent to your application is run for a custom resource or object. At that time, the bracket patterns comes in very handy.

In this article, I will go through an example of implementing the bracket pattern using the context manager. The code is here.

A custom resource object

Suppose, you have a custom resource object as follows:

class Resource:
    def __init__(self, with_error = False):

        if with_error:

            raise ResourceAcquisitionError()
        
        return

    def doSomething(self):

        logging.info("Resource doing something!")
        logging.info("Performing some actions...")

    def doSomethingWithError(self):

        logging.error("Resource doing something that will throw an errors!")

        raise ResourceActionError()
    
    def release(self):

        logging.info("Releasing resource now.")

Here the resource object is a little contrived for demo purposes. During its initiation, if with_error is True, then an exception is raised to simulate a failure in acquiring the resource, such as FileNotFoundException.

There are then two methods: doSomething and doSomethingWithError, to simulate interacting with the resource and in the case of the latter, something goes wrong.

Lastly, there is a method to release the resource.

The bracket pattern code

The implementation of the bracket pattern uses the yield keyword. As such, control of the main program is passed from one function to another. To make it easier to follow, I’ll break the code down into parts.

First is the main program interacting with the resource shown below.

There are three main parts labelled “1” for the acquisition of the resource, “2” for actions to perform when the acquisition fails and “3” for actually using the resource.

Next is the resource manager which is decorated with the context manager from contextlib. This implements the bracket pattern. There are again 3 main parts labelled “4” to return the resource, “5” to handle any errors that arise from the use of the resource and “6” for performing the finalizing actions when everything is done.

Demonstrating the bracket pattern

The main code in the repo demonstrates the use of the bracket pattern three scenarios: the “happy path”, the case where something goes wrong when interacting with the resource and failure in resource acquistion.

In the “happy path” where everything goes smoothly. The resource gets acquired in “1”. Then using the resource manager the resource is yielded in “4”. This yields control from the resourceManager back to program. Then we interact with the resource in “3” before we exit the resourceManager context which then performs the finalizer actions in “6”.

So “1” → “4” → “3” → “6”.

The output is seen below.

In the case where something goes wrong while interacting with the resource, we again acquire the resource in “1”, enter the resourceManager which yields the resource in “4” but this time, an exception is raised in “3”. At this point, control goes back to the resourceManager and the error is handled in “5”. This is where you as the developer can perform custom actions like logging detailed information of the error. And as usual, we arrive at the finalizer actions in “6”.

The process is “1” → “4” → “3” → “5” → “6”.

The output is shown below.

Lastly, in the case where the resource fails to be acquired at all, we go from “1” to “2” (where we can implement other behaviours like retries or logging) and finally ending the program.

Conclusion

I hope that in this short post, you can see how with the use of try… except and context managers, we can ensure that resources and the exceptions raised in interacting with them can be safely handled.

This safety may not seem much in prototypes but will definitely be worth more than its weight in gold in production systems.

Separately, I was inspired by a talk by Fabio Labella that talked about using fs2, a library for coding “functional, effectful, concurrent streams in Scala”. This gave me an idea to try and apply an asynchronous, stream-based way of coding to the AI model training process.

In this article, I’ll walk you through my experiment on using fs2 and Cats Effect to string together the various steps in a basic AI model training. I encourage you to clone the repo to follow along.

A little on fs2 and Cats Effect

Cats effect and fs2 are two key libraries in the typelevel ecosystem that enables functional programming in Scala.

Cats effect implements an asynchronous runtime in Scala via the IO object as well as typeclasses such as Ref, which we will be using later to maintain shared states.

fs2 is a stream library that is widely used to manage complicated data streams. It relies on a key benefits that often comes out of functional programming: compositionality, which allows a process to be broken down into parts and seamlessly combined. It also implements primitives such as topics, queues and channels for more complex ways of combining data streams.

Side note: Personally, fs2 is one of the most amazing libraries I’ve come across. It’s super powerful and expressive. On top of that, the pull-based mechanism for streaming data allows constant-time processing, keeping compute resource usage manageable.

Setup

My aim here is to demonstrate the use of functional streams for deep learning with GPU on Scala. So I’ll be using an oldie-but-goodie, the MNIST digits classification, as a use case.

In this way, I can focus a little bit more on Storch and fs2.

Storch also conveniently provides an API to the MNIST dataset (like how PyTorch has an API for commonly used ML datasets) shown below (click here for link to code).

As mentioned, Storch implements the PyTorch API. The neural network used is a simple multilayer perceptron seen below. The syntax should be familiar to any PyTorch user.

Training step

fs2 works on the concept of Streams. For those who are not familiar, think of it as a river of data (or “1”s and “0”s) which keeps following until the program ends or the data is exhausted. Importantly, fs2 streams are pull-based. This means that data isn’t continuously fed from the input end but pulled in from the exit end of the stream.

The nice thing about pull-based streams is that data is only delivered and computed just in time. Only when the consumer end requires the data will the data be delivered. Compare this to a push-based stream where data is delivered until either some queue is full or the consumer can no longer handle more computation, a pull-based stream means that the resource usage can be kept manageable.

The training step of updating the model parameters by showing the model various batches of data can be seen below. The training data is delivered in batches via the training data stream on the left. Each batch is then put through the model update step which updates the model weights followed by the storing of the training statistics such as training cost. The AI model here is initiated as a shared object so that each batch of data updates the same model. The same goes for the training statistics. This proceeds until all the data in the training data stream is exhausted.

The code for the above is here. Note that the return type of Pipe[IO, Sample, Unit]. An fs2 pipe is a function that takes a stream of one type and transforms it into a stream of another type. In our case, Pipe[IO, Sample, Unit] is the same as Stream[IO, Sample] => Stream[IO, Unit]. It’s job is to perform the key model training steps for each batch of data. I won’t spend too much time dwelling on the syntax of fs2. Interested readers should read the documentation.

Taking a closer look, trainPipe takes in a sampleStream as input, this is the training data stream. The data is batched using sliding and stackXY reorganises each chunk of data into a tuple of inputs and targets. The key operations are in the evalMap section, where each batch of data is then mapped to the asynchronous function that updates the model and training statistics.

Note that the model (model) and training statistics (stats) such as are passed to trainPipe as Ref. Ref is a typeclass in Cats Effect that allows for shared states. This allows for the training and validation streams to refer to the same data objects.

Validation step

The validation step is coded similarly to the training step where there is a validation data stream fed to a pipe. See below (code here).

The key thing to note here is the fold method which accumulates all the results from the validation batches into a final validation cost and accuracy of the entire validation data stream. Since fs2 is built on Cats and Cats Effect, typeclasses like Applicative, Monad and Foldable are available. This enables semantics like fold and flatMap which opens up the expressiveness of the language.

Putting it all together

Now that we have each step of the training process in place, we need to put them together as follows.

After each training and validation epoch, we need to repeat the process for a number of times. Now this is where fs2 with its functional programming roots really shine. One of the key benefits of functional programming is compositionality. I.e. we can create complex patterns by composing functions. This can be seen in the main run function here.

Taking a closer look, the loop is implemented simply as an append and repeat.

I can safely do this because of the semantics made available by fs2. I didn’t have to do any boilerplate to manipulate types or additional control flow.

One last thing… IOs…

You would have notices that the data type IO throughout this demo. IOs are a basic higher order data type from Cats Effect that allows for asynchronous computing. In a nutshell, what I’m doing when I’m instantiating the functions in my program is I’m collecting a series of instructions to be run at a later time. Kind of how Apache Spark collects the data processing instructions into a Directed Acyclic Graph (DAG) to be lazily evaluated later.

For those coming from an interpreted language like Python, myself included, this can take a little getting used to.

Final Thoughts

Out-of-memory datasets

In this demo, I had the MNIST dataset loaded in memory since it is small enough. But you can read data from files or databases in a stream-like fashion, keeping memory usage capped.

Complex training patterns with fs2 primitives

More complex model training patterns can also be enabled by fs2’s primitives. For example, training a model from various disparate data sources can be enabled with Queue. Or training multiple models concurrently from a single data source can be done via publish-subscribe with Topics.

Scala ML Ecosystem

While Scala has found widespread use on the data engineering scene (think Apache Spark and Apache Flink), libraries for numerical computing is generally lacking compared to Python. For example, Scala’s numpy equivalents like Breeze are not as actively maintained. Availability of graphing libraries like matplotlib for Python is also not very good. If I needed to do machine learning, Apache Spark is likely my best bet, but I don’t quite want to run a Spark clusters (even a local one) just to train a simple model.

Storch is a step in the right direction by introducing a Scala wrapper over LibTorch. It’s still not as fully featured as PyTorch but the support for tensors and autogradients offer an opportunity for numerical computation on both CPUs and GPUs.

Other nice things in the Scala ecosystem include:

• Support of Scala kernel in Jupyter notebooks

• Polynote, a Scala notebook which allows sharing data objects to Python for plotting

One thing that I thought would be a good idea is a Scala API for the Polars dataframe library.

Metabase is what I would describe as a Business Intelligence (BI) and dashboard building tool that seeks to democratise the use and exploration of data. It is an alternative to well-known tools like PowerBI, Tableau and Superset. I came across it sometime back and decided to take it out for a quick spin, just because…

Getting things up and running

Metabase was pretty simple to get up and running in an local setup using Docker compose. This is provided in their setup instructions. Just use the Docker compose YAML below and you’ll be up and running in less than a minute (depending on your network speed of course). Please note that this is not suited for production/enterprise setup.

version: "3.9"
services:
metabase:
image: metabase/metabase:latest
container_name: metabase
hostname: metabase
volumes:
- /dev/urandom:/dev/random:ro
ports:
- 3000:3000
environment:
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabaseappdb
MB_DB_PORT: 5432
MB_DB_USER: metabase
MB_DB_PASS: mysecretpassword
MB_DB_HOST: postgres
networks:
- metanet1
healthcheck:
test: curl --fail -I http://localhost:3000/api/health || exit 1
interval: 15s
timeout: 5s
retries: 5
postgres:
image: postgres:latest
container_name: postgres
hostname: postgres
environment:
POSTGRES_USER: metabase
POSTGRES_DB: metabaseappdb
POSTGRES_PASSWORD: mysecretpassword
networks:
- metanet1
networks:
metanet1:
driver: bridge 

The first thing once you navigate to localhost:3000, you’ll be greeted with a user creation screen. Just enter the details as you see fit, hit “Take me to Metabase” and you’re in!

Connecting a database

I decided I should try and connect a BiqQuery public dataset as my test data to test out how easy it is to connect to remote databases. For those of you who don’t know, Google BiqQuery hosts a series of public datasets for people to use.

In this experiment, I chose to use the Chicago Crime dataset. I started by first copying the dataset into my own Google Cloud project. This is pretty simple. Navigate to the dataset and click “Copy” on the top right and enter the required details (you can even set the dataset to expire after a certain number days so that you don’t keep it in your project indefinitely).

Next to allow metabase to reach the dataset in my Google Cloud project, I created a service account with the requisite privileges. See instructions here.

After which, I downloaded the JSON key for the service account and added it to Metabase in the “Add Database” page. As can be seen below, many other database connectors are available.

Once that was done, I was able to preview the data immediately. Tada!

Using Metabase

The overall experience was pretty much similar to that of Superset. Once you have your dataset ready and connected, you are able to filter and summarise via a drag-and-drop UI (seen below) or via SQL statements.

The thing I had to get used to was the idea of Questions and Models. Questions are basically your SQL statements. Each question is coupled with a visualisation, of which there are many to choose from. Models are like your database views. They too can have a visualisation coupled with it. The key difference between a question and a model, as far as I can tell, is that models allow other questions to be built on top of it, which makes sense.

There’s also the idea of Collections to group your questions and models. I guess, it’s to support different use cases on the same datasets.

Once you have all the visualisations read, you can put them all in a dashboard like below.

There were also other features like updating the data sync frequency and allowing people to subscribe to dashboard changes that I didn’t try.

Final thoughts

On the whole, I thought the user experience was rather pleasant and somewhat similar to what you would get in Superset or PowerBI. There was not much to complain about. The data onboarding was uneventful. The data refresh was pretty snappy (of course, it depends on the network and infra conditions you have. BigQuery’s pretty quick for most things). The organisation of the entire app was rather logical once you understand its concepts of questions and models.

I can also see Metabase’s utility to non-technical users with some knowledge of data modelling and manipulation. You would still need to understand group by and all that but you don’t need to code.

Although like any other BI tool out there, once the team gets big and you start to have many datasets, effort needs to be put in to maintain organisation and ensure that performance does not degrade. Even in the simple experiment I did here, BigQuery was queried several times with each question formulation. I can only imagine what that would do to on-premise databases with multiple teams trying to get their questions answered.

As to how such a tool will fit into the “ChatGPT-everything” age, I still feel that having drag-and-drop filters help and can be more efficient than simply using Large Language Models (LLMs) to go from question to answer. This is because it can be hard to express things in our heads to an LLM, whereas having the right buttons to click allows us to quickly narrow down the search field. That being said, LLMs can perhaps play a role in correcting our SQL statements or helping us discover other datasets within the organisation.

Lastly, as I was using Metabase to generate the visualisations, I can’t help but feel a little frustrated at the visualisation controls. I felt that I could have done things a lot faster in Python. Perhaps this is due to my training as a data scientist. Perhaps I’m not the intended audience for Metabase. But I thought it’d be great if there’s a way for me to run Python-based queries and create graphs using matplotlib or plotly. In this way, I can do things quickly and yet use the other functions of Metabase like dashboards or notifications.

So, here’s my attempt at coding my own Sudoku generator and solver.

The code repository is here. As usual, I recommend that the reader download the code and follow along. Links to the exact lines of code will be provided in the post.

Nomenclature

A Sudoku game is a 9x9 grid. Each grid is a 3x3 grid of blocks, which are themselves 3x3 grids of cells. Each cell can be empty or filled with the numbers “1” to “9”. In the code you will see later, a cell with a value of “0” means it is empty. The aim of the game is to fill the entire grid with the numbers “1” to “9” while ensuring that the numbers do not repeat in each row, column and block simultaneously.

A Sudoku grid with empty cells waiting to be filled in is a Sudoku puzzle. The pre-filled numbers in a Sudoku puzzle are called clues.

Some design choices in the code

Let me first highlight some design choices I made so that it might be easier to understand why I coded the Sudoku solver and generator the way I did. The first design choice I made was to use Vector as the data structure to store the Sudoku grid. In Scala, Vector is an immutable data structure. This means that you can’t change it, you can only make a copy of it with an updated/new element. This is because, as you will see later, I will be storing the grid at different stages of the solution. Using an immutable data structure, gives me the piece of mind that I won’t have any weird behaviours that affect earlier states while editing later states.

Secondly, I approached the problem with Functional Programming. As a result, you will see that I will often return a new instance of an object after some action taken, instead of returning a modified object. This is also related to the immutability of Scala vectors where each update results in a new copy.

Grid Trajectories

Another thing that you will see in the code is what I call grid trajectories. (This is my own definition, not a conventionally known term.) Basically, the act of playing Sudoku is to move the grid from one state to another via the action of filling up a cell.

A grid trajectory is the history of all the states that the Sudoku grid has been through together with the actions that resulted in that state. The definition in code is here, as well as below. A CellAction, contains the row and the column of the cell being edited as well as the number that was used to fill the cell.

This will become useful to implement memoization in the backtracking algorithm which I will describe presently.

Backtracking Algorithm

The backtracking algorithm is a well known algorithm that is used when exploring a solution space where there are many valid solutions in a systematic way. Basically, you keep trying taking actions that are valid, transitioning the Sudoku grid from one state to another. If you reach an invalid state, you undo your previous action, revert the Sudoku grid into its previous state and select from the remaining valid actions from the previous state. You keep doing this until at least one solution is found.

To do this in code, usually you would use a recursive function that tracks the history of the states. An example can be seen in my implementation of the solver function. Note that the main recursive function takes a GridTrajectory that enables us to revert to previous states. See also the backTrack function that takes in a GridTrajectory and returns a GridTrajectory.

Taking a closer look at how the backtracking algorithm progresses below, we start from “State 2” and say we select a cell action that progresses to “State 3”. After “State 3”, let’s say there is only one state left to which we can go. However, that state is an invalid state. In Sudoku, an invalid state is one in which there is at least one cell where there are no valid options. As a result, we can only move back to “State 3” (see “Backtrack 1”).

Since there are no more further states from “State 3”, we can only backtrack further to “State 2” (see “Backtrack 2”). The only state left from “State 2” is also invalid, which means we end up in “State 2” again. Having no more options, we again backtrack, this time to “State 1” (see “Backtrack 5”).

We repeat this process until we are able to find a solution or we exhaust all possible actions.

Sudoku Data Objects

There are two main data objects in the code that will enable the interactions with the Sudoku grid.

Cell

The first data object is the Cell. It contains information about the row and column of the cell (0-indexed), the number it contains as well as the options that are available.

The available options are derived based on the constraints that for the number “1” to “9” there should no number occurring more than once in each row, column and block. This is found using the isValidCellValue function.

def isValidCellValue(row: Int, col: Int)(trialValue: Int): Boolean = {

            (getRow(row).filter(c => c.col != col).forall(c => c.number != trialValue)) &&
                (getCol(col).filter(c => c.row != row).forall(c => c.number != trialValue)) &&
                (getBlock(row, col).filter(c => (c.row != row) && (c.col != col)).forall(c => c.number != trialValue))

        }

Grid

The grid data object contains all the methods used to interact with the Sudoku grid, such as getting/setting cells or checking whether a number is valid for a cell.

Solver

The Solver implements the backtracking algorithm using a recursive function. The main logic is here and shown below as well. As described before, if there are cells with no options (i.e. an invalid state) the Sudoku grid will be backtracked to the previous state otherwise, a new number is added to one of the empty cells.

A note on the backtracking logic. The first state in GridTrajectory is the starting state of the Sudoku puzzle and the corresponding cell action is None. If so, an empty GridTrajectory is returned. Importantly, if there is a state to backtrack to, then not only is the number of the cell that is previously changed reverted back to zero, the number that was set is also removed from the options. For example, if setting a cell to the number “3” leads to an invalid state, then backtracking means that that cell is set back to “0” and importantly “3” is removed from the options available to the cell.

This is ensure that the option that leads to an invalid state will not be selected later.

Lastly, the solver has 3 solver modes.

SolveOne is the mode that seeks to find the first solution available. SolveAtLeastOne is the mode that will continue to find more solution until more than one solution is found or the maximum iterations allowed is reached. This useful as can be seen later in the Generator section for testing whether there is a unique solution to a particular Sudoku puzzle. SolveAll as the name suggests will keep looking for as many solutions as maxiumum iterations will allow or when all possible moves are exhausted. The return logic for each of these states are shown here.

Generator

Generating a full grid

To create a Sudoku puzzle, we first start with a full grid where all the numbers are filled. This is generated using the createFullGrid function here. It fills the cells one by one in sequential order (instead of randomly selecting a cell each time). The backtracking algorithm is also used here to allow the generation to continue should an invalid state is reached.

The cells are filled in sequentially in row-major order because it allows the cell to be filled to take advantage of the information from numbers in the same row/column/block. If a random cell is selected each time, then the selected cell might be one where there are no other filled cells in the same row/column/block and no information to inform the choice of number to fill.

Generating a Sudoku puzzle

An important thing about a Sudoku puzzle is that it should have a unique solution. However this is not guaranteed if you simply remove a bunch cells from a full grid at random.

To ensure that the puzzle has a unique solution, the createPuzzle function removes cells one-by-one. With each removal, the solver is used to solve the puzzle in SolveAtLeastOne mode. This ensures that the puzzle still has a unique solution with the removal of that cell (at least up to the maximum iterations limit). You keep repeating this remove-and-solve step until you reach the minimum number of clues desired (see the minClues parameter) or if there is no other way to remove more cells.

Note that it is not guaranteed that the final puzzle generated will only have minClues clues. Also it is not always that the generation will succeed there is a certain level of randomness involved as the way the cells are removed may not result in a uniquely solvable puzzle. For example, although I had set minClues to 17, the final puzzle that resulted was one with 25 clues.

Results

The results of a sample generation of the Sudoku puzzle and its solution is shown below.

Solving puzzle with 25 clues...
Correct answer found.
Puzzle
============
000 | 305 | 700
000 | 000 | 058
080 | 060 | 003
---------------
060 | 053 | 000
500 | 410 | 000
004 | 800 | 000
---------------
100 | 000 | 279
006 | 000 | 100
800 | 009 | 006
---------------
Solution
============
491 | 385 | 762
637 | 921 | 458
285 | 764 | 913
---------------
769 | 253 | 841
528 | 416 | 397
314 | 897 | 625
---------------
153 | 648 | 279
976 | 532 | 184
842 | 179 | 536
---------------

Conclusion

It was fun learning about and implementing the backtracking algorithm as well as satisfying to finally peek behind the scenes of the game I’ve played so often. I also enjoyed being able to practice Functional Programming.

One of those “rough edges” is that LLMs being large, as its name suggests, are mostly being run by big tech companies such as OpenAI and Google on their massive server farms and exposed to the public via APIs. As a result, you would have to continuously send your queries/data to these companies in order to interact with their LLMs. This means that 1) there is a potential loss of data privacy and 2) a lot of network costs are incurred.

To be fair, the LLMs (or rather multimodal foundation models) are large for a reason and the big tech models often push the frontiers of what is possible. But what if you had a smaller use case? What if you wanted to incorporate it into a web portal? What if you were desperately concerned about your data privacy?

Well, two things work in your favour. Firstly, LLMs are getting increasing capable with new training techniques as well as over-training. As a result, a 2 billion parameter model like Gemma2 2-b is pretty capable already. Secondly, there is a framework called WebLLM that uses WebGPU to tap on your local GPU’s power to run LLMs locally!

In this post, I’ll be trying to enable it for myself on my desktop. Note that you do need a machine with a GPU. Tried it on my 4-5 year old laptop and didn’t work. I believe it should work on Macs with ARM-based chips, but I can’t be sure. If you’re still interested, read on…

Pre-requisites

You will need Chrome for this if you are on Ubuntu. Firefox is not supported yet. See list of supported browsers here.

Step 1: Enable webgpu and vulkan support

1. Go to chrome://flags

2. Search for WebGPU and enable both features

3. Search for Vulkan and just enable everything

Took me a bit of searching to figure these steps out. Somehow people just assume that WebGPU should work out of the box, which didn’t for me. Kept running into errors of the browser not being able to acquire access to the GPU even though my machine is WebGPU-capable.

Step 2: Clone WebLLM demos

Clone WebLLM git repo so that we can quickly test using the examples.

git clone https://github.com/mlc-ai/web-llm.git

Step 3: Run a simple chat example

Here we run the simplest chat example in web-llm/examples/simple-chat-js using Python.

cd <path_to_webllm_repo>/examples/simple-chat-js && python3 -m http.server 8080

Navigate to localhost:8080 in your Chrome browser.

Step 4: Start playing!

You will see a screen that looks like the one below. The first thing you do is you choose a model you want to load (there are many) and click “Download”.

This will download the model and load it onto your GPU. The first run takes a while to get started. Subsequent runs will use the cached model parameters.

As you can see, the model is indeed loaded into my GPU.

Final Thoughts

I think it is a step in the right direction to enable running LLMs on browser for privacy preservation purposes. It also makes web application development much more light-weight as we don’t have to keep querying OpenAI/Google’s APIs incessantly. Instead we only use the big-tech models when we need some heavy multimodal firepower.

It is also nice that the WebLLM library can be CDN delivered. You just need to do import * as webllm from "https://esm.run/@mlc-ai/web-llm";, as mentioned on their GitHub README.

At the point of writing, the WebLLM library supports mainly chat completions, which might be useful for chat or summarisation use-cases. I would have love that embeddings are supported so that we can also think about RAG-based use-cases, which was why I was looking at this in the first place. I was thinking that it’d be great if I could have a simple backend with my document embeddings that can then be matched up with the query embeddings directly coming from the frontend. Apparently, this feature is in the works. So fingers crossed!

WebLLM is built on the very capable MLC-LLM framework which is a cross-platform LLM inference engine. I’m hoping to dig a little deeper into MLC-LLM.

Lastly, WebLLM still assumes that you have a GPU on your machine somehow. While this might become increasingly true in the near future (even our mobile phones have GPUs now), it is not completely true. Just like how my laptop would not have been able to utilise WebLLM. This means that developers will still have to find a way to cope with users without a GPU (maybe falling back to using OpenAI) or there might come a day where the AI paradigm shifts to enable really lightweight and capable models that can run fast enough of CPUs.

The Bellman’s Principle of Optimality is an important idea in the study of sequential decision making processes. It has found its way into many areas from software algorithms to inventory management in the form of dynamic programming and reinforcement learning.

In this post, I would like to describe the general idea behind the Bellman’s Principle of Optimality. I won’t go deeply into specific application of the principle but really try to take a look at its mathematical form and see what it means. I believe it will be easier to apply the principle once we can appreciate the general idea.

Problem Setup

Consider a decision problem over T time steps. At any given time t, our environment is described by a state x_t. This “state” is simply the collection of “things” that tells us about the environment we are in. It could be temperature in the case of an oven. Or it could be the position, velocity and acceleration in the case of a car.

At each time step t, we have to make a decision d_t given the state x_t that would lead to an action a_t.

We can think of d_t like a decision rule that tells us what to do when given situation described by x_t. For example, when the traffic light turns red, hit the brakes.

And as a consequence of a_t, our environment changes from x_t to x_t+1. Mathematically we can describe such a change using a transition function, Φ, which describes how the state changes given an action.

With our action, there also comes a cost, c_t(x_t, a_t).

As an example, when we step on the brakes while driving (action), the car slows down (state change) and we use more fuel during our journey (cost).

The overall process is shown below.

Thus we can express the total cost of all our actions over T time steps as a sum of all the costs incurred at each time step, C, as follows,

Taking a closer look at the total cost expression, we can write it in two ways. First, in the forward time direction, where it is the sum of the cost at time 0 plus the total cost from t=1 to t=T. This is how we experience things, starting at the start time and progressing forwards.

The second way is backwards in time, where we have the total cost up to the second last time step plus the cost at the last time. This is more like the planning way of thinking where we start with the end in mind and work our way back.

In both cases, notice the term where the next state is dependent on the previous state. This introduces the dependency of the costs incurred from the actions taken over the course of the time horizon.

For example, taking a closer look at the second term for the forward direction, the cost from t=1 to t=T is a result of x₁ which in turn is the result of the action taken at t=0, a₀ at state x₀.

Lastly, to introduce one last piece of math machinery, the collection of all the decision rules is called the policy, π.

Some call it a strategy. It is a way of thinking that informs us what decisions to take at each point in time under any given situation.

Now we are ready to start looking at the Bellman’s Principle of Optimality.

Bellman’s Principle of Optimality

Let me first quote the Bellman’s Principle of Optimality so that we can come back to it later.

Principle of Optimality: An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. [Bellman1957]

Our overall objective is to minimise the total cost over our time horizon T and is expressed below

Now suppose we are able to arrive at an optimal policy, π*, which gives us the best decisions rules that we can have such that the total cost is at its minimum. In other words,

Expanding the total cost expression, we have

The above expression means that at t=0, we take an action that is optimal and leads us to state x₁ (first term) and follow the optimal policy thereafter (second term), we will arrive at a total cost that is the smallest it can be.

Now consider if we didn’t take the first action, a₀ according to the optimal policy, we would have

Of course, the total cost in this case would be greater or equal to the case where all actions were taken according to the optimal policy. But Bellman’s Principle of Optimality states that, by following the optimal policy after t=0, we would still arrive at the minimum cost given the initial state and action.

Two things to note here. Firstly, a policy is not a sequence of states. We do not specify the best path for the system to traverse through. Instead it is a sequence of decision rules (using our current nomenclature). That is why even if we didn’t make the optimal decision at t=0, our remaining policy decision rules will still be able to guide us to the best possible outcome given that we made a “mistake” initially by not following the optimal policy. Again, this does not mean that there is no cost to our mistake. The actual outcome would likely be worse than the optimal one just that it is the best that could have been achieved given the circumstances.

Secondly, the Bellman’s Principle of Optimality highlights the importance of knowing how to make good decisions and if we know how to make good decisions, then we only need to worry about the next step. This is because make the first good decision would set us on the path to get to a good state where we can make the next good decision to get us to the next good state and so on and so on.

The keen-eyed reader would notice at this point that while the Principle of Optimality is all well and good, it is not clear as to how one would arrive at an optimal policy. For that, I can only refer the reader to full length treatments by Sutton [Sutton2018] and Bertsekas [Bertsekas2020].

Problem Variations and Additional Considerations

In this article, I have used a simplified context to present the Principle of Optimality. There are variations and additional considerations that makes the Principle widely applicable in many situations. Here are some of them.

Rewards instead of Costs

In some cases, rewards are considered instead of costs. This is a simple modification to our treatment of the Principle here. Rewards are simply negative costs or vice versa. So in dealing with rewards we maximise instead of minimise. In books where rewards are used, you would see the total cost function being called the value function. See [Sutton2018].

Continuous Actions

If instead of discrete actions like start or stop, you have actions that can take continuous values such as how much to steer, then you would also approximate the decision rules as functions. In fact, you can approximate the entire policy as a policy function. See [Sutton2018].

Continuous Time

We have dealt with discrete time steps in this article. The time steps could be 1 second each, or 1 hour, 1 day, 1 month, doesn’t matter. But the shorter the time interval represented by each step the closer we come to considering things in continuous time where we would have to use the mathematical apparatus of stochastic differential equations (SDEs). Interested readers might consider look at [Hassler2015] and [Oksendal2014].

Stochasticity

In this article, I have assumed that the transition from one state to another is deterministic. In other words, if a particular action is taken at a particular state, it will always lead to a known next state. This is not true in general. There is unpredictability present and the next state will be one of many possible states each with its own probability of occurrence.

To that end, the objective function stated above will have to be augmented with an expectatoin over all possible states. In other words, we will have to weight the costs of each possible next state according to their likelihood of occurrence.

This means that the optimal policy would give you the best outcome on average. But not that on any particular “run” or “episode” you are not guaranteed to always achieve the lowest cost or highest rewards.

I won’t dwell too much on this. Interested readers can refer to [Sutton2018] and [Bertsekas2020].

Books by R. Howard [Howard2007a, Howard2007b] also offer very good insights on how to treat non-deterministic state transitions in Markovian systems.

Conclusion

Bellman’s Principle of Optimality is an interesting concept that I picked up when I was into reinforcement learning and dynamical systems. I have found it applicable not just in the technical arena but also in life when thinking about decision making.

References

[Sutton2018] R.Sutton, A. Barto, 2018, Reinforcement Learning: An Introduction

[Bertsekas2020] D. Bersekas, 2020, Dynamic Programming and Optimal Control: Volume 1, 4th edition

[Bellman1957] R. Bellman, 1957, Dynamic Programming

[Hassler2015] U. Hassler 2015, Stochastic Processes and Calculus: An elementary introduction with applications

[Oksendal2014] B. Oksendal, 2014, Stochastic Differential Equations

[Howard2007a] R. Howard, 2007, Dynamic Probabilistic Systems: Markov Models (Volume 1)

[Howard2007b] R. Howard, 2007, Dynamic Probabilistic Systems: Semi-Markov and Decision Processes (Volume 2)