Seeing through Abstractions

I've been on my current learning journey for a while now. Recently, I've become cognisant of moments where ideas have clicked in my head, allowing further introspection. These insights have leaked into the rest of my thoughts and transformed my perception of reality.

What do I mean by this?

Well, I used to think programming was about controlling computers. However, it's more than that.

The Chair and the Class

OOP is one of those things that seems obvious in retrospect but opaque when first introduced. I, like many people, struggled with it and contended with it mentally, which took a lot of time. However, I think I remember the pivotal moment when it finally made sense to me.

I often have problems whirring away in my subconscious somewhere, this was one of those moments. I distinctly remember sitting on my bed and looking at my chair. I can't recall the precise thoughts leading to this, but I remember thinking about what qualities the chair had and what you could do with it. I had a distinct mental image of a HUD overlaid across my vision, data-sheet style, with blue lines and boxes popping out of the chair detailing different qualities. There are many instances of chairs, but they share common qualities you could define (attributes) and common things you can do with them, like sit (behaviours). A class is like an abstract version of the blueprint that must have existed to produce the chair I was looking at.

Neural Networks and the Vision at the Desk

Programming is my first love, but deep learning is a joint second. This goes back to when I first learnt what DeepMind was doing with deep reinforcement learning on Atari games. You gave a neural network a game and a way of taking actions, and somehow it learned strategies to solve these games without being told how. At the time, I had no notion of problem formulation, what neural nets could even be applied to, or how they connected with programming.

I would eventually understand years later. I was deeply studying and thinking, having thrown myself repeatedly at the proverbial wall of learning ML and deep learning without significant leaps in understanding. One day, after deeply pondering datasets, I literally had an uncontrollable vision (open-eyed) of data floating in space in front of me. It was as though the data had leaped from the screen into my room. It floated in front of me, in 3D, with a machine learning algorithm (in the case of classification) dividing the space into subsets based on defined criteria.

This insight might seem obvious (perhaps it took me longer because I'm self-taught), but it had such a powerful impact on me that it made me tear up. I could finally see what was happening, I could see the game. All these different, overlapping pieces of the puzzle were finally laid out clearly in front of me, and I could understand it all. A dataset is a snapshot of a statistical reality you wish to model.

Representation Learning as Compression

It became clear to me that deep learning really ought to be called deep representation learning. We want to coax an algorithm to learn a useful representation of something that makes sense and aligns with how we think it should be. As an aside: I find it deeply strange that neural nets can learn representations at all. Obviously, we formulate a problem with constraints and an objective, but the volume of the input space is so sparse that neural networks shouldn't necessarily be able to learn anything meaningful at all (the curse of dimensionality).

The major insight here was realising how a neural network takes raw data and applies transformations to make it useful or simpler to solve in some way, unlike classical machine learning with handcrafted features.

This approach mimics what our own brains do (assuming we follow predictive coding): updating a mental model of our environment based upon raw sensory data (auditory, visual, balance, etc.). Of course, we can only learn from what we can see and understand, which makes me think of perception as a lossy compression of reality. A model is a compression of reality into useful abstractions.

Programming vs ML

Following this idea of models as compressions of reality, you can think of a computer program as a model of reality, written by hand. With machine learning, we are searching for the program that models reality. Building models is literally perceiving reality through abstraction.

World Models and Data as Partial View

This perspective makes everything else fall into place. One idea I grokked from a great image in the Platonic Representation Hypothesis paper is that observations (like images or datasets) are snapshots of reality. This sort of thinking applies to every subject. Even music feels like a representation of energy swelling and falling, which we perceive as emotion or passion. (Maybe that's a flawed analogy, but it helps me think.)

How I Think When I Code Now

This thinking directly influences how I program. I begin by defining the reality that I am painting with code. One specific idea that sticks with me is how names are handles on ideas. I think about what first-class citizens I have (e.g. Job, Person, Animal), how they're composed, how they interact, and chains of operations.

I start by writing function stubs and test cases to crystallise and anchor the reality I'm painting in, well, reality. This gives me a way to encode my assumptions about reality into code and make sure it adheres to them. If my assumptions are incorrect or reality shifts, the tests and code shift too.

Alignment and Uncertainty

I'm currently trying to reconcile how to know when my model aligns with reality. I don't think it's truly possible to definitively arrive at the model of reality, but you can get pretty close through testing, exploration, or first-principles thinking. One useful heuristic I've found is paying attention to the tension I feel when thinking through an idea or learning something new. As I mentioned earlier about names being handles on ideas, I think about tension as a discrete thing and attempt to maximise it, following it to its root to understand why I'm struggling.

Even in areas far from code, like learning guitar, I've noticed the same tension guiding me. I realised I couldn't play freely because I only knew how to mimic, not express. That tension pointed me toward melody, toward movement, toward noticing how my ring finger pulls my pinkie with it, breaking flow. Even that's a kind of modelling.

I suppose overall I am trying to speak to how my learning journey has deeply formed the bedrock of how I perceive and approach life. I haven't figured much out yet, but I'm trying and exploring, and that is good enough.