I Can’t Think of an Apple vs Wolfs pun, sorry

Okay, for everyone who manages their information diet better than I do, here’s the short, short version: As part of Apple’s expanding TV&Movie ambitions, they made a movie called Wolfs starring George Clooney and Brad Pitt, directed by Jon Watts (the guy who did the Tom Holland Spider-Mans. Spider-Men?) Apparently, at the last second, Apple changed their mind about doing a real theatrical release despite earlier promises, and now Jon Watts is going all scorched earth on the promo circuit.

For more:

Director Explains "Wolfs 2" Cancellation - Dark Horizons

Michael Tsai - Blog - Apple’s Movie Theater Beef With Hollywood

My initial reaction was, well, could have been worse, we all could have woken up with a copy of Wolfs on our phones we can never delete.

But my real reaction to all this is: Aaahhhhh, Hollywood is learning why there aren’t any exclusive video games on Macs. Because man, whatever else, it is so funny to watch Apple treat literally George Clooney and Brad Pitt like they were independent iOS devs.

It feels really clarifying, though. Turns out that old thing about “Apple is number 1, Users are Number 2, Devs are Number 3” applies to everyone. And that’s the thing about Apple, is they just can’t do “partners”. They just institutionally can’t handle doing something that might make them less money than they could just so their partner will be happy. They just can’t do it. There’s no spreadsheet column for “this option will lose us less money but make two of the biggest movie stars in the world mad.”

So the stuff they do well are things they can do completely on their own, and things that require an actual partner, as opposed to a vendor, always fail.

I don’t have a snappy conclusion here, but I do expect a lot of sixcolors apple report cards next year to include a phrase along the lines of “but they didn’t even treat George Clooney right, so I don’t know what I expected.”

Delicious Library Crosses Over

Okay, this one hurts. Amazon has finally turned off the thing Delicious Library used, so the app has been discontinued.

Over at Daring Fireball, Gruber has a really nice writeup about how Delicious Library was the exemplar for the era of apps as art in their own right that seems to have mostly passed: Daring Fireball: The End of the Line for Delicious Library

Personally? I have a Delicious Library that contains (almost) everything I own. “Dad, did you scan these yet” was a standard phase of any new purchase or gift. Books, games, toys, whatever, the thing where it could pull data on anything with a UPC or EAN from Amazon was amazing. Even more amazing was the barcode scanner app for the iPhone that used the camera. Just walk around the house zip, zip, zip.

You know how they say you should have a list of everything you own for insurance purposes? I had that! Plus, the ability to see when something was bought was surprisingly useful. “What birthday was that?” You could answer those questions! I loved it.

It was clear DL was on its way out: Wil Shipley has been an Apple employee for years now, the app hasn’t been updated in ages, the Amazon link was getting… “sketchy.” But there just isn’t a replacement.

This is the sort of think where all you can do is throw your hands up and make a sort of “ecchhhhh” sound. One more great thing we used to have that’s gone because the easiest way for some middle manager somewhere to make one of their KPIs was to break it.

Although, maybe the most maddening thing about this one is that I would have absolutely paid some kind of subscription fee to Amazon (directly or indirectly) to keep this working, and… nope. Not an option.

Software Forestry 0x07: The Trellis Pattern

In general, rewriting a system from scratch is a mistake. There are times, however, where replacing a system makes sense.

You’ve checked off every type in the Overgrowth Catalogue, entropy is winning, and someone does a back of the envelope cost analysis and finally says “you know, I think it really would be cheaper to replace this thing.”

Part of what makes this tricky is that presumably, this system has customers and is bringing in revenue. (If it didn’t you could just shut it down and build something new without a hassle.) And so, you need to build and deploy a new system while the old one is still running.

The solution is to use the old system as a Trellis.

The ultimate goal is to eventually replace the legacy system, but you can’t do it all at once. Instead, use the existing system as a Trellis, supporting the new system as it grows along side.

This way, you can thoughtfully replace a part at a time, carefully roll it out to customers, all while maintaining the existing—and working—functionality. The system as a whole will evolve from the current condition to the improved final form.

As you work through each capability, you can either use the legacy system as a blueprint to base the new one on, or harvest code for re‐use.

The great thing about a Trellis is that the Trellis and the Tree are partners. They have the same goals: for the tree to outgrow the trellis. But the trellis can be an active partner. I have a tree right now that still has part of a trellis holding up some branches. It’s one of the those trees that got a little too big a little too fast, put on a little too much fruit. A few years ago it had supports and trellises all around, now it’s down to just one whimsically-leaning stick. If things go well, I’ll be able to pull that out next summer.

The old system isn’t abandoned, it transitions into a new role. You can add features to make it work better as a trellis: an extra API endpoint here, a data export job there. The new system calls into the old one to accomplish something, and then you throw the switch and the old system starts calling into the new one to do that same thing.

Eventually the new system pulls away from the Trellis, grows beyond it. And if you do it right, the trellis falls away when its job is done. Ideally, in such a way that you could never tell it was there in the first place.

Sometimes, you can schedule the last switchover and have a big party when you turn the old system off. But if you’re really lucky, there comes a day where you realize the old load balancer crashed a month ago and no one noticed.

🌲

There’s a social & emotional aspect to this as well, which goes almost entirely undiscussed.

If we’re replacing a system, it’s probably been around for a while. We’re probably talking about systems with a decade+ of run time. The original architects may have moved on, but there are people who work on it, probably people who have built a whole career out of keeping it running.

There’a always some emotions when it comes time to start replacing the old stuff. Some stereotypes exist for a reason, and the sorts of people who become successful software engineers or business people tend to be the sorts of folks for whom “empathy” was their dump stat. There’s something galling about the newly arrived executive talking about moving on from the old and busted system, or the new tech lead espousing how much better the future is going to be. The old system may have gotten chocked out by overgrowth, left behind by the new growth of the tech industry, but it’s still running, still pulling in revenue. It’s paying for the electricity in the projector that’s showing the slide about how bad it is. It deserves respect, and so do the people who worked on it.

Thats the point: The Trellis is a good thing, it’s positive. The old system and—the old system’s staff—have a key role to play. Everyone is on the same team, everyone has the same goal.

🌲

There’s an existing term that’s often used for a pattern similar to this. I am, of course, talking about the Strangler Fig pattern. I hate this term, and I hate the usual shorthand of “strangler pattern” even more.

Really? Your mental model for biring in new software is that it’s an invasive parasite that slowly drains nutrients and kills its host? There are worse ways to go than being strangled, but not by much.

This isn’t an isolated style of metaphor, either. I used to work with someone—who was someone I liked, by the way—who used to say that every system being replaced needed someone to be an executioner and an undertaker.

Really? Your mental model for something ending is violent, state-mandated death?

If Software Forestry has a central thesis, it is this: the ways we talk about what we do and how we do it matter. I can think of no stronger example of what I mean than otherwise sane professionals describing their work as murdering the work of their colleagues, and then being surprised when there’s resistance.

What we do isn’t violent or murderous, it is collaborative and constructive.

What I dislike the most about Strangler Figs, though, is that a Strangler Fig can never exceed the original host, only succede. The Fig is bound to the host forever, at first for sustenance, and then, even after the host has died and rotted away, the Fig has an empty space where the host once stood, a ghost haunting the parasite that it can never fully escape from.

🌲

So if we’re going to use a real tree as our example for how to do this, let’s use my favorite trees.

Let me tell you about the Coastal Redwoods.

The Redwood forest is a whole ecosystem to itself, not just the trees, but the various other plants growing beneath them. When a redwood gets to the end of its life, it falls over. But that fallen tree then serves as the foundation to a whole new mini-ecosystem. The ferns and sorrel cover the fallen trunk. Seedlings sprout up in the newly exposed sunlight. Burls or other nodes sprout new trees from the base of the old, meaning maybe the tree really didn’t die at all, it just transitioned. From one tree springs a whole new generation of the forest.

There are deaths other than murder, and there are endings other than death.

Let’s replace software like a redwood falling; a loud noise, and then an explosion of new possibilities.

Software Forestry 0x06: The Controlled Burn

Did you know earthworms aren’t native to North America? Sounds crazy, but it’s true; or at least it has been since the glaciers of the last ice age scoured the continent down to the bedrock and took the earthworms with them. North America certainly has earthworms now, but as a recently introduced invasive species. (For everyone who just thought “citation needed”, Invasive earthworms of North America.)

As such, the biomes North America have very different lifecycles than their counterparts in Eurasia do. In, say, a Redwood Forest, organic matter builds up in a way it doesn’t across the water. Things still rot, there’s still fungus and microbes and bugs and things, but there isn’t a population of worms actively breaking everything down. The biomass decays slower. Some buildup is a good thing, it provides a habitat for smaller plants and animals, but if it builds up too much, it can start choking plants out before it can break down into nutrients.

So what happens is, the forest catches on fire. In a forest with earthworms, a fire pretty much always a bad thing. No so much in the Redwoods, or other Californian forests. The trees are fire resistant, the fire clears away the excess debris, frees those nutrients, and many species of cone-bearing conifer trees—redwoods, pines, cypresses, and the like—have what are called “serotinous” cones, which means they only germinate after a fire. Some are literally covered in a layer of resin that has to melt off before the seeds can sprout. The fire rips though, clears out the debris, and the new plants can sprut in the newly fertilized ground. Fire isn’t a hazard to be endured, it’s been adopted as a critical part of the entire ecosystem’s lifecycle.

Without human intervention, fires happen semi-regularly due to lighting. Of course, that’s a little unpredictable and doesn’t always turn out great. But the real problem is when humans prevent fires from taking hold, and then no matter how much you “sweep the forest,” the debris and overgrowth builds up and builds up, until you get the really huge fires we’ve been having out here.

The people who used to live here (Before, ahh… a bunch of other people “showed up and took over” who only knew how to manage forests with earthworms) knew what the solution was: the Controlled Burn. You choose a time, make some space, and carefully set the fire, making sure it does what it needs to do in the area you’ve made safe, but keep it out of places where the people are. In CA at least, we’re starting to adopt controlled burns as an intentional management technique again, a few hundred years later. (The biology, politics, history, and sociology of setting a forest on fire on purpose are beyond our scope here, but you get the general idea.)

I think a lot of Software Forests are like this too.

Every place I’ve ever worked has struggled with figuring out how to plan and estimate ongoing maintenance outside of a couple of very narrow cases. If it’s something specific, like a library upgrade, or a bug, you can usually scope and plan that without too much trouble. But anything larger is a struggle, because those larger maintenance and care efforts are harder to estimate, especially when there isn’t a specific & measurable customer-facing impact. You don’t have a “thing” you can write a bug on. You don’t know what the issues are, specifically, it’s just acting bad.

The problem requires sustained focus, the kind that lasts long enough to actually make a difference. And that’s hard to get.

One of the reasons why Cutting Trails is so effective is that it doesn’t take that much more time than the work the trail is being cut towards. Back when estimating via Fibonacci Sequence was all the rage, the extra work to cut the trail usually didn’t get you up to the next fibonacci number.

Furthermore, the effort to get in and actually estimate and scope some significant maintenance work is often more work than the actual changes. It’s wasteful to spend a week investigating and then write up a plan for someone to do later. You’re already in there!

Finally, rarely is there a direct advocate. There’s nearly always someone who acts as the Voice of the Customer, or the Voice of the Business, but very rarely is anyone the Voice of the Forest.

(I suspect this is one of the places where agile leads us astray. The need to have everything be a defined amount of work that someone can do in under a week or two makes it incredibly easy to just not do work that doesn’t lend itself to being clearly defined ahead of time.)

So the overgrowth and debris builds up, and you get the software equivalent of an unchecked forest fire: “We need to just rewrite all of this.”

No you don’t! What you need are some Controlled Burns.

It goes like this:

Most Forests have more than one application, for a wide definition of “application.” There’s always at least one that’s limping along, choked with Overgrowth. Choose one. Find a single person to volunteer. (Or get volun-told.) Clear their schedule for a month. Point them at the app with overgrowth and let them loose to fix stuff.

We try to be process-agnostic here at Software Forestry, but we acknowledge most folks these days are doing something agile, or at least agile adjacent. Two-week sprints seems to have settled as the “standard” increment size; so a month is two sprints. That’s not nothing! You gotta mean it to “lose” a resource for that much time. But also, you should be able to absorb four weeks of vacation in a quarter, and this is less disruptive than that. Maybe schedule it as one sprint with the option to extend to a second depending on how things look “next week.”

It helps, but isn’t mandatory, to have success metrics ahead of time. Sometimes, the right move is to send the person in there and assume you’ll find something to paint a bullseye around. But most of the time you’ll want to have some kind of measurement you can do a before-and-after comparison with. The easiest ones are usually performance related, because you can measure those objectively, but probably aren’t getting handled as part of the normal “run the business.” Things like “we currently process x transactions per second, we need to get that to 2x,” or “cut RAM use by 10%,” or “why is this so laggy sometimes?”

I did a Controlled Burn once on a system that needed to, effectively, scan every record in a series of database tables to check for things that needed to be deleted off of a storage device. It scanned everything, then started over and scanned everything again. When I started, it was taking over a day to get through a cycle, and that time was increasing, because it wasn’t keeping up with the amount of new work sliding in. No one knew why it took that long, and everyone with real experience with that app was long gone from the company. After a month of dedicated focus, it got through a cycle in less than two hours. Fixed a couple bits of buggy behavior while I was at it. No big changes, no re-architecture, no platform changes, just a month of dedicated focus and cleanup. A Controlled Burn.

This is the time to get that refactoring done—fix that class hierarchy, split that object into some collaborators. Write a bunch of tests. Refactor until you can write a bunch of tests. Fix that thing in the build process everyone hates. Attach some profilers and see where the time is going.

Dig in, focus, and burn off as much overgrowth as you can. And then leave a list of things to do next time. You should know enough now to do a reasonable job scoping and estimating the next steps, write those up for the to do list. Plants some seeds for new growth. You shouldn’t have to do a Controlled Burn on the same system twice.

Deploying this kind of directed focus can be incredibly powerful. The average team can absorb maybe one or two of these a year, so deploy them with purpose.

🌲

Sometimes, all the care in the world won’t do the trick, and you really do need to replace a system. Next time: The Trellis Pattern

Software Forestry 0x05: Cutting Trails

The lived reality of most Software Foresters is that we spend all our time with large systems we didn’t see the start of, and won’t see the end of. There’s a lot of takes out there about what makes a system “Legacy” and not “just old”, but one of the key things is that Legacy Systems are code without continuity of philosophy.

Because almost certainly, the person who designed and built in the first place isn’t still here. The person that person trained probably isn’t still here. Given the average tenure time in tech rolls, it’s possible the staff has rolled over half-a-dozen or more times.

Let me tell you a personal example. A few lifetimes ago, I worked on one of these two-decade old Software Forests. Big web monolith, written in what was essentially a homebrew MVC-esque framework which itself was written on top of a long-deprecated 3rd party UI framework. The file layout was just weird. There clearly was a logic to the organization, but it was gone, like tears in the rain.

Early on, I had a task where I needed to add an option to a report generator. From the user perspective, I needed to add an option to a combobox on a web form, and then when the user clicked the Generate button, read that new option and punch the file out differently.

I couldn’t find the code! I finally asked my boss, “hey, is there any way to tell which file has which UI pages?” The response was, “no, you just gotta search.”

As they say, Greppability Is an Underrated Code Metric.

(Actually, I’ve worked on two big systems now where I had essentially this exact conversation. The other one was the one where one of the other engineers described it as having been built by “someone who knew everything about how JSP tags worked except when to use them.”)

So you search for the distinctive text in the button, or the combo box, or something on the page. You find the UI. Then you start tracing in. Following the path of execution, dropping into undocumented methods with unhelpful names, bouncing into weird classes with no clear design, strange boundaries, one minute a function with a thousand lines, the next minute an inheritance hierarchy 10 levels of abstraction deep to call one line.

At this point you start itching. “I could rewrite all of this,” you think. “I could get this stood up in a weekend with Spring Boot/React/Ruby on Rails/Elixir/AWS Lambdas/Cool Thing I Used Last”. You start gazing meaningfully at the copy of the Refactoring book on the shelf. But you gotta choke down that urge to rebuild everything. You have a bug you have to fix or a feature to deploy. But it’s not actually going to get better if you keep digging the hole deeper. You gotta stop digging, and leave things better for the next person.

You need to Cut a Trail.

1. Leave Trail Markers.

First thing is, you have to figure out what it does now before you change anything. In a sane, kind world, there would be documentation, diagrams, clear training. And that does happen, but very, very rarely. If you’re very lucky, there’s someone who can explain it to you. Otherwise, you have to do some Forensic Architecture.

Talk to people. Add some logging. Keep clicking and watching what it does. Step through it in a debugger, if you can and if that helps, although I’ve personally found that just getting a debugger working on a live system is often times more work than is worth it for the information you get out of it. But most of all, read. Read the code closely, load as much of that system’s state and behavior into your mind as you can. Read it like you’re in High School English and trying to pull the symbolism out of The Grapes or Wrath or The Great Gatsby. That weird function call is the light at the end of the pier, those abstractions are the eyes on the billboard—what do they mean? Why are they here? How does any of this work?

You’ll get there, that’s what we do. There’s a point where you’ll finally understand enough to make the change you want to make. The trick is to stop at this point, and write everything down. There’s a “haha only serious” joke that code comments are for yourself in six months, but—no. Your audience here is you, a week ago. Write down everything you needed to know when you started this. Every language has a different way to do embedded documentation or comments, but they all have a way to do it. Document every method or function that your explored call path went through. Write down the thing you didn’t understand when you started, the strange overloaded behavior of that one parameter, what that verb really means in the function name, as much as possible, why it does what it does.

Take an hour and draw the diagram you wish you’d had. Write down your notes. And then leave all that somewhere that other people can find it. If you’re using a language where the imbedded documentation system can pull in external files, check that stuff right on in to the codebase. Most places have an internal wiki. Make a page for you team if there isn’t one. Under that, make a page for the app if it doesn’t have one. Then put all that you’ve learned there.

Something else to make sure to document early on: terminology. Everyone uses the same words to mean totally different things. My personal favorite example: no two companies on earth use the word “flywheel” the same way. It doesn’t matter what it was supposed to mean! Ask. Then write it down. The weird noun you tripped over at the start of this? Put the internal definition somewhere you would have found it.

People frequently object that they don’t have the time to do this, to which I say: you’ve already done the hard part! 90% of the time for this task was figuring it out! Writing it down will take a fraction of the time you already had to spend, I promise. And when you’re back here in a year, the time you save in being able to reload all that mental state is going to more than pay for that afternoon you spent here.

2. Write Tests.

Tests are really underrated as a documentation and exploration technique. I mean, using them to actually test is good too! But for our purposes we’re not talking about a formal TDD or Red-Green-Refactor–style approaches. That weird function? Slap some mocks and stubs together and see what it does. Throw some weird data at it. You’re goal isn’t to prove it correct, but to act like one of those Edwardian Scientists trying to figure out how air works.

Another Forest I inherited once, which was a large app that customers paid real money to use, had a test suite of 1 test—which failed. But that was great, because there was already a place to write and run tests.

Tests are a net benefit, they don’t all have to be thorough, or fall into strict unit/integration/acceptance boundaries. Sometimes, it’s okay to put a couple of little weird ones in there that exist to help explain what some undocumented code does.

If you’re unlucky enough to run into a Forest with no test runner, trust me, take the time to bolt one on. It doesn’t have to be perfect! But you’ll make that time back faster than you’d believe.

When you get done, in additional to whatever “normal” Unit or Integration tests your process requires or requests, write a really small test that demonstrates what you had to do. Link that back to the notes you wrote, or the documentation you checked in.

3. A Little Cleanup, some Limited Refactoring

Once you have it figured out, and have a test or two, there’s usually two strong responses: either “I need to replace all of this right now”, or “This is such a mess it’ll never get better.”

So, the good news is that both of those are wrong! It can get better, and you really probably shouldn’t rework everything today.

What you should do a little cleanup. Make something better. Fix those parameter names, rename that strangely named function. Heck, just fix the tenses of the local variables. Do a little refactoring on that gross class, split up some responsibilities. It’s always okay to slide in another shim or interface layer to add some separation between tangled things.

(Don’t leave a huge mess in the VC diff, though, please.)

Leave the trail a little cleaner than when you found it. Doesn’t have to be a lot, we don’t need to re-landscape the whole forest today.

4. Write the New Stuff Right (as possible)

A lot of the time, you know at least one thing the original implementors didn’t: you know how the next decade went. It’s very easy to come in much later, and realize how things should have been done in the first place, because the system has a lot more years on it now than it used to. So, as much as you can, build the new stuff the right way. Drop that shim layer in, encapsulate the new stuff, lay it out right. Leave yourself a trail to follow when you come back and refactor the rest of it into shape.

But the flip side of that is:

5. Don’t be a Jerk About It

Everyone has worked on a codebase where there’s “that” module, or library, or area, where “that guy” had a whole new idea about how the system should be architected, and it’s totally out of place with everything else. A grove of palm trees in the middle of a redwood forest. Don’t be that guy.

I worked on an e-commerce system once where the Java package name was something like com.company.services.store, and then right next to it was com.company.services.store2. The #2 package was one former employee’s personal project to refactor the whole system; they had left years before with it half done, but of course it was all in production, and it was a crapshoot which version other parts of the system called into. Don’t do that.

After you’re gone, when someone looks at the version control change log for this part of the system, you want them to see your name and think “oh, this one had the right idea.”

Software Forestry is a group project, for the long term. Most of the time, “consistency and familiarity” are more valuable than some kind of quixotic quest for the ideal engineering. It’s okay, we’ll get there. Keep leaving it better than you found it. It’ll be worth it.

🌲

You’ll be amazed what your overgrown codebase looks like after a couple months of doing this. That tangled overgrowth starts to look positively tidy.

But sometimes, just cutting trails doesn’t get you there. Next Time: The Controlled Burn.

Software Forestry 0x04: Library Upgrade Week

Here at Software Forestry we do occasionally try to solve problems instead of just turning them into lists of smaller problems, so we’re goona do a little mood pivot here and start talking about how to manage some of those forms of overgrowth we talked about last time.

First up: let me tell you the good news about Library Upgrade Week.

Just about any decently sized software system uses a variety of third party libraries. And why wouldn’t you? The multitude of high-quality libraries and frameworks out there is probably the best outcome of both the Open Source movement and Object-Oriented Software design. The specific mechanics vary between languages and their practitioner’s cultures, but the upshot is that very rarely does anyone build everything from scratch. There’s no need to go all historical reenactment and write your own XML parser.

Generally, people keep those libraries pinned and stay on one fixed version, rather than schlepping in a new version every-time an update happens. This is a good thing! Change is risk, and risk should be taken on intentionally. The downside is that those libraries keep moving forward, the version you’re using slips out of date, and now you have a bunch of Overgrowth. And so that means you need to upgrade them.

The upshot of all that is on a semi-regular basis, we all need to slurp in a bunch of new code no-one on the payroll wrote, and don’t really know how to test. Un-Reviewed Code Is Tech Debt, and one of the mantras of writing good tests is “don’t test the framework”, so this is always a little iffy.

It’s incredibly easy to just keep letting those weeds grow a little longer. “The new version doesn’t have anything we need”, “there’s no bugs”, “if it ain’t broke don’t fix it”, and so on. They always take too long, don't usually deliver immediate gratification, and are hard to schedule. It’s no fun, and no one likes to do it.

The trick is to turn it into a party.

It works like this: set aside the last week of the quarter to concentrate on 3rd party library upgrades. Regardless of what your formal planning cycle or process is, most businesses tends to operate on quarters, and there’s usually a little dead time at the end of the quarter you can repurpose.

The Process:

1. Form squads. Each squad is a group of like-minded individuals focused on a single 3rd party library or framework. Squads are encouraged to be cross-team. Each squad will focus on updating that 3rd party library in all applications or places where it is used. The intent is to make this a group event, where people can help each other out. Participation is not mandatory.

2. Share squad membership and goals ahead of time. Leadership should reserve the right to veto libraries as “too scary” or “not scary enough”. Libraries with a high severity alerts or known CVE are good candidates.

3. That week, each squad self organizes and works as a group through any issues caused by the upgrade. Other than major outages or incidents, squad members should be excused from other “run the business” type work for that week; or rather, the library upgrades are “the business.” Have fun!

4. On that Friday hold the Library Upgrade Week Show-n-Tell. Every team should demo what they did, how they did it, and what it took to pull it off. Tell war stories, hold a happy hour, swap jokes. If a squad doesn’t finish that's okay! The expectation is that they’ll have learned a lot about what it'll take to finish, and that work will be captured in the relevant team’s todo lists. If you’re in a process with short develop-deploy increments (like sprints) you can make the library upgrade(s) a release on its own. Ideally you already have a way to sign off a release as not containing regressions, and so a short release with just a library upgrade is a great way to make sure you didn’t knock some dominos over.

But wait! There's more! All participants will vote on awards to give to squads, for things like:

• Error message with least hits on Stack Overflow
• Largest version number jump
• Most lines changed
• Fewest lines changed
• Best team name
• Best presentation

Go nuts! Have a great time!

🌲

Yes, it’s a little silly, but that’s the point. I’ve deployed a version of this at a couple of jobs now, and it’s remarkable how effective it is. The first couple of cycles people hit the “easy” ones—uprev the logging library or a JSON parser or something. But then, once people know that Library Upgrade Week is coming, they start thinking about the harder stuff, and you start getting people saying they want to take a swing at the main framework, or the main language version, or something else load-bearing. It’s remarkable how much progress two or three people can make on a problem that looks unsolvable when they have an uninterrupted week to chew on it. (If you genuinely can’t spare a handful of folks to do some weeding four weeks out the the year, that’s a much larger problem than out of date libraries, and you should go solve that problem first. Like, right now.)

There’s an instinct to take the core idea to schedule this kind of maintenance for a few times a year, but leave off the part where it’s a party. This is a mistake. This is work people want to do even less than their usual work; the trick is to make everthing around it fun.

We’re Foresters, and both we and the Forest are here long term. The long term health of both depends on the care of the Forest being something that the Foresters enjoy, and it’s okay to stack that deck in your favor.

Wacky Times for AI

Been a wacky month or two for AI news! Open AI is reorganizing! Apple declined to invest! Whatsisname decided he wanted equity after all! The Governor of CA vetoed an AI safety bill! Microsoft is rebooting Three Mile Island, which feels like a particularly lazy piece of satire from the late 90s that escaped into reality! Study after study keeps showing no measurable benefit to AI deployment! The web is drowning in AI slop that no one likes!

I don’t know what any of that means, but it’s starting to feel like we’re getting real close to the part of The Sting where Kid Twist tells Quint from Jaws something confusing on purpose.

But, in our quest here at Icecano to bring you the best sentences from around the web, I’d like to point you at The Subprime AI Crisis because it includes this truly excellent sentence:

Generative AI must seem kind of magical when your entire life is either being in a meeting or reading an email

Oh Snap!

Elsewhere in that piece it covers the absolutely eye-watering amounts of money being spent on the plagiarism machine. There are bigger problems than the cost; the slop itself, the degraded information environment, the toll on the actual environment. But man, that is a hell of a lot of money to just set on fire to get a bunch of bad pictures no one likes. The opportunity cost is hard to fathom; imagine what that money could have been spent on! Imagine how many actually cool startups that would have launched! Imagine how much real art that would have bought!

But that’s actually not what I’m interested in today, what I am interested in are statements like these:

State of Play: Kobold Press Issues the No AI Pledge

Legendary Mario creator on AI: Nintendo is “going the opposite direction”

I stand by my metaphor that AI is like asbestos, but increasingly it’s also the digital equivalent of High Fructose Corn Syrup. Everyone has accepted that AI stuff is “not as good”, and it’s increasingly treated as low-quality filler, even by the people who are pushing it.

What’s intriguing to me is that companies whose reputation or brand centers around creativity or uniqueness are working hard to openly distance themselves. There’s a real “organic farm” energy, or maybe more of a “restaurant with real food, not fast food.”

Beyond the moral & ethical angle, it gives me hope that “NO AI” is emerging as a viable advertising strategy, in a way that “Made with AI” absolutely isn’t.

Software Forestry 0x03: Overgrowth, Catalogued

Previously, we talked about What We Talk About When We Talk About Tech Debt, and that one of the things that makes that debt metaphor challenging is that it has expanded to encompass all manner of Overgrowth, not all of which fits that financial mental model.

From a Forestry perspective, not all the things that have been absorbed by “debt” are necessarily bad, and aren’t always avoidable. Taking a long-term, stewardship-focused view, there’s a bunch of stuff that’s more like emergent properties of a long-running project, as opposed to getting spendy with the credit card.

So, if not debt, what are we talking about when we talk about tech debt?

It’s easy to get over-excited about Lists of Things, but I got into computer science from the applied philosophy side, rather than the applied math side. I think there are maybe seven categories of “Overgrowth” that are different enough to make it useful to talk about them separately:

1. Actual Tech Debt.

Situations where an explicit decision to do something “not as good” to ship faster. There’s two broad subcategories here: using a hacky or unsustainable design to move faster, and cutting scope to hit a date.

In fairness, the original Martin Fowler post just talks about “cruft” in a broad sense, but generally speaking “Formal” (orthodox?) tech debt assumes a conscious choice to accept that debt.

This is the category where the debt analogy works the best. “I can’t buy this now with cash on hand, but I can take on more credit.” (Of course, this also includes “wait, what’s a variable rate?”)

In my experience, this is the least common species of Overgrowth, and the most straightforwardly self correcting. All development processes have some kind of “things to do next” list or backlog, regardless of the formal name. When making that decision to take on the debt, you put an item on the todo list to pay it off.

That list of cut features becomes the nucleus of the plan for the next major version, or update, or DLC. Sometimes, the schedule did you a favor, you realize it was a bad idea, and that cut feature debt gets written off instead of paid off.

The more internal or infrastructure–type items become those items you talk about with the phrase “we gotta do something about…”; the logging system, the metrics observability, that validation system, adding internationalization. Sometimes this isn’t a big formal effort, just a recognition that the next piece of work in that area is going to take a couple extra days to tidy up the mess we left last time.

Fundamentally, paying this off is a scheduling and planning problem, not a technical one. You had to have some kind of an idea about what the work was to make the decision to defer the work, so you can use that same understanding to find it a spot on the schedule.

That makes this the only category where you can actually pay it off. There’s a bounded amount of work you can plan around. If the work keeps getting deferred, or rescheduled, or kicked down the road, you need to stop and ask yourself if this is actually debt or a something asperational that went septic on you.

2. We made the right decision, but then things happened.

Sometimes you make the right decisions, don’t choose to take on any debt, and then things happen and the world imposes work on you anyway.

The classic example: Third party libraries move forward, the new version isn’t cleanly backwards compatible, and the version you’re using suddenly has a critical security flaw. This isn’t tech debt, you didn’t take out a loan! This is more like tech property taxes.

This is also a planning problem, but tricker, because it’s on someone else’s schedule. Unlike the tech debt above, this isn’t something you can pay down once. Those libraries or frameworks are going to keep getting updated, and you need to find a way to stay on top of them without making it a huge effort every time.

Of course, if they stop getting updated you don’t have an ongoing scheduling problem anymore, but you have the next category…

3. It seemed like a good idea at the time.

Sometimes you just guess wrong, and the rest of the world zigs instead of zags. You do your research, weigh the pros and cons, build what you think is the right thing, and then it’s suddenly a few years later and your CEO is asking why your best-in-class data rich web UI console won’t load on his new iPad, and you have to tell him it’s because it was written in Flash.

You can’t always guess right, and sometimes you’re left with something unsupported and with no future. This is very common; there’s a whole lot of systems out there that went all-in on XML-RPC, or RMI, or GWT, or Angular 1, or Delphi, or ColdFusion, or something else that looked like it was going to be the future right up until it wasn’t.

Personally, I find this to be the most irritating. Like Han Solo would say, it’s not your fault! This was all fine, and then someone you never met makes a strategic decision, and now you have to decide how or if you’re going to replace the discontinued tech. It’s really easy to get into a “if it ain’t broke don’t fix it” headspace, right up until you grind to a halt because you can’t hire anyone who knows how to add a new screen to the app anymore. This is when you start using phrases like “modernization effort”.

4. We did the best we could but there are better options now.

There’s a lot more stuff available than there used to be, and so sometimes you roll onto a new project and discover a home-brew ORM, or a hand-rolled messaging queue, or a strange pattern, and you stop and realize that oh wait, this was written before “the thing I would use” existed. (My favorite example of this is when you find a class full of static final constants in an old Java codebase and realize this was from before Java 5 added enums.)

A lot of the time, the custom, hand-rolled thing isn’t necessarily “worse” than some more recent library or framework, but you have to have some serious conversations about where to spend your time; if something isn’t your core business and has become a commodity, it’s probably not worth pouring more effort in to maintaining your custom version. Everyone wants to build the framework, but no one really wants to maintain the framework. Is our custom JSON serializer really still worth putting effort into?

Like the previous, it’s probably time to take a deep breath and talk about re-designing; but unlike the previous, the persion who designed the current version is probably still on the payroll. This usually isn’t a technical problem so much as it is a grief management one.

5. We solved a different problem.

Things change. You built the right thing at the time, but now we got new customers, shifted markets, increased scale, maybe the feds passed a law. The business requirements have changed. Yesterday, this was the right thing, and now it isn’t.

For example: Maybe you had a perfectly functional app to sell mp3 files to customers to download and play on their laptops, and now you have to retrofit that into a subscription-based music streaming platform for smartphones.

This is a good problem to have! But you still gotta find a way to re-landscape that forest.

6. Context Drift.

There’s a pithy line that Legacy Code is “code without tests,” but I think that’s only part of the problem. Legacy code is code without continuity of philosophy. Why was it built this way? There’s no one left who knows! A system gets built in a certain context, and as time passes that context changes, and the further away we get from the original context, the more overgrown and weedy the system appears to become. Tests—good tests—are one way to preserve context, but not the only way.

A whole lot of what’s called “cruft” is here, because It’s harder to read code than to write it.. A lot of that “cruft” is congealed knowledge. That weird custom string utility thats only used the one place? Sure, maybe someone didn’t understand the standard library, or maybe you don’t know about the weekend the client API started handing back malformed data and they wouldn’t fix it—and even worse, this still happens at unpredictable times.

This is both the easiest and least glamorous to treat, because the trick here is documentation. Don’t just document what the code does, document why the code does what it does, why it was built this way. A very small amount of effort while something is being planted goes a long way towards making sure the context is preserved. As Henry Jones Sr, says, you write it down so you don’t have to remember.

To put all that another way: Documentation debt is still tech debt.

7. Not debt, just an old mistake.

The one no one likes to talk about. For whatever reason, someone didn’t do A-quality work. This isn’t necessarily because they were incompetent or careless, sometimes shit happens, you know? This the flip side of the original Tech Debt category; it wasn’t on purpose, but sometimes people are in a hurry, or need to leave early, or just can’t think of anything better.

And so for whatever the reason, the doors aren’t straight, there’a bunch of unpainted plywood, those stairs aren’t up to code. Weeds everywhere. You gotta spend some time re-cutting your trails through the forest.

🌲

As we said at the start, each of those types of Overgrowth has their own root causes, but also needs a different kind of forest management. Next week, we start talking about techniques to keep the Overgrowth under control.

Checking In On Space Glasses

It’s been a while since we checked in on Space Glasses here at the old ‘cano, and while we weren’t paying attention this happens: Meta Unveils 'Orion' Augmented Reality Glasses.

Most of the discussion—rightly—has focused on the fact that they’re a non-production internal prototype that reportedly costs 10 Gs a pop. And they’re a little, cough, “rubenesque”?

Alert readers will recall I spent several years working on Space Glasses professionally, trying to help make fetch happen as it were, and I looked at a lot of prototype or near-prototype space glasses. These Orion glasses were the first time I sat up and said “ooh, they might be on to something, there.”

My favorite piece about the Orions was Ben Thompson at Stratechery’s Interview with Meta CTO Andrew Bosworth About Orion and Reality Labs, which included this phenomenal sentence:

Orion makes every other VR or AR device I have tried feel like a mistake — including the Apple Vision Pro.

There are a lot of things that make Space Glasses hard, but the really properly hard part is the lenses. You need something optically clear and distortion-free enough to do regular tasks while looking through, while also being able to display high-enough resolution content to be readable without eyestrain, and do all that will being spectacle lens–sized, stay in sync with each other, and do it all while mounted in something roughly glasses-sized, and ideally without shooting lasers at anyone’s eyeballs or having a weird prism sidecar.

The rest of it: chunky bodies, battery life, software, those aren’t “Easy”, but making those better is a known quantity; it doesn’t require super-science, it’s “just work.”

I’m personally inclined to believe that a Steve Jobs–esque “one more thing” surprise reveal is more valuable than a John Sculley–style fantasy movie about Knowledge Navigators, but if I’d solved the core problem with space glasses while my main competitor was mired in a swamp full of Playthings for the Alone, I’d probably flex too.

Software Forestry 0x02: What We Talk About When We Talk About Tech Debt

Tell me if this sounds familiar: you start a new job, or roll onto a new project, or even have a job interview, and someone says in a slightly hushed tone, words to the effect of “Now, I don’t want to scare you, but we have a lot of tech debt.” Or maybe, “this system is all Legacy Code.” Usually followed by something like “but don’t worry! We’ve got a modernization effort!”

Everyone seems to be absolutely drowning in “tech debt”; hardly a day goes by where you don’t read another article about some system with some terrible problem that was caused by being out of date, deferred maintenance, “in debt.” We constantly joke about the fragile house-of-cards nature of basically, everything. Everyone is hacking their way, pun absolutely intended, through overgrown forests.

There’s a lot to unpack from all that. Other engineering disciplines don’t beg to rebuild their bridges or chemical plants after a couple of years, but they also don’t need to; they build them to last. How does this happen? Why is it like this?

For starters, I think this is one of those places where our metaphors are leading us wrong.

I can’t now remember when I first heard the term Technical Debt. I think it was early twenty-teens, the place I was working in the mid-aughts had a lot of tech debt but I don’t ever remember anyone using that term, the place I was working in the early teens also had a lot, and we definitely called it that.

One of the things metaphors are for is to make it easier to talk to people with a different background—software developers and business folks, for example. We might use different jargon in our respective areas of expertise, but if we can find an area of shared understanding, we can use that to talk about the same thing. “Debt” seems like a kind of obvious middle-ground: basically everyone who participates in the modern economy has a basic, gut-level understanding of how debt works.

Except, do they have the same understanding?

Personally, I think “debt” is a terrible metaphor, bordering on catastrophic. Here’s why: it has very, very different moral dimensions depending on whose talking about it.

To the math and engineering types who popularized the term, “debt” is obviously bad, bordering on immoral. They’re the kind of people who played enough D&D as kids to understand how probability works, probably don’t gamble, probably pay off their credit cards in full every month. Obviously we don’t want to run up debt! We need to pay that back down! Can’t let it build up! Queue that scene in Ghostbusters where Egon is talking about the interest on Ray’s two mortgages.

Meanwhile, when the business-background folks making the decisions about where to put their investments hear that they can rack up “debt” to get features faster, but can pay it off in their own time with no measurable penalty or interest, they make the obvious-to-them choice to rack up a hell of a lot of it. They debt-financed the company with real money, why not the software with metaphorical currency? “We can do it, but that’ll add tech debt” means something completely different to the technical and business staff.

Even worse, “debt” as a metaphor implies that it ends. In real life, you can actually pay the debt off; pay off the house, end the car payment, pay back the municipal bonds, keep your credit cards up to date, whatever. But keeping your systems “debt free” is a process, a way of working, not really something you can save up and pay off.

I’m not sure any single metaphor has done more damage to our industry’s ability to understand itself than “tech debt.”

Of course, the definiton “tech debt” expanded and has come to encompass everything about a software system that makes it hard to work on or the developers don’t like. “Cruft” is the word Fowler uses. “Tech debt”, “legacy”, “lack of maintenance” all kind of swirl into a big mish-mash, meaning, roughly, “old code that’s hard to work on.” Which makes it even less useful as a metaphor, because it covers a lot of different kinds of challenges, each of which calls for different techniques to treat and prevent. In fairness, Fowler takes a swing at categorizing tech debts via the Technical Debt Quadrant, which isn’t terrible, but is a little too abstract to reflect the lived reality.

This is a place where our Forestry metaphor offers up an obvious alternate metaphor: Overgrowth. Which gets close to heart of what the problem feels like: that we built a perfectly fine system, and now, after no action on our part, its not fine. Weeds. There’s that sense that it gets worse when you’re not looking,

There’s something very vexing about this. As Joel said: As if source code rusted. But somehow, that system that was just fine not that long ago is old and hard to work on now. We talk about maintenance, but the kind of maintenance a computer system needs is very different from a giant engine that needs to get oiled or it’ll break down.

I think a big part of the reason why it seems so endemic nowadays is that there was a whole lot of appetite to rewrite “everything for the web” in either Java or .Net around the turn of the century, at the same time a lot of other software got rebuilt mostly from scratch to support, depending on your field, Linux, Mac OS X, or post-NT Windows. There hasn’t been a similar “replant the forest” mood since, so by the teens everyone had a decade-old system with no external impetus to rebuild it. For a lot of fields, this was the first point where we had to think in terms of long term maintenance instead of the OS vendor forcing a rebuild. (We all became mainframe programmers, metaphorically speaking.) And so, even though the Fowler article dates to ’03, and the term is older than that, “Tech Debt” became a twenty-teens concern. Construction stopped being the main concern, replaced with care and feeding.

Software Forests need a different kind of tending than the old rewrite-updates-rewrite again loop. As Foresters, we know the codebases we work on were here before us, and will continue on after us. The occasional greenfield side project, the occasional big new feature, but mostly out job is to keep the forest healthy and hand it along to the next Forester. It takes a different, longer term, continuous world view than counting down the number of car payments left.

Of course, there’s more than one way a forest can get out of hand. Next Time: Types of Overgrowth, catalogued

Software Forestry 0x01: Somewhere Between a Flower Pot and a Rainforest

“Software” covers a lot of ground. As there are a lot of different kinds and ecosystems of forests, there are a lot of kinds and ecosystems of software. And like forests, each of those kinds of software has their own goals, objectives, constraints, rules, needs.

One of the big challenges when reading about software “processes” or “best practices” or even just plain general advice is that people so rarely state up front what kind of software they’re talking about. And that leads to a lot of bad outcomes, where people take a technique or a process or an architecture that’s intrinsically linked to its originating context out of that context, recommend it, and then it gets applied to situations that are wildly inappropriate. Just like “leaves falling off” means something very different in an evergreen redwood forest than it does in one full of deciduous oak trees, different kinds of software projects need different care. As practitioners, it’s very easy for us to talk past each other.

(This generally gets cited in cases like “if you aren’t a massive social network with a dedicated performance team you probably don’t need React,” but also, pop quiz: what kind of software were all the signers of the Agile Manifesto writing at the time they wrote and signed it?¹)

So, before we delve into the practice of Software Forestry, let’s orient ourselves in the landscape. What kinds of software are there?

As usual for our industry, one of the best pieces written on this is a twenty-year old Joel On Software Article,² where he breaks software up into Five Worlds:

1. Shrinkwrap (which he further subdivides into Open Source, Consultingware, and Commercial web based)
2. Internal
3. Embedded
4. Games
5. Throwaway

And that’s still a pretty good list! I especially like the way he buckets not based on design or architecture but more on economic models and business contraints.

I’d argue that in the years since that was written, “Commercial web-based” has evolved to be more like what he calls “Internal” than “Shrinkwrap”; or more to the point, those feel less like discrete categories than they do like convenient locations on a continuous spectrum. Widening that out a little, all five of those categories feel like the intersections of several spectrums.

I think spectrums are a good way to view the landcape of modern software development. Not discrete buckets or binary yes/no questions, but continuous ranges where various projects land somewhere in between the extremes.

And so, in the spirit of an enthusiastic “yes, and”, I’d like to offer up what I think are the five most interesting or influential spectrums for talking about kinds of software, which we can express as questions sketching out a left-to-right spectrum:

1. Is it a Flower Pot or a Sprawling Forest?
2. Does it Run on the Customer’s Computers or the Company’s Computers?
3. Are the Users Paid to Use It or do they Pay to Use It?
4. How Often Do Your Customers Pay You?
5. How Much Does it Matter to the Users?

🌲

Is it a Flower Pot or a Sprawling Forest?

This isn’t about size or scale, necessarily, as mich as it is about overall “complexity”, the number of parts. On one end, you have small, single-purpose scripts running on one machine, on the other end, you have sprawling systems with multiple farms or clusters interacting with each other over custom messaging busses.

How many computers does it need? How many different applications work together? Different languages? How many versions do you have to maintain at once? What scale does it operate at?³ How many people can draw an accurate diagram from memory?

This has huge impacts on not only the technology, but things like team structure, coordination, and planning. Joel’s Shrinkwrap and Internal categories are on the right here, the other three are more towards the left.

🌳

Does it Run on the Customer’s Computers or the Company’s Computers?

To put that another way, how much of it works without an internet connection? Almost nothing is on one end or the other; no one ships dumb terminals or desktop software that can’t call home anymore.

Web apps are pretty far to the right, depending on how complex the in-browser client app is. Mobile apps are usually in the middle somewhere, with a strong dependency on server-side resources, but also will usually work in airplane mode. Single-player Games are pretty far to the left, only needing server components for things like updates and achievement tracking; multiplayer starts moving right. Embedded software is all the way to the left. Joel’s Shrinkwrap is left of center, Internal is all the way to the right.

This has huge implications for development processes; as an example, I started my career in what we then called “Desktop Software”. Deployment was an installer which got burned to a disk. Spinning up a new test system was unbelievably easy, pull a fresh copy of the installer and install it into a VM! Working in a micoservice mesh environment, there are days that feels like the software equivalent of greek fire, a secret long lost. In a world of sprawling services, spinning up a new environment is sometimes an insurmountable task.

A final way to look at this: how involved do your users have to be with an update?

🌲

Are the Users Paid to Use It or do they Pay to Use It?

What kind of alternate options do the people actually using the software have? Can they use something else? A lot of times you see this talked about as being “IT vs commercial,” but it’s broader than that. On the extreme ends here, the user can always choose to play a different mobile game, but if they want to renew their driver’s license, the DMV webpage is the only game in town. And the software their company had custom built to do their job is even less optional.

Another very closely related way of looking at this: Are your Customers and Users the same people? That is, are the people looking at the screen and clicking buttons the same people who cut the check to pay for it? The oft-repeated “if you’re not the customer you’re the product” is a point center-left of this spectrum.

The distance between the people paying and the people using has profound effects on the design and feedback loops for a software project. As an extreme example, one of the major—maybe the most significant—differences between Microsoft and Apple is that Microsoft is very good at selling things to CIOs, and Apple is very good at selling things to individuals, and neither is any good at selling things the other direction.

Bluntly, the things your users care about and that you get feedback on are very, very different depending on if they paid you or if they’re getting paid to use it.

Joel’s Internal category is all the way to the left here, the others are mostly over on the right side.

🌳

How Often Do Your Customers Pay You?

This feels like the aspect that’s exploded in complexity since that original Joel piece. The traditional answer to this was “once, and maybe a second time for big upgrades.” Now though, you’ve got subscriptions, live service models, “in-app purchases”, and a whole universe of models around charging a middle-man fee on other transactions. This gets even stranger for internal or mostly-internal tools, in my corporate life, I describe this spectrum as a line where the two ends are labeled “CAPEX” and “OPEX”.

Joel’s piece doesn’t really talk about business models, but the assumption seems to be a turn-of-the-century Microsoft “pay once and then for upgrades” model.

🌲

How Much Does it Matter to the Users?

Years and years ago, I worked on one of the computer systems backing the State of California’s welfare system. And on my first day, the boss opened with “however you feel about welfare, politically, if this system goes down, someone can’t feed their kids, and we’re not going to let that happen.” “Will this make a kid hungry” infused everything we did.

Some software matters. Embedded pacemakers. The phone system. Fly-by-wire flight control. Banks.

And some, frankly, doesn’t. If that mobile game glitches out, well, that’s annoying, but it was almost my appointment time anyway, you know?

Everyone likes to believe that what they’re working on is very important, but they also like to be able to say “look, this isn’t aerospace” as a way to skip more testing. And thats okay, there’s a lot of software that if it goes down for an hour or two, or glitches out on launch and needs a patch, that’s not a real problem. A minor inconvenience for a few people, forgotten about the next day.

As always, it’s a spectrum. There’s plenty of stuff in the middle: does a restaurant website matter? In the grand scheme of things, not a lot, but if the hours are wrong that’ll start having an impact on the bottom line. In my experience, there’s a strong perception bias towards the middle of this spectrum.

Joel touches on this with Embedded, but mostly seems to be fairly casual about how critical the other categories are.

🌳

There are plenty of other possible spectrums, but over the last twenty years those are the ones I’ve found myself thinking about the most. And I think the combination does a reasonable job sketching out the landscape of modern software.

A lot of things in software development are basically the same regardless of what kind of software you’re developing, but not everything. Like Joel says, it’s not like Id was hiring consultants to make UML diagrams for DOOM, and so it’s important to remember where you are in the landscape before taking advice or adopting someone’s “best practices.”

As follows from the name, Software Forestry is concerned with forests—the bigger systems, with a lot of parts, that matter, with paying customers. In general, the things more on the right side of those spectrums.

As Joel said 22 years ago, we can still learn something from each other regardless of where we all stand on those spectrums, but we need to remember where we’re standing.

🌲

Next Time: What We Talk About When We Talk About Tech Debt

---

1. I don’t know, and the point is you don’t either, because they didn’t say.
2. This almost certainly wont be the last Software Forestry post to act as extended midrash on a Joel On Software post.
3. Is it web scale?

Internet Archive Loses Appeal

In an unsurprising but nevertheless depressing ruling, the Internet Archive’s has lost its appeal in the case about their digital library. (Ars, Techdirt.)

So let me get this straight; out of everything that happened in 2020, the only people facing any kinds of legal consequences are the Internet Archive, for checks notes letting people read some books?

Software Forestry 0x00: Time For More Metaphors

Software is a young field. Creating software as a mainstream profession is barely 70 years old, depending on when you start counting. Its legends are still, if just, living memory.

Young enough that it still doesn’t have much of its own language. Other than the purely technical jargon, it’s mostly borrowed words. What’s the verb for making software? Program? Develop? Write? Similarly, what’s the name for someone who makes software? Programmer? Developer? We’ve settled, more or less, on Engineer, but what we do has little in common with other branches of engineering. Even the word “computer” is borrowed; not that long ago a computer was something like an accountant, a person who computed.¹ None of this is a failing, but it is an indication of how young a field this is.

This extends to the metaphors we use to talk about the practice of creating that software. Metaphors are a cognitive shortcut, a way to borrow a different context to make the current one easier to talk about. But they can also be limiting, you can trap yourself in the boundaries of the context you borrowed.

Not that we’re short on metaphors, far from it! In keeping with the traditions of American Business, we use a lot of terms from both Sports (“Team”, “Sprint,” “Scrum”) and the Military (“Test Fire,” “Strategy vs. Tactics”). The seminal Code Complete proposed “Construction”. Knuth called it both an Art and a branch of Literature. We co-opted the term “Architecture” to talk about larger designs. In recent years, you see a lot of talk about “Craft.” “Maintenance-Oriented Programming.” For a while, I used movies. (The spec is the script! Specialized roles all coming together! But that was a very leaky abstraction.)

The wide spread of metaphors in use shows how slippery software can be, how flexible it is conceptually. We haven’t quite managed to grow a set of terms native to the field, so we keep shopping around looking for more.

I bring this up because what’s interesting about the choice of metaphors isn’t so much the direct metaphors themselves but the way they reflect the underlying philosophy of the people who chose them.

There’s two things I don’t like about a lot of those borrowed metaphors. First, most of them are Zero Sum. They assume someone is going to lose, and maybe even worse, they assume that someone is going to win. I’d be willing to entertain that that might be a useful way to think about a business as a whole in some contexts, but for a software team, that’s useless to the point of being harmful. There’s no group of people a normal software team interacts with that they can “beat”. Everyone succeeds and fails together, and they do it over the long term.

Second, most of them assume a clearly defined end state: win the game, win the battle, finish the building. Most modern software isn’t like that. It doesn’t get put in a box in Egghead anymore. Software is an ongoing effort, it’s maintained, updated, tended. Even software that’s not a service gets ongoing patches, subscriptions, DLC, the next version. There isn’t a point where it is complete, so much as ongoing refinement and care. It’s nurtured. Software is a continuous practice of maintenance and tending.

As such, I’m always looking for new metaphors; new ways of thinking about how we create, maintain, and care for software. This is something I’ve spent a lot of time stewing on over the last two decades and change. I’ve watched a lot of otherwise smart people fail to find a way to talk about what they were doing because they didn’t have the language. To quote every informercial: there has to be a better way.

What are we looking for? Situations where groups of people come together to accomplish a goal, something fundamentally creative, but with strict constraints, both physical and by convention. Where there’s competition, but not zero sum, where everyone can be successful. Most importantly, a conceptual space that assumes an ongoing effort, without a defined ending. A metaphor backed by a philosophy centered around long-term commitment and the way software projects sprawl and grow.

“Gardening” has some appeal here, but that’s a little too precious and small-scale, and doesn’t really capture the team aspect.² We want something larger, with people working together, assuming a time scale beyond a single person’s career, something focused on sustainable management.

So, I have a new software metaphor to propose: Software Forestry.

These days, software isn’t built so much as it’s grown, increment by increment. Software systems aren’t a garden, they’re a forest, filled with a whole ecosystem of different inhabitants, with different sizes, needs, uses. It’s tended by a team of practitioners who—like foresters—maintain its health and shape that growth. We’re not engineers as much as caretakers. New shoots are tended, branches pruned, fertilizer applied, older plants taken care of, the next season’s new trees planned for. But that software isn’t there for its own sake, and as foresters we’re most concerned with how that software can serve people. We’re focused on sustainability, we know now that the new software we write today is the legacy software of tomorrow. Also “Software Forestry” means the acronym is SWF, which I find hilarious. And personally, I really like trees.³ Like with trees, if we do out jobs right this stuff will still be there long after we’ve moved on.

It’s easy to get too precious about this, and let the metaphor run away with you; that’s why there were all those Black Belts and Ninjas running around a few years ago. I’m not going to start an organization to certify Software Rangers.⁴ But I think a mindset around care and tending, around seasons, around long-term stewardship, around thinking of software systems as ecosystems, is a much healthier relationship to the software industry we actually have than telling your team with all seriousness that we have to get better at blocking and tackling. We’re never going to win a game, because there’s no game to win. But we might grow a healthy forest of software, and encourage healthier foresters.

Software Forestry is a new weekly feature on Icecano. Join us on Fridays as we look at approaches to growing better software. Next Time: What kind of software forests are we growing?

---

1. My grandmother was a “civillian computer” during the war, she computed tables describing when and how to release bombs from planes to hit a target; the bombs in those tables were larger than normal, needing new tables computed late in the war. She thought nothing of this at the time, but years later realized she had been working out tables for atomic bombs. Her work went unused, she became a minister.

2. Gardening seems to pop up every couple of years; searching the web turns up quite a few abandoned swings at Software Gardening as a concept.
3. I did briefly consider “Software Arborists”, but that’s a little too narrow.
4. Although I assume the Dúnedain would make excellent programmers.

Ableist, huh?

Well! Hell of a week to decide I’m done writing about AI for a while!

For everyone playing along at home, NaNoWriMo, the nonprofit that grew up around the National Novel Writing Month challenge, has published a new policy on the use of AI, which includes this absolute jaw-dropper:

We also want to be clear in our belief that the categorical condemnation of Artificial Intelligence has classist and ableist undertones, and that questions around the use of Al tie to questions around privilege.

Really? Lack of access to AI is the only reason “the poors” haven’t been able to write books? This is the thing that’s going to improve access for the disabled? It’s so blatantly “we got a payoff, and we’re using lefty language to deflect criticism,” so disingenuine, and in such bad faith, that the only appropriate reaction is “hahahha Fuck You.”

That said, my absolute favorite response was El Sandifer on Bluesky:

"Fucking dare anyone to tell Alan Moore, to his face, that working class writers need AI in order to create."; immediately followed by "“Who the fuck said that I’ll fucking break his skull open” said William Blake in a 2024 seance."

It’s always a mistake to engage with Bad Faith garbage like this, but I did enjoy these attempts:

You Don't Need AI To Write A Novel - Aftermath

NaNoWriMo Shits The Bed On Artificial Intelligence – Chuck Wendig: Terribleminds

There’s something extra hilarious about the grifters getting to NaNoWriMo—the whole point of writing 50,000 words in a month is not that the world needs more unreadable 50k manuscripts, but that it’s an excuse to practice, you gotta write 50k bad words before you can get to 50k good ones. Using AI here is literally bringing a robot to the gym to lift weights for you.

If you’re the kind of ghoul that wants to use a robot to write a book for you, that’s one (terrible) thing, but using it to “win” a for-fun contest that exists just to provide a community of support for people trying to practice? That’s beyond despicable.

The NaNoWriMo organization has been a mess for a long time, it’s a classic volunteer-run non-profit where the founders have moved on and the replacements have been… poor. It’s been a scandal engine for a decade now, and they’ve fired everyone and brought in new people at least once? And the fix is clearly in; NoNoWiMo got a new Executive Director this year, and the one thing the “AI” “Industry” has at the moment is gobs of money.

I wonder how small the bribe was. Someone got handed a check, excuse me, a “sponsorship”, and I wonder how embarrassingly, enragingly small the number was.

I mean, any amount would be deeply disgusting, but if it was, “all you have to do is sell out the basic principles non-profit you’re now in charge of and you can live in luxury for the rest of your life” that’s still terrible but at least I would understand. But you know, you know, however much money changed hands was pathetically small.

These are the kind of people who should be hounded out of any functional civilization.

---

And then I wake up to the news that Oprah is going to host a prime time special on The AI? Ahhhh, there we go, that’s starting to smell like a Matt Damon Superbowl Ad. From the guest list—Bill Gates?—it’s pretty clearly some high-profile reputation laundering, although I’m sure Oprah got a bigger paycheck than those suckers at NaNoWriMo. I see the discourse has already decayed through a cycle of “should we pre-judge this” (spoiler: yes) and then landed on whether or not there are still “cool” uses for AI. This is such a dishonest deflection that it almost takes my breath away. Whether or not it’s “cool” is literally the least relevant point. Asbestos was pretty cool too, you know?

And Another Thing… AI Postscript

I thought I was done talking about The AI for a while after last week’s “Why is this Happening” trilogy (Part I, Part II, Part III,) but The AI wasn’t done with me just yet.

First, In one of those great coincidences, Ted Chiang has a new piece on AI in the New Yorker, Why A.I. Isn’t Going to Make Art (and yeah, that’s behind a paywall, but cough).

It’s nice to know Ted C. and I were having the same week last week! It’s the sort of piece where once you start quoting it’s hard to stop, so I’ll quote the bit everyone else has been:

The task that generative A.I. has been most successful at is lowering our expectations, both of the things we read and of ourselves when we write anything for others to read. It is a fundamentally dehumanizing technology because it treats us as less than what we are: creators and apprehenders of meaning. It reduces the amount of intention in the world.

Intention is something he locks onto here; creative work is about making lots of decisions as you do the work which can’t be replaced by a statistical average of past decisions by other people.

Second, continuing the weekend of coincidences, the kids and I went to an Anime convention this past weekend. We went to a panel on storyboarding in animation, which was fascinating, because storyboarding doesn’t quite mean the same thing in animation that it does in live-action movies.

At one point, the speaker was talking about a character in a show he had worked on named “Ai”, and specified he meant the name, not the two letters as an abbreviation, and almost reflexively spitted out “I hate A. I.!” between literally gritted teeth.

Reader, the room—which was packed—roared in approval. It was the kind of noise you’d expect to lead to a pitchfork-wielding mob heading towards the castle above town.

Outside of the more galaxy-brained corners of the wreckage of what used to be called twitter or pockets of techbros, real people in the real world hate this stuff. I can’t think of another technology from my lifetime that has ever gotten a room full of people to do that. Nothing that isn’t armed can be successful against that sort of disgust; I think we’re going to be okay.

Why is this Happening, Part III: Investing in Shares of a Stairway to Heaven

Previously: Part I, Part II.

We’ve talked a lot about “The AI” here at Icecano, mostly in terms ranging from “unflattering” to “extremely unflattering.” Which is why I’ve found myself stewing on this question the last few months: Why is this happening?

The easy answer is that, for starters, it’s a scam, a con. That goes hand-in-hand with it also being hype-fueled bubble, which is finally starting to show signs of deflating. We’re not quite at the “Matt Damon in Superbowl ads” phase yet, but I think we’re closer than not to the bubble popping.

Fad-tech bubbles are nothing new in the tech world, in recent memory we had similar grifts around the metaverse, blockchain & “web3”, “quantum”, self-driving cars. (And a whole lot of those bubbles all had the same people behind them as the current one around AI. Lots of the same datacenters full of GPUs, too!) I’m also old enough to remember similar bubbles around things like bittorrent, “4gl languages”, two or three cycles on VR, 3D TV.

This one has been different, though. There’s a viciousness to the boosters, a barely contained glee at the idea that this will put people out of work, which has been matched in intensity by the pushback. To put all that another way, when ELIZA came out, no one from MIT openly delighted at the idea that they were about to put all the therapists out of work.

But what is it about this one, though? Why did this ignite in a way that those others didn’t?

⁂

A sentiment I see a lot, as a response to AI skepticism, is to say something like “no no, this is real, it’s happening.” And the correct response to that is to say that, well, asbestos pajamas really didn’t catch fire, either. Then what happened? Just because AI is “real” it doesn’t mean it’s “good”. Those mesothelioma ads aren’t because asbestos wasn’t real.

(Again, these tend to be the same people who a few years back had a straight face when they said they were “bullish on bitcoin.”)

But I there’s another sentiment I see a lot that I think is standing behind that one: that this is the “last new tech we’ll see in our careers”. This tends to come from younger Xers & elder Millennials, folks who were just slightly too young to make it rich in the dot com boom, but old enough that they thought they were going to.

I think this one is interesting, because it illuminates part of how things have changed. From the late 70s through sometime in the 00s, new stuff showed up constantly, and more importantly, the new stuff was always better. There’s a joke from the 90s that goes like this: Two teams each developed a piece of software that didn’t run well enough on home computers. The first team spent months sweating blood, working around the clock to improve performance. The second team went and sat on a beach. Then, six months later, both teams bought new computers. And on those new machines, both systems ran great. So who did a better job? Who did a smarter job?

We all got absolutely hooked on the dopamine rush of new stuff, and it’s easy to see why; I mean, there were three extra verses of “We Didn’t Light the Fire” just in the 90s alone.

But a weird side effect is that as a culture of practitioners, we never really learned how to tell if the new thing was better than the old thing. This isn’t a new observation, Microsoft figured out to weaponize this early on as Fire And Motion. And I think this has really driven the software industry’s tendency towards “fad-oriented development,” we never built up a herd immunity to shiny new things.

A big part of this, of course, is that the press tech profoundly failed. A completely un-skeptical, overly gullible press that was infatuated shiny gadgets foisted a whole parade of con artists and scamtech on all of us, abdicating any duty they had to investigate accurately instead of just laundering press releases. The Professionally Surprised.

And for a long while, that was all okay, the occasional CueCat notwithstanding, because new stuff generally was better, and even if was only marginally better, there was often a lot of money to be made by jumping in early. Maybe not “private island” money, but at least “retire early to the foothills” money.

But then somewhere between the Dot Com Crash and the Great Recession, things slowed down. Those two events didn’t help much, but also somewhere in there “computers” plateaued at “pretty good”. Mobile kept the party going for a while, but then that slowed down too.

My Mom tells a story about being a teenager while the Beatles were around, and how she grew up in a world where every nine months pop music was reinvented, like clockwork. Then the Beatles broke up, the 70s hit, and that all stopped. And she’s pretty open about how much she misses that whole era; the heady “anything can happen” rush. I know the feeling.

If your whole identity and worldview about computers as a profession is wrapped up in diving into a Big New Thing every couple of years, it’s strange to have it settle down a little. To maintain. To have to assess. And so it’s easy to find yourself grasping for what the Next Thing is, to try and get back that feeling of the whole world constantly reinventing itself.

But missing the heyday of the PC boom isn’t the reason that AI took off. But it provides a pretty good set of excuses to cover the real reasons.

⁂

Is there a difference between “The AI” and “Robots?” I think, broadly, the answer is “no;” but they’re different lenses on the same idea. There is an interesting difference between “robot” (we built it to sit outside in the back seat of the spaceship and fix engines while getting shot at) and “the AI” (write my email for me), but that’s more about evolving stories about which is the stuff that sucks than a deep philosophical difference.

There’s a “creative” vs “mechanical” difference too. If we could build an artificial person like C-3PO I’m not sure that having it wash dishes would be the best or most appropriate possible use, but I like that as an example because, rounding to the nearest significant digit, that’s an activity no one enjoys, and as an activity it’s not exactly a hotbed of innovative new techniques. It’s the sort of chore it would be great if you could just hand off to someone. I joke this is one of the main reasons to have kids, so you can trick them into doing chores for you.

However, once “robots” went all-digital and became “the AI”, they started having access to this creative space instead of the physical-mechanical one, and the whole field backed into a moral hazard I’m not sure they noticed ahead of time.

There’s a world of difference between “better clone stamp in photoshop” and “look, we automatically made an entire website full of fake recipes to farm ad clicks”; and it turns out there’s this weird grifter class that can’t tell the difference.

Gesturing back at a century of science fiction thought experiments about robots, being able to make creative art of any kind was nearly always treated as an indicator that the robot wasn’t just “a robot.” I’ll single out Asimov’s Bicentennial Man as an early representative example—the titular robot learns how to make art, and this both causes the manufacturer to redesign future robots to prevent this happening again, and sets him on a path towards trying to be a “real person.”

We make fun of the Torment Nexus a lot, but it keeps happening—techbros keep misunderstanding the point behind the fiction they grew up on.

Unless I’m hugely misinformed, there isn’t a mass of people clamoring to wash dishes, kids don’t grow up fantasizing about a future in vacuuming. Conversely, it’s not like there’s a shortage of people who want to make a living writing, making art, doing journalism, being creative. The market is flooded with people desperate to make a living doing the fun part. So why did people who would never do that work decide that was the stuff that sucked and needed to be automated away?

⁂

So, finally: why?

I think there are several causes, all tangled.

These causes are adjacent to but not the same as the root causes of the greater enshittification—excuse me, “Platform Decay”—of the web. Nor are we talking about the largely orthogonal reasons why Facebook is full of old people being fooled by obvious AI glop. We’re interested in why the people making these AI tools are making them. Why they decided that this was the stuff that sucked.

First, we have this weird cultural stew where creative jobs are “desired” but not “desirable”. There’s a lot of cultural cachet around being a “creator” or having a “creative” jobs, but not a lot of respect for the people actually doing them. So you get the thing where people oppose the writer’s strike because they “need” a steady supply of TV, but the people who make it don’t deserve a living wage.

Graeber has a whole bit adjacent to this in Bullshit Jobs. Quoting the originating essay:

It's even clearer in the US, where Republicans have had remarkable success mobilizing resentment against school teachers, or auto workers (and not, significantly, against the school administrators or auto industry managers who actually cause the problems) for their supposedly bloated wages and benefits. It's as if they are being told ‘but you get to teach children! Or make cars! You get to have real jobs! And on top of that you have the nerve to also expect middle-class pensions and health care?’

“I made this” has cultural power. “I wrote a book,” “I made a movie,” are the sort of things you can say at a party that get people to perk up; “oh really? Tell me more!”

Add to this thirty-plus years of pressure to restructure public education around “STEM”, because those are the “real” and “valuable” skills that lead to “good jobs”, as if the only point of education was as a job training program. A very narrow job training program, because again, we need those TV shows but don’t care to support new people learning how to make them.

There’s always a class of people who think they should be able to buy anything; any skill someone else has acquired is something they should be able to purchase. This feels like a place I could put several paragraphs that use the word “neoliberalism” and then quote from Ayn Rand, The Incredibles, or Led Zeppelin lyrics depending on the vibe I was going for, but instead I’m just going to say “you know, the kind of people who only bought the Cliffs Notes, never the real book,” and trust you know what I mean. The kind of people who never learned the difference between “productivity hacks” and “cheating”.

The sort of people who only interact with books as a source of isolated nuggets of information, the kind of people who look at a pile of books and say something like “I wish I had access to all that information,” instead of “I want to read those.”

People who think money should count at least as much, if not more than, social skills or talent.

On top of all that, we have the financializtion of everything. Hobbies for their own sake are not acceptable, everything has to be a side hustle. How can I use this to make money? Why is this worth doing if I can’t do it well enough to sell it? Is there a bootcamp? A video tutorial? How fast can I start making money at this?

Finally, and critically, I think there’s a large mass of people working in software that don’t like their jobs and aren’t that great at them. I can’t speak for other industries first hand, but the tech world is full of folks who really don’t like their jobs, but they really like the money and being able to pretend they’re the masters of the universe.

All things considered, “making computers do things” is a pretty great gig. In the world of Professional Careers, software sits at the sweet spot of “amount you actually have to know & how much school you really need” vs “how much you get paid”.

I’ve said many times that I feel very fortunate that the thing I got super interested in when I was twelve happened to turn into a fully functional career when I hit my twenties. Not everyone gets that! And more importantly, there are a lot of people making those computers do things who didn’t get super interested in computers when they were twelve, because the thing they got super interested in doesn’t pay for a mortgage.

Look, if you need a good job, and maybe aren’t really interested in anything specific, or at least in anything that people will pay for, “computers”—or computer-adjacent—is a pretty sweet direction for your parents to point you. I’ve worked with more of these than I can count—developers, designers, architects, product people, project managers, middle managers—and most of them are perfectly fine people, doing a job they’re a little bored by, and then they go home and do something that they can actually self-actualize about. And I suspect this is true for a lot of “sit down inside email jobs,” that there’s a large mass of people who, in a just universe, their job would be “beach” or “guitar” or “games”, but instead they gotta help knock out front-end web code for a mid-list insurance company. Probably, most careers are like that, there’s the one accountant that loves it, and then a couple other guys counting down the hours until their band’s next unpaid gig.

But one of the things that makes computers stand out is that those accountants all had to get certified. The computer guys just needed a bootcamp and a couple weekends worth of video tutorials, and suddenly they get to put “Engineer” on their resume.

And let’s be honest: software should be creative, usually is marketed as such, but frequently isn’t. We like to talk about software development as if it’s nothing but innovation and “putting a dent in the universe”, but the real day-to-day is pulling another underwritten story off the backlog that claims to be easy but is going to take a whole week to write one more DTO, or web UI widget, or RESTful API that’s almost, but not quite, entirely unlike the last dozen of those. Another user-submitted bug caused by someone doing something stupid that the code that got written badly and shipped early couldn’t handle. Another change to government regulations that’s going to cause a remodel of the guts of this thing, which somehow manages to be a surprise despite the fact the law was passed before anyone in this meeting even started working here.

They don’t have time to learn how that regulation works, or why it changed, or how the data objects were supposed to happen, or what the right way to do that UI widget is—the story is only three points, get it out the door or our velocity will slip!—so they find someting they can copy, slap something together, write a test that passes, ship it. Move on to the next. Peel another one off the backlog. Keep that going. Forever.

And that also leads to this weird thing software has where everyone is just kind of bluffing everyone all the time, or at least until they can go look something up on stack overflow. No one really understands anything, just gotta keep the feature factory humming.

The people who actually like this stuff, who got into it because they liked making compteurs do things for their own sake keep finding ways to make it fun, or at least different. “Continuous Improvement,” we call it. Or, you know, they move on, leaving behind all those people whose twelve-year old selves would be horrified.

But then there’s the group that’s in the center of the Venn Diagram of everything above. All this mixes together, and in a certain kind of reduced-empathy individual, manifests as a fundamental disbelief in craft as a concept. Deep down, they really don’t believe expertise exists. That “expertise” and “bias” are synonyms. They look at people who are “good” at their jobs, who seem “satisfied” and are jealous of how well that person is executing the con.

Whatever they were into at twelve didn’t turn into a career, and they learned the wrong lesson from that. The kind of people who were in a band as a teenager and then spent the years since as a management consultant, and think the only problem with that is that they ever wanted to be in a band, instead of being mad that society has more open positions for management consultants than bass players.

They know which is the stuff that sucks: everything. None of this is the fun part; the fun part doesn’t even exist; that was a lie they believed as a kid. So they keep trying to build things where they don’t have to do their jobs anymore but still get paid gobs of money.

They dislike their jobs so much, they can’t believe anyone else likes theirs. They don’t believe expertise or skill is real, because they have none. They think everything is a con because thats what they do. Anything you can’t just buy must be a trick of some kind.

(Yeah, the trick is called “practice”.)

These aren’t people who think that critically about their own field, which is another thing that happens when you value STEM over everything else, and forget to teach people ethics and critical thinking.

Really, all they want to be are “Idea Guys”, tossing off half-baked concepts and surrounded by people they don’t have to respect and who wont talk back, who will figure out how to make a functional version of their ill-formed ramblings. That they can take credit for.

And this gets to the heart of whats so evil about the current crop of AI.

These aren’t tools built by the people who do the work to automate the boring parts of their own work; these are built by people who don’t value creative work at all and want to be rid of it.

As a point of comparison, the iPod was clearly made by people who listened to a lot of music and wanted a better way to do so. Apple has always been unique in the tech space in that it works more like a consumer electronics company, the vast majority of it’s products are clearly made by people who would themselves be an enthusiastic customer. In this field we talk about “eating your own dog-food” a lot, but if you’re writing a claims processing system for an insurance company, there’s only so far you can go. Making a better digital music player? That lets you think different.

But no: AI is all being built by people who don’t create, who resent having to create, who resent having to hire people who can create. Beyond even “I should be able to buy expertise” and into “I value this so little that I don’t even recognize this as a real skill”.

One of the first things these people tried to automate away was writing code—their own jobs. These people respect skill, expertise, craft so little that they don’t even respect their own. They dislike their jobs so much, and respect their own skills so little, that they can’t imagine that someone might not feel that way about their own.

A common pattern has been how surprised the techbros have been at the pushback. One of the funnier (in a laugh so you don’t cry way) sideshows is the way the techbros keep going “look, you don’t have to write anymore!” and every writer everywhere is all “ummmmm, I write because I like it, why would I want to stop” and then it just cuts back and forth between the two groups saying “what?” louder and angrier.

We’re really starting to pay for the fact that our civilization spent 20-plus years shoving kids that didn’t like programming into the career because it paid well and you could do it sitting down inside and didn’t have to be that great at it.

What future are they building for themselves? What future do they expect to live in, with this bold AI-powered utopia? Some vague middle-management “Idea Guy” economy, with the worst people in the world summoning books and art and movies out of thin air for no one to read or look at or watch, because everyone else is doing the same thing? A web full of AI slop made by and for robots trying to trick each other? Meanwhile the dishes are piling up? That’s the utopia?

I’m not sure they even know what they want, they just want to stop doing the stuff that sucks.

And I think that’s our way out of this.

⁂

What do we do?

For starters, AI Companies need to be regulated, preferably out of existence. There’s a flavor of libertarian-leaning engineer that likes to say things like “code is law,” but actually, turns out “law” is law. There’s whole swathes of this that we as a civilization should have no tolerance for; maybe not to a full Butlerian Jihad, but at least enough to send deepfakes back to the Abyss. We dealt with CFCs and asbestos, we can deal with this.

Education needs to be less STEM-focused. We need to carve out more career paths (not “jobs”, not “gigs”, “careers”) that have the benefits of tech but aren’t tech. And we need to furiously defend and expand spaces for creative work to flourish. And for that work to get paid.

But those are broad, society-wide changes. But what can those of us in the tech world actually do? How can we help solve these problems in our own little corners? We can we go into work tomorrow and actually do?

It’s on all of us in the tech world to make sure there’s less of the stuff that sucks.

We can’t do much about the lack of jobs for dance majors, but we can help make sure those people don’t stop believing in skill as a concept. Instead of assuming what we think sucks is what everyone thinks sucks, is there a way to make it not suck? Is there a way to find a person who doesn’t think it sucks? (And no, I don’t mean “Uber for writing my emails”) We gotta invite people in and make sure they see the fun part.

The actual practice of software has become deeply dehumanizing. None of what I just spent a week describing is the result of healthy people working in a field they enjoy, doing work they value. This is the challenge we have before us, how can we change course so that the tech industry doesn’t breed this. Those of us that got lucky at twelve need to find new ways to bring along the people who didn’t.

⁂

With that in mind, next Friday on Icecano we start a new series on growing better software.

---

Several people provided invaluable feedback on earlier iterations of this material; you all know who you are and thank you.

And as a final note, I’d like to personally apologize to the one person who I know for sure clicked Open in New Tab on every single link. Sorry man, they’re good tabs!

Why is this Happening, Part II: Letting Computers Do The Fun Part

Previously: Part I

Let’s leave the Stuff that Sucks aside for the moment, and ask a different question. Which Part is the Fun Part? What are we going to do with this time the robots have freed up for us?

It’s easy to get wrapped up in pointing at the parts of living that suck; especially when fantasizing about assigning work to C-3PO’s cousin. And it’s easy to spiral to a place where you just start waving your hands around at everything.

But even Bertie Wooster had things he enjoyed, that he occasionally got paid for, rather than let Jeeves work his jaw for him.

So it’s worth recalibrating for a moment: which are the fun parts?

As aggravating as it can be at times, I do actually like making computers do things. I like programming, I like designing software, I like building systems. I like finding clever solutions to problems. I got into this career on purpose. If it was fun all the time they wouldn’t have to call it “work”, but it’s fun a whole lot of the time.

I like writing (obviously.) For me, that dovetails pretty nicely with liking to design software; I’m generally the guy who ends up writing specs or design docs. It’s fun! I owned the customer-facing documentation several jobs back. It was fun!

I like to draw! I’m not great at it, but I’m also not trying to make a living out of it. I think having hobbies you enjoy but aren’t great at is a good thing. Not every skill needs to have a direct line to a career or a side hustle. Draw goofy robots to make your kids laugh! You don’t need to have to figure out a the monetization strategy.

In my “outside of work” life I think I know more writers and artists than programmers. For all of them, the work itself—the writing, the drawing, the music, making the movie—is the fun part. The parts they don’t like so well is the “figuring out how to get paid” part, or the dealing with printers part, or the weird contracts part. The hustle. Or, you know, the doing dishes, laundry, and vacuuming part. The “chores” part.

So every time I see a new “AI tool” release that writes text or generates images or makes video, I always as the same question:

Why would I let the computer do the fun part?

The writing is the fun part! The drawing pictures is the fun part! Writing the computer programs are the fun part! Why, why, are they trying to tell us that those are the parts that suck?

Why are the techbros trying to automate away the work people want to do?

It’s fun, and I worked hard to get good at it! Now they want me to let a robot do it?

⁂

Generative AI only seems impressive if you’ve never successfully created anything. Part of what makes “AI art” so enragingly radicalizing is the sight of someone whose never tried to create something before, never studied, never practiced, never put the time in, never really even thought about it, joylessly showing off their terrible AI slop they made and demanding to be treated as if they made it themselves, not that they used a tool built on the fruits of a million million stolen works.

Inspiration and plagiarism are not the same thing, the same way that “building a statistical model of word order probability from stuff we downloaded from the web” is not the same as “learning”. A plagiarism machine is not an artist.

But no, the really enraging part is watching these people show off this garbage realizing that these people can’t tell the difference. And AI art seems to be getting worse, AI pictures are getting easier spot, not harder, because of course it is, because the people making the systems don’t know what good is. And the culture is following: “it looks like AI made it” has become the exact opposite of a compliment. AI-generated glop is seen as tacky, low quality. And more importantly, seen as cheap, made by someone who wasn’t willing to spend any money on the real thing. Trying to pass off Krusty Burgers as their own cooking.

These are people with absolutely no taste, and I don’t mean people who don’t have a favorite Kurosawa film, I mean people who order a $50 steak well done and then drown it in A1 sauce. The kind of people who, deep down, don’t believe “good” is real. That it’s all just “marketing.”

⁂

The act of creation is inherently valuable; creation is an act that changes the creator as much as anyone. Writing things down isn’t just documentation, it’s a process that allows and enables the writer to discover what they think, explore how they actually feel.

“Having AI write that for you is like having a robot lift weights for you.”

AI writing is deeply dehumanizing, to both the person who prompted it and to the reader. There is so much weird stuff to unpack from someone saying, in what appears to be total sincerity, that they used AI to write a book. That the part they thought sucked was the fun part, the writing, and left their time free for… what? Marketing? Uploading metadata to Amazon? If you don’t want to write, why do you want people to call you a writer?

Why on earth would I want to read something the author couldn’t be bothered to write? Do these ghouls really just want the social credit for being “an artist”? Who are they trying to impress, what new parties do they think they’re going to get into because they have a self-published AI-written book with their name on it? Talk about participation trophies.

⁂

All the people I know in real life or follow on the feeds who use computers to do their thing but don’t consider themselves “computer people” have reacted with a strong and consistant full-body disgust. Personally, compared to all those past bubbles, this is the first tech I’ve ever encountered where my reaction was complete revulsion.

Meanwhile, many (not all) of the “computer people” in my orbit tend to be at-least AI curious, lots of hedging like “it’s useful in some cases” or “it’s inevitable” or full-blown enthusiasm.

One side, “absolutely not”, the other side, “well, mayyybe?” As a point of reference, this was the exact breakdown of how these same people reacted to blockchain and bitcoin.

One group looks at the other and sees people musing about if the face-eating leopard has some good points. The other group looks at the first and sees a bunch of neo-luddites. Of course, the correct reaction to that is “you’re absolutely correct, but not for the reasons you think.”

⁂

There’s a Douglas Adams bit that gets quoted a lot lately, which was printed in Salmon of Doubt but I think was around before that:

I’ve come up with a set of rules that describe our reactions to technologies:

1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.

2. Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.

3. Anything invented after you’re thirty-five is against the natural order of things.

The better-read AI-grifters keep pointing at rule 3. But I keep thinking of the bit from Dirk Gently’s Detective Agency about the Electric Monk:

The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe.

So, what are the people who own the Monks doing, then?

⁂

Let’s speak plainly for a moment—the tech industry has always had a certain…. ethical flexibility. The “things” in “move fast and break things” wasn’t talking about furniture or fancy vases, this isn’t just playing baseball inside the house. And this has been true for a long time, the Open Letter to Hobbyists was basically Gates complaining that other people’s theft was undermining the con he was running.

We all liked to pretend “disruption” was about finding “market inefficiencies” or whatever, but mostly what that meant was moving in to a market where the incumbents were regulated and labor had legal protection and finding a way to do business there while ignoring the rules. Only a psychopath thinks “having to pay employees” is an “inefficiency.”

Vast chunks of what it takes to make generative AI possible are already illegal or at least highly unethical. The Internet has always been governed by a sort of combination of gentleman’s agreements and pirate codes, and in the hunger for new training data, the AI companies have sucked up everything, copyright, licensing, and good neighborship be damned.

There’s some half-hearted attempts to combat AI via arguments that it violates copyright or open source licensing or other legal approach. And more power to them! Personally, I’m not really interested in the argument the AI training data violates contract law, because I care more about the fact that it’s deeply immoral. See that Vonnegut line about “those who devised means of getting paid enormously for committing crimes against which no laws had been passed.” Much like I think people who drive too fast in front of schools should get a ticket, sure, but I’m not opposed to that action because it was illegal, but because it was dangerous to the kids.

It’s been pointed out more than once that AI breaks the deal behind webcrawlers and search—search engines are allowed to suck up everyone’s content in exchange for sending traffic their way. But AI just takes and regurgitates, without sharing the traffic, or even the credit. It’s the AI Search Doomsday Cult. Even Uber didn’t try to put car manufacturers out of business.

⁂

But beyond all that, making things is fun! Making things for other people is fun! It’s about making a connection between people, not about formal correctness or commercial viability. And then you see those terrible google fan letter ads at the olympics, or see people crowing that they used AI to generate a kids book for their children, and you wonder, how can these people have so little regard for their audience that they don’t want to make the connection themselves? That they’d rather give their kids something a jumped-up spreadsheet full of stolen words barfed out instead of something they made themselves? Why pass on the fun part, just so you can take credit for something thoughtless and tacky? The AI ads want you to believe that you need their help to find “the right word”; what thay don’t tell you is that no you don’t, what you need to do is have fun finding your word.

⁂

Robots turned out to be hard. Actually, properly hard. You can read these papers by computer researchers in the fifties where they’re pretty sure Threepio-style robot butlers are only 20 years away, which seems laughable now. Robots are the kind of hard where the more we learn the harder they seem.

As an example: Doctor Who in the early 80s added a robot character who was played by the prototype of an actual robot. This went about as poorly as you might imagine. That’s impossible to imagine now, no producer would risk their production on a homemade robot today, matter how impressive the demo was. You want a thing that looks like Threepio walking around and talking with a voice like a Transformer? Put a guy in a suit. Actors are much easier to work with. Even though they have a union.

Similarly, “General AI” in the HAL/KITT/Threepio sense has been permanently 20 years in the future for at least 70 years now. The AI class I took in the 90s was essentially a survey of things that hadn’t worked, and ended with a kind of shrug and “maybe another 20?”

Humans are really, really good at seeing faces in things, and finding patterns that aren’t there. Any halfway decent professional programmer can whip up an ELIZA clone in an afternoon, and even knowing how the trick works it “feels” smarter than it is. A lot of AI research projects are like that, a sleight-of-hand trick that depends on doing a lot of math quickly and on the human capacity to anthropomorphize. And then the self-described brightest minds of our generation fail the mirror test over and over.

Actually building a thing that can “think”? Increasingly seems impossible.

You know what’s easy, though, comparatively speaking? Building a statistical model of all the text you can pull off the web.

On Friday: conclusions, such as they are.

Why is this Happening, Part I: The Stuff That Sucks

When I was a kid, I had this book called The Star Wars Book of Robots. It was a classic early-80s kids pop-science book; kids are into robots, so let’s have a book talking about what kinds of robots existed at the time, and then what kinds of robots might exist in the future. At the time, Star Wars was the spoonful of sugar to help education go down, so every page talked about a different kind of robot, and then the illustration was a painting of that kind of robot going about its day while C-3PO, R2-D2, and occasionally someone in 1970s leisureware looked on. So you’d have one of those car factory robot arms putting a sedan together while the droids stood off to the side with a sort of “when is Uncle Larry finally going to retire?” energy.

The image from that book that has stuck with me for four decades is the one at the top of this page: Threepio, trying to do the dishes while vacuuming, and having the situation go full slapstick. (As a kid, I was really worried that the soap suds were going to get into his bare midriff there and cause electrical damage, which should be all you need to know to guess exactly what kind of kid I was at 6.)

Nearly all the examples in the book were of some kind of physical labor; delivering mail, welding cars together, doing the dishes, going to physically hostile places. And at the time, this was the standard pop-culture job for robots “in the future”, that robots and robotic automation were fundamentally physical, and were about relieving humans from mechanical labor.

The message is clear: in the not to distant future we’re all going to have some kind of robotic butler or maid or handyman around the house, and that robot is going to do all the Stuff That Sucks. Dishes, chores, laundry, assorted car assembly, whatever it is you don’t want to do, the robot will handle for you.

I’ve been thinking about this a lot over the last year and change since “Generative AI” became a phrase we were all forced to learn. And what’s interesting to me is the way that the sales pitch has evolved around which is the stuff that sucks.

Robots, as a storytelling construct, have always been a thematically rich metaphor in this regard, and provide an interesting social diagnostic. You can tell a lot about what a society thinks is “the stuff that sucks” by looking at both what the robots and the people around them are doing. The work that brought us the word “robot” itself represented them as artificially constructed laborers who revolted against their creators.

Asimov’s body of work, which was the first to treat robots as something industrial and coined the term “robotics” mostly represented them as doing manual labor in places too dangerous for humans while the humans sat around doing science or supervision. But Asimov’s robots also were always shown to be smarter and more individualistic than the humans believed, and generally found a way to do what they wanted to do, regardless of the restrictions from the “Laws of Robotics.”

Even in Star Wars, which buries the political content low in the mix, it’s the droids where the dark satire from THX-1138 pokes through; robots are there as a permanent servant class doing dangerous work either on the outside of spaceships or translating for crime bosses, are the only group shown to be discriminated against, and have otherwise unambiguous “good guys” ordering mind wipes of, despite consistently being some of the smartest and most capable characters.

And then, you know, Blade Runner.

There’s a lot of social anxiety wrapped up in all this. Post-industrial revolution, the expanding middle classes wanted the same kinds of servants and “domestic staff” as the upper classes had. Wouldn’t it be nice to have a butler, a valet, some “staff?” That you didn’t have to worry about?

This is the era of Jeeves & Wooster, and who wouldn’t want a “gentleman’s gentleman” to do the work around the house, make you a hangover cure, drive the car, get you out of scrapes, all while you frittered your time away with idiot friends?

(Of course, I’m sure it’s a total coincidence this is also the period where the Marxists & Socialist thinkers really got going.)

But that stayed asperational, rather than possible, and especially post-World War II, the culture landed on sending women back home and depending on the stay-at-home mom handle “all that.”

There’s a lot of “robot butlers” in mid-century fiction, because how nice would it be if you could just go to the store and buy that robot from The Jetsons, free from any guilt? There’s a lot to unpack there, but that desire for a guilt-free servant class was, and is, persistant in fiction.

Somewhere along the lines, this changes, and robots stop being manual labor or the domestic staff, and start being secretaries, executive assistants. For example, by the time Star Trek: The Next Generation rolls around in the mid-80s, part of the fully automated luxury space communism of the Federation is that the Enterprise computer is basically the perfect secretary—making calls, taking dictation, and doing research. Even by the then it was clear that there was a whole lot of “stuff to know”, and so robots find themselves acting as research assistants. Partly, this is a narrative accelerant—having the Shakespearian actor able to ask thin air for the next plot point helps move things along pretty fast—but the anxiety about information overload was there, even then. Imagine if you could just ask somebody to look it up for you! (Star Trek as a whole is an endless Torment Nexus factory, but that’s a whole other story.)

I’ve been reading a book about the history of keyboards, and one of the more intersting side stories is the way “typing” has interacted with gender roles over the last century. For most of the 1900s, “typing” was a woman’s job, and men, who were of course the bosses, didn’t have time for that sort of tediousness. They’re Idea Guys, and the stuff that sucks is wrestling with an actual typewriter to write them down.

So, they would either handwrite things they needed typed and send it down to the “typing pool”, or dictate to a secretary, who would type it up. Typing becomes a viable job out of the house for younger or unmarried women, albeit one without an actual career path.

This arrangement lasted well into the 80s, and up until then the only men who typed themselves were either writers or nerds. Then computers happened, PCs landed on men’s desks, and it turns out the only thing more powerful than sexism was the desire to cut costs, so men found themselves typing their own emails. (Although, this transition spawns the most unwittingly enlightening quote in the whole book, where someone who was an executive at the time of the transition says it didn’t really matter, because “Feminism ruined everything fun about having a secretary”. pikachu shocked face dot gif)

But we have a pattern; work that would have been done by servants gets handed off to women, and then back to men, and then fiction starts showing up fantasizing about giving that work to a robot, who won’t complain, or have an opinion—or start a union.

Meanwhile, in parallel with all this “chat bots” have been cooking along for as long as there have been computers. Typing at a computer and getting a human-like response was an obvious interface, and spawned a whole set of thought similar but adjacent to all those physical robots. ELIZA emerged almost as soon as computers were able to support such a thing. The Turing test assumes a chat interface. “Software Agents” become a viable area of research. The Infocom text adventure parser came out of the MIT AI lab. What if your secretary was just a page of text on your screen?

One of the ways that thread evolved emerged as LLMs and “Generative AI”. And thanks to the amount of VC being poured in, we get the last couple of years of AI slop. And also a hype cycle that says that any tech company that doesn’t go all-in on “the AI” is going to be left in the dust. It’s the Next Big Thing!

Flash forward to Apple’s Worldwide Developer Conference earlier this summer. The Discourse going into WWDC was that Apple was “behind on AI” and needed to catch up to the industry, although does it really count as behind if all your competitors are up over their skis? And so far AI has been extremely unprofitable, and if anything, Apple is a company that only ships products it knows it can make money on.

The result was that they rolled out the most comprehensive vision of how a Gen AI–powered product suite looks here in 2024. In many ways, “Apple Intelligence” was Apple doing what they do best—namely, doing their market research via letting their erstwhile competitors skid into a ditch, and then slide in with a full Second Mover Advantage by saying “so, now do you want something that works?”

They’re very, very good at identifying The Stuff That Sucks, and announcing that they have a solution. So what stuff was it? Writing text, sending pictures, communicating with other people. All done by a faceless, neutral, “assistant,” who you didn’t have to engage with like they were a person, just a fancy piece of software. Computer! Tea, Earl Gray! Hot!

I’m writing about a marketing event from months ago because watching their giant infomercial was where something clicked for me. They spent an hour talking about speed, “look how much faster you can do stuff!” “You don’t have to write your own text, draw your own pictures, send your own emails, interact directly with anyone!”

Left unsaid was what you were saving all that time for. Critically, they didn’t annouce they were going to a 4-day work week or 6-hour days, all this AI was so people could do more “real work”. Except that the “stuff that sucks” was… that work? What’s the vision of what we’ll be doing when we’ve handed off all this stuff that sucks?

Who is building this stuff? What future do they expect to live in, with this bold AI-powered economy? What are we saving all this time for? What future do these people want? Why are these the things they have decided suck?

I was struck, not for the first time, by what a weird dystopia we find ourselves in: “we gutted public education and non-STEM subjects like writing and drawing, and everyone is so overworked they need a secretary but can’t afford one, so here’s a robot!”

To sharpen the point: why in the fuck am I here doing the dishes myself while a bunch of techbros raise billions of dollars to automate the art and poetry? What happened to Threepio up there? Why is this the AI that’s happening?

On Wednesday, we start kicking over rocks to find an answer...

Feature Request: I Already Know That Part, Siri

I live pretty close to a major interstate highway. If you stand in the right place in my backyard, you can see the trucks! But, thanks to the turn-of-the-century suburb I live in, it’s at least 5 “turns” to get from my house to the freeway. I also live in one of those cities that’s a major freeway confluence, which means I’m another 2 or 3 “turns” away from at least 5 different numbered freeways?

So of course, when I need directions from Apple Maps (or any other nav system,) Siri very patiently explains how to get from my house to the freeway, which, yes Siri, I know that part.

I wish there was a way to mark an area on the the map as “look, I grew up here, I got this.” I wish, when I’m driving out to the mountains or whatever, Siri would start with “Get on I-5 south, I’ll be back with you in half an hour.” I want to be able to tell it “no, look, I know all these turns, I just want you to tell me when we’re at the destination so I don’t drive past the weird driveway again.” That’s an Apple Intelligence feature I’d be impressed by.

Apple vs Games

Apple Arcade is in the news again, for not great reasons; as always, Tsai has the roundup, the but the short, short version is that Arcade is going exactly as well as all of Apple’s other video game–related efforts have gone for the last “since forever.”

My first take was that games might be the most notable place Apple’s “one guy at the top” structure falls down. Apple’s greatest strength and greatest weakness has always been that the whole company is laser focused on whatever the guy in charge cares about, and not much of anything else. Currently, that means that Apple’s priorities are, in no particular order, privacy, health, thin devices, operational efficiency, and, I guess, becoming “the new HBO.” Games aren’t anywhere near that list, and never have been. I understand the desire to keep everything flowing through one central point, and not to have siloed-off business units or what have you. On the other hand Bill Gates wasn’t a gamer either, but he knew to hire someone to be in charge of X-everything and leave them alone.

But then I remembered AppleTV+. Somehow, in a very short amount of time, Apple figured out how to be a production company, and made Ted Lasso, a new Fraggle Rock, some new Peanuts, and knocked out a Werner Herzog documentary for good measure. I refuse to believe that happened because Tim Apple was signing off on every production decision or script; they found the right people and enabled them correctly.

At this point, there’s just no excuse why AppleTV has something like Ted Lasso, and Apple Arcade doesn’t. There’s obvious questions like “why did I play Untitled Goose Game on my Switch instead of my Mac” and “why did they blow acquiring Bungie twice”. Why isn’t the Mac the premier game platform? Why? What’s the malfunction?