Software Forestry 0x04: Library Upgrade Week
Here at Software Forestry we do occasionally try to solve problems instead of just turning them into lists of smaller problems, so we’re goona do a little mood pivot here and start talking about how to manage some of those forms of overgrowth we talked about last time.
First up: let me tell you the good news about Library Upgrade Week.
Just about any decently sized software system uses a variety of third party libraries. And why wouldn’t you? The multitude of high-quality libraries and frameworks out there is probably the best outcome of both the Open Source movement and Object-Oriented Software design. The specific mechanics vary between languages and their practitioner’s cultures, but the upshot is that very rarely does anyone build everything from scratch. There’s no need to go all historical reenactment and write your own XML parser.
Generally, people keep those libraries pinned and stay on one fixed version, rather than schlepping in a new version every-time an update happens. This is a good thing! Change is risk, and risk should be taken on intentionally. The downside is that those libraries keep moving forward, the version you’re using slips out of date, and now you have a bunch of Overgrowth. And so that means you need to upgrade them.
The upshot of all that is on a semi-regular basis, we all need to slurp in a bunch of new code no-one on the payroll wrote, and don’t really know how to test. Un-Reviewed Code Is Tech Debt, and one of the mantras of writing good tests is “don’t test the framework”, so this is always a little iffy.
It’s incredibly easy to just keep letting those weeds grow a little longer. “The new version doesn’t have anything we need”, “there’s no bugs”, “if it ain’t broke don’t fix it”, and so on. They always take too long, don't usually deliver immediate gratification, and are hard to schedule. It’s no fun, and no one likes to do it.
The trick is to turn it into a party.
It works like this: set aside the last week of the quarter to concentrate on 3rd party library upgrades. Regardless of what your formal planning cycle or process is, most businesses tends to operate on quarters, and there’s usually a little dead time at the end of the quarter you can repurpose.
The Process:
Form squads. Each squad is a group of like-minded individuals focused on a single 3rd party library or framework. Squads are encouraged to be cross-team. Each squad will focus on updating that 3rd party library in all applications or places where it is used. The intent is to make this a group event, where people can help each other out. Participation is not mandatory.
Share squad membership and goals ahead of time. Leadership should reserve the right to veto libraries as “too scary” or “not scary enough”. Libraries with a high severity alerts or known CVE are good candidates.
That week, each squad self organizes and works as a group through any issues caused by the upgrade. Other than major outages or incidents, squad members should be excused from other “run the business” type work for that week; or rather, the library upgrades are “the business.” Have fun!
On that Friday hold the Library Upgrade Week Show-n-Tell. Every team should demo what they did, how they did it, and what it took to pull it off. Tell war stories, hold a happy hour, swap jokes. If a squad doesn’t finish that's okay! The expectation is that they’ll have learned a lot about what it'll take to finish, and that work will be captured in the relevant team’s todo lists. If you’re in a process with short develop-deploy increments (like sprints) you can make the library upgrade(s) a release on its own. Ideally you already have a way to sign off a release as not containing regressions, and so a short release with just a library upgrade is a great way to make sure you didn’t knock some dominos over.
But wait! There's more! All participants will vote on awards to give to squads, for things like:
- Error message with least hits on Stack Overflow
- Largest version number jump
- Most lines changed
- Fewest lines changed
- Best team name
- Best presentation
Go nuts! Have a great time!
Yes, it’s a little silly, but that’s the point. I’ve deployed a version of this at a couple of jobs now, and it’s remarkable how effective it is. The first couple of cycles people hit the “easy” ones—uprev the logging library or a JSON parser or something. But then, once people know that Library Upgrade Week is coming, they start thinking about the harder stuff, and you start getting people saying they want to take a swing at the main framework, or the main language version, or something else load-bearing. It’s remarkable how much progress two or three people can make on a problem that looks unsolvable when they have an uninterrupted week to chew on it. (If you genuinely can’t spare a handful of folks to do some weeding four weeks out the the year, that’s a much larger problem than out of date libraries, and you should go solve that problem first. Like, right now.)
There’s an instinct to take the core idea to schedule this kind of maintenance for a few times a year, but leave off the part where it’s a party. This is a mistake. This is work people want to do even less than their usual work; the trick is to make everthing around it fun.
We’re Foresters, and both we and the Forest are here long term. The long term health of both depends on the care of the Forest being something that the Foresters enjoy, and it’s okay to stack that deck in your favor.