Hello there! Hope your weekend was good.
Today I'd like to talk about code that evolved organically, and code that was planned, or had a vision, or was structured according to some explicit rules.
Really I think I want to talk about having a vision for a system.
Back at a certain music company, I found myself working with a rather large PHP system. Scatman was used by employees to navigate the content, debug systems that would push content into the platform, and in particular by specific teams in charge of our relationship with providers.
The system worked, whatever anyone else may tell you, and it absolutely brought a lot of value to the table. Without it many functions of the company wouldn't have been able to do their job.
But this system also had one big problem: Masterchief was gone, and behind he left a beautiful garden of idiosyncracies that no one else fully understood.
It took a team of 3 (grown to 8) and over a year to tame this beast, understand its original vision (or lack of one), and sketch out what this system should look like today.
I've had to work on several systems like this over the years, and not one of them seemed to have any sort of upfront vision.
There's an old quote from John Gall that goes
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.
I haven't read John Gall's "The Systems Bible", and odds are that I won't anytime soon. But as far as quotes go, I don't really know if complex here means "hard to understand" or just "involving multiple connected parts", or maybe "incomplete specification of the system's behaviors". And the truth is that quotes become memes, not entire books. So lets run with the quote.
The only definition of complex that I can agree with here is "hard to understand". I don't believe it matters how many interconnected pieces your design has, if you can't understand the whole, you will to make second-order mistakes.
I mean, you can understand a small part of the whole, and make a change there. All good in that small part. But that change now made other almost seemingly unrelated parts of the system break. Second order effects.
But if they whole is laid out in a way that you can understand it, it may not be easy to work with it, but it'll certainly be possible.
Anyway. We set out to redesign this system by breaking apart the separate things it did. Primarily from a capabilities perspective, but also from the POV of the user flows.
By Capabilities I mean the things that the system is supposed to be able to do: uploading content, editing content, searching, liking, sharing, blocking.
By User Flows I mean the ways our users were using the system to get their job done: certain flows were related to monitoring content for people trying to abuse the system, other flows were for developers doing debugging stuff.
The resulting design was completely artificial. And kind of beautiful:
No big bang. Frog boiling. "Change the Engines Midflight" thinking.
So what was wrong with this?
Nothing, at least not from a purely technical perspective. We have the old organic design, grown like a garden for over 7 years. We have the new artificial design, where we brought all of our hard won wisdom in building systems and drew some pretty cool blueprints that would make our moms proud.
But the old stuff prevailed, and it was so entrenched that to this day it is still in heavy use. In fact, they'll likely never get rid of it entirely.
So why did we do it then? Because it was The Right Thing ™ to do. Or so I thought back then.
In general it is hella easier to throw tech debt under the rug and provide business value than it is to make a good case for an architectural vision that could deliver better value going forward.
And that's the thing. We value technologies that give us productivity today because business value that is tangible today is perceived as of higher worth.
Phoenix lets you do this. PHP lets you do this. Rails lets you do this. You are up and running in a minute and you can deliver that precious business value the very next.
Side-note: Justifying working with Erlang or OCaml, even though you know that in the mid and long term will make much more robust software (let's scope out the hiring pool / social aspects of this for a moment), will be a lot harder because the immediate value is actually a lot less.
Organic designs are valuable from day 1 because they grow with the business needs. Artificial designs are much riskier and only theoretically more valuable.
tldr: justifying sustainable engineering is part of our job, but if you fail at that, hack shit up and make bank.
Let me know what you'd like to read more about and have a great start of the week <3
/ Leandro