That Time I Almost Joined a Software Cult

Alloy Workshop

There’s just one slot left for the Alloy Workshop! Use the code YARLSNARTH for $1200 off.

Are We Engineers

Toxx clause is still in effect: if I don’t have a complete first draft by the end of this month I’m giving 1,000 USD to a random subscriber. Currently about 7k words in.

…I never said it had to be a good first draft.

That time I almost joined a software cult

No software lessons here, just me reminiscing about a weird thing I discovered.

NOTE: I included links to a community I’m being spicy about. Please don’t be jerks to them. That means, among other things, don’t post until you’ve spent time lurking and learning community norms. Thanks.¹

So it was back in the utopian days of 2016, when our president could read and I worked as a back-end engineer at eSpark Learning. I was the maintainer of one of our complex legacy systems that had accidentally become a distributed system, leading to all sorts of chicanery. I had just fixed a critical issue by, and I swear this was the best engineering decision we could have made, adding a message queue that spun up an automated browser to click buttons on a vendor’s GUI. It saved our butts but really underscored to me that this system was bad and getting worse. It wasn’t enough to just apply familiar engineering practices, I had to treat it as a distributed system first and radically rethink how we were going to maintain it.

(The system did get worse, leading me to discover TLA+ and permanently changing my career. But that will all happen several months later.)

So I went to searching. I needed an approach to distributed system architecture that 1) would make things easier to reason about, 2) could be implemented with Rails, and 3) scratched my eternal love of extremely weird tech. I had also unrelatedly done horrible things with AWS Data Pipelines² and favored the idea of seeing everything as a directed graph, of information flowing from nodes to other nodes. I envisioned something like a circuit, or an effectful ETL, or a network of firing synapses. I wanted asynchronous queues and pub-sub meshes.

Looking back, I probably “wanted” something like Smalltalk-style OOP or an actor model or Petri nets. But instead I discovered flow-based programming. According to the Wikipedia entry:

FBP is “a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. … FBP is a particular form of dataflow programming based on bounded buffers, information packets with defined lifetimes, named ports, and separate definition of connections.

I immediately latched onto it as a potential solution to all of our problems. This was a way of thinking about our product as a holistic system, not just a bunch of services stirred around in a big ol’ pot. It could work as well with Ruby as it did with Java or Erlang. If we just organized things in an FBP way, then I could easily debug everything we had and I would’t have to sleep in the office anymore!³

There was just one problem: all the examples of the time were way too abstract for me. There were sketches of toy examples that showed some diagrams and said that they worked but didn’t show what FBP looks like as a production system. There was nothing deeply comparing and contrasting the FBP way of doing things from the barbaric way. There was nothing talking about the patterns and where you’d start with implementing this in an actual system. So it was cool in theory but I needed something talking about practice to actually put it in practice.

Now there was a book for sale, but I completely ignored it for some reason and looked for free material instead. I started trawling online. The website is good now but wasn’t back then and had very little information. Nobody seemed to have heard of it in any of the programming circles I was in.⁴ I couldn’t even find any mentions on Hacker News or Reddit programming, which at the time I thought represented what “the broader community of tech” thought.

And then I found the Google Group. Wasn’t really linked anywhere, kind of hard to find if you’re not looking for it. But it was there, it was active, and the inventor of FBP was even answering questions! I could go here and read what the inventor himself had to say about things! I could finally learn FBP inside and out!

All that excitement went sour when I clicked on the very first topic and read this:

If I were of a suspicious frame of mind, I would think many of you are still thinking procedurally, not data flow! :-)

Now this might not seem like a lot to you, but it was a huge red flag to me. It’s one thing to say that FBP has a different mode of thought than imperative programming, but it’s quite another to accuse someone of not thinking right about this problem. It’s a bit tongue-in-cheek, sure, but as my introduction to this space? It put me in just the right suspicious frame of mind to notice the some other warning signs:

• A central “most powerful” figure in the community
• An absolute certainty that they have discovered the secret to making good software, that FBP is “more similar to engineering” than other programming.
• Portraying the paradigm not just as a paradigm of programming but a completely new way of thought that will revolutionize the industry.
• Creating a single “enemy paradigm” that explains why the current system is broken and that can only be fixed by FBP. In this case, it’s “von Neumann paradigm”, which is considered the root cause of all conventional programming’s ills.⁵
• An all or nothing view of what makes the paradigm, so that any deviation from that specific conception doesn’t count as being “correct”. There’s a page about how one system, NoFlo, isn’t real FBP because it’s not multithreaded and allows for data to be cloned.
• Only one success story used in examples, which cannot be examined, verified, or reproduced.
• Very little understanding of other fields of programming or other paradigms of design. The comparison of FBP to OOP is circa ~1993 and assumes OOP is a purely academic thing. Some argued that FBP is FP despite FBP’s central use of mutable data, global state, and (to my understanding) lack of higher-order functions.
• A lot of complex jargon with few or no affordances to outsiders.

This isn’t a revolutionary paradigm that would reshape the world, it’s a cult!

…

Okay that’s a little bit harsh. And I’m sure they’re all nice people. But in aggregate, these things create an outside view that FBP is not nearly as powerful or important as these people believe and that their judgment on things isn’t necessarily trustworthy. If they say something like “mutable information packets is essential to FBP”, they haven’t been truly challenged on it in a way that would make me believe it can withstand such a challenge. And I don’t think these people have done the absolutely critical hybridization with other ideas that FBP would need to mature.

(Couple important tangents: just because the community is this way doesn’t mean that FBP can’t be useful to people: NoFlo and NiFi are both FBP systems, though the FBP community would argue that they’re just “FBP-inspired”. Also, FBP is very similar to the general principles of dataflow programming, which has been independently invented in a lot of places and is considerably more popular than FBP.)

Of course, at the time I wasn’t interested in what it meant for FBP to be “cult-like”, I was just looking to make better systems. This all spooked me enough that I dropped it and went onto other ideas, like property-based testing and (eventually) formal methods. When I started writing about formal methods, I thought again about FBP: what could I do to not make FM seem cultish? Some of the things I’ve tried:

• Talk about formal methods as part of a broader defense-in-depth strategy, where it augments existing approaches instead of denigrating them.
• Recognize the inherent complexity of software and the limitations of FM, and acknowledge the tradeoffs involved in using it.
• Share FM as a spectrum of techniques vs an “all-or-nothing” paradigm. People can learn simple techniques like decision tables and finite state machines and benefit from them, and then decide if they want to push further into design and code verification.
• Use examples and case studies from all over software, from access control to distributed systems to workflow modeling to thread scheduling. Make it easy for people to study, adapt, and even criticize models.
• Regularly draw attention to other people in the field, recommending their work, lectures, and ideas.
• Build connections with other communities in software, even if they’re not in FM or software correctness at all. Exchange ideas, make friendships, send chocolate.
• Make FM as accessible as possible so beginners are encouraged to try it out.

I think I’ve been mostly successful with this. Of course it’s not just me, FM is a broad space with a lot of different people in it, most of whom have never heard of me. But I think I did a much better job by trying to avoid the failure modes I saw in FBP. I think these lessons might apply well to any community that’s trying to grow.

Okay I lied I guess there is a software lesson here!