I recently had my final week working in the data center planning side of Google. July would have marked five years of my focus on integer linear programming, modeling data center planning problems as such, and building supporting infrastructure to help explain and understand optimization models.

I’ve decided to transition to a more ambitious and challenging role in cryptography, still at Google. Specifically, I’ll be joining Shruthi Gorantala and Cathie Yun (and a variety of community contributors) on a mission to bring Fully Homomorphic Encryption (FHE) to production systems. Last year Shruthi and her collaborators open-sourced a transpiler (associated paper) that accepts as input a subset of C++ programs and produces as output an equivalent program that operates on ciphertexts in the Torus-FHE scheme. The team has big ambitions, which I can’t disclose yet, but my role will be closer to my mathematical expertise and passion than my current role.

Working on FHE will be something of a dream come true, and a once in a lifetime opportunity. FHE was discovered to be possible around the same time I discovered mathematics as a passion (2009). When I learned about it in the intervening years, it was still not considered efficient enough to be practical. Today, through the sweat and tears of incremental progress on multiple research avenues, it seems we (humanity) are on the cusp of being able to make FHE practical enough for useful applications, with the right combination of software ingenuity, hardware accelerators, and fine tuning. Combine that with Google’s iteratively improving culture around privacy, and it’s clear that now is the best time to get involved at the forefronts of privacy innovation. These formerly theoretical breakthroughs will soon be ready for production.

With so much of the cryptography ecosystem usurped by fraud, Ponzi schemes, and extremist politics, it’s refreshing to immerse myself in a side of that field that feels unambiguously wholesome. In past newsletters I’ve hinted at the question that Jenny Odell posed in her book, How to do Nothing, namely, “What’s it all for?” When I think about software reliability and “five 9’s,” the only realistic answer I can think of is, “Keep people shopping on Black Friday.” Or when I think about social media, it nags me to think it’s keeping your eyes glued to YouTube/Netflix, Twitter outrage, TikTok, or conspiracy theory Facebook groups, ultimately all for shallow engagement metrics and more ad clicks.

Last May I wrote about a new book I’m working on, tentatively titled “Practical Math for Programmers” (working subtitle, “A Tour of Math in Production”, h/t to everyone who contributed suggestions). I thought now would be a good time to share some updates.

In May I had written 9 chapters (each 3-5 pages with a code sample). As of today I have written 16, with two more in an in-progress state. It’s been slow going in part because I haven’t been prioritizing book writing (until the start of 2022), in part because I had other life changes going on (bought a house, moved to Portland, OR, helped my parents move/buy a house, renovations, just needing to relax, etc.), and in part because I’ve been tackling more challenging and unfamiliar topics that require more research.

On the positive side, this year I joined an amazing writer’s group for technical nonfiction, and it’s incentivized me to write at least one new chapter every month so I can send it to these peers for feedback. I’ve also made an agreement with my wife to set aside at least 4 hours each week for undisturbed writing, which has been fantastic, but not enough to make the sort of rapid progress as I’d like. I.e., at this rate, 1-2 chapters per month, it’ll take me 2-4 more years to finish.

Another interesting aspect is that I’ve started to incorporate proper interviews into my writing routine. For example, I had a lovely interview with Josh Menke on TrueSkill and Bayesian methods for skill ranking in online games (cf. TrueSkill2). It’s been great to talk to people working in the trenches, so to speak, who can really elucidate the advantages and limitations of particular techniques. And most importantly, validate that yes, these techniques are used in prod and we like it that way.

Ideas for math tools

Mathematicians have some great software available. Even domain-specific systems like Macaulay2, a programming language for commutative algebra and algebraic geometry. And then there are some amazing projects like the OEIS that, in addition to maintaining a huge database of integer sequences, has a search engine for integer sequences and a database of “facts” and relationships between sequences.

But I think there’s a lot more cool stuff, on a smaller and more tractable scale, that could be done today if someone put in the time and energy. Here are some random ideas in that vein. Apologies if it’s been done, I’d love to hear from you with references.

An open problem tracker. Today open problems are tracked entirely manually, with conjectures scattered through papers and no clear indication of whether a problem has been solved. OEIS’s “under the fold” submissions with facts about particular integer sequences is probably the closest thing I’ve seen to a structured database of open conjectures and results, and it’s not all that structured. It would be cool to have a stackexchange-type website where each entry is a problem statment and it can be solved when a paper is published that claims to solve it, and with sufficient time and corroboration. Imagine it also has crossreferences to variants, generalizations, specializations, and updates on progress and retracted attempts.

Programming as a Vehicle for Math

In March 2020, I gave a talk at Math for America, an organization that fosters professional development for K-12 math teachers in the New York City area. It was part of my PIM “book tour,” though for this talk I focused on showcasing computers as a tool for creating things you care about, while framing the obstacles to creation as a perfect time to introduce mathematics. Most of the talk is examples of this.

The MfA organizers never posted my talk online, and at this point I’ve lost hope that they will (thanks, Covid). So I’ll recap the content of the talk, linking to my slides (click there for nice images and gifs) and the transcript I prepared in advance of that talk. This post will summarize the main ideas and provide some extra color.

Math lets you write cool programs. And it does it in a way that is fundamentally different from any fashionable programming topic of the week. Most programs that rely on mathematics have, at their core, an algorithm or formula that is indecipherable without learning the math it’s based on. Bit somehow it’s simple once you understand the math.

[Nb. this is a very rough draft but I’m sad I’m not writing much these days so I want to get the juices flowing again.]

Mathematicians (especially pop-math book authors) like to wax poetic about math as unveiling the secrets of the universe, or as the furthering of human knowledge. But many mathematicians also denigrate the algorithmic questions in their fields, even if in private they rely on them. The mathematician Irving Kaplansky once said,

[Halmos and I] share a philosophy about linear algebra: we think basis-free, we write basis-free, but when the chips are down we close the office door and compute with matrices like fury.

My view much stronger than the average mathematician, who might say that “knowledge” is the end goal, while efficient algorithms are a nice byproduct. I would say that knowledge is equivalent to efficient algorithms. I don’t mean this in a Curry-Howard sense, that all proofs are programs and vice versa. I mean that mathematical knowledge is defined by its computation-granting abilities. Knowledge exists if and only if you can compute answers to questions exhaustively in the domain of that knowledge.

Monorepos and Forced Migrations

By most metrics, Google is a great place to work. Top tier compensation, free food, and largely co-workers who want do quality engineering. And yet, many Google engineers burn out. Putting aside problems with bad managers, or ethical objections—both of which are more important than whats in this article—I talk to many Google engineers whose heart simply isn’t in their work. On top of that, there’s a lot of friction to doing anything at Google. Few tasks are simple. Many engineers, myself included, often feel like their most creative and productive years are being wasted to jump over needless hurdles, all to optimize consumer behavior to boost ad revenue.

Let’s zoom in on one particular component of friction that affects me regularly. It’s about how software migrations work at Google. Most open source software works in terms of versioned releases. New releases add features, and major releases can be backwards-incompatible, but for the most part, clients are rarely forced to upgrade (sometimes at their own risk). And while eventually very old things will naturally lose support, the most laudable software projects take decades to do this. For example, most of the Java API is backwards compatible with its early versions. This allows me to run on a modern system without changing a line of code.

Mathy Jobs

Some folks expressed interest in hearing about what sort of mathy jobs there are, beyond the standard of academia and teaching. I had to think about this recently because I moved to Portland, and while my employer has discovered a newfound flexibility for relocation (albeit at reduced pay), in the year-ish interim I started compiling a spreadsheet of options. I’ll share them with you, and if you have any more ideas, please send them to me (by replying to this email) and I’ll compile them into a longer blog post.

My perspective is obviously biased toward jobs where you write software, because that’s what I do. My feeling is that it’s more interesting to work on tools that use math to help other people do their domain-specific jobs better, than it is to do the domain-specific job myself. That said, entirely avoiding any domain-knowledge is a detriment. The balance is key.

Layering Abstractions

When designing software systems, we care about flexibility as requirements change. One might break that down into two rough parts. First, flexibility across systems—i.e., when the way that two or more systems interacts needs to change, one can adapt without significantly changing any individual component. A lot has been written about this, including the Unix Philosophy, SOLID (some of which overlaps this article’s content), , and .

tl;dr: I’ve started working in earnest on a new book, tentatively titled, Practical Math for Programmers. Sign up for a mailing list to be notified when it’s published.

Modeling state in linear programs

There’s a neat way to model decision making as an (integer) linear program for managing a collection of resources. In my head it’s called the “state-dynamics” model. In other places I’ve seen it, such as this set of notes, it’s an unnamed “formulette,” and they describe it as a “periodic review of a continuous process.” That name also makes sense, because you can view this model as periodically sampling a continuous function, and optimizing over an (approximate) integral of the function.

I’ll explain the situation with bananas. Say you’re managing an inventory of bananas. You occasionally get orders from customers to buy your bananas, and you occasionally place orders for new bananas from wholesale suppliers. As you read this, you may substitute “bananas” for any good that you can bucket over time and according to specific features.

Tracking the progress of a distribution

I’ve been super busy lately, with work, a baby, and house hunting. Fun projects will have to take a back seat for now.

But! I recently found an opportunity to exercise some basic linear algebra when working on a metrics problem. It’s simple enough to write up in the moments between house showings and baby duties.

To anonymize the domain, let’s say we’re working with sheep colored white, green, or purple. You start with a flock of sheep, each of which may be any of the three colors. You’re given a target distribution over colors , and you want to make your flock have that distribution of colors. You can paint your existing sheep, and/or buy new pre-painted sheep in a desired color, or sell sheep.

Ever since I wrote What’s in Production? I’ve grown steadily more obsessed with it. The act of writing these thoughts down focused the discordant, nagging voice in my head into a church choir singing the praises of production applications.

The best part is that the more I wonder, the more interesting examples I learn about. Some are pleasantly expected, like bandit learning being used for COVID testing, some are surprisingly simplistic, like bandit learning NOT being used for , some are genuinely new, like an , and some are old favorites I’m revisiting, like .

Last week Tai-Danae Bradley invited me to give an Alphabet-wide math talk, in part advertising a series of internal courses she’s teaching on pure math for software engineering applications.

I introduced the talk from the perspective of a skeptical engineer who is asked to learn math, but doesn’t believe the investment will pay dividends. I then framed the question,

What specific, practical program could I reasonably claim to be to write without math?

Little annoyances, big decisions

A story

I think software engineers, myself included, are influenced heavily by annoyances, and those annoyances drive major decisions in our industry far more than we’d like to admit.

I worked with someone, a much more senior engineer than me at the time, who strongly dislikes Java and strongly prefers C++. I’ll call this person Bjarne to remind you that they love C++ (Bjarne Stroustrup created C++).

Apologies for the delays in this newsletter. With the baby and the coup, the newsletter took a momentary backseat. I also wrote a few drafts (not about LaTeX) that I was unhappy with and scrapped completely.

What’s not to like about LaTeX?

TeX/LaTeX (and its cousins XeTeX, LuaTeX, etc.) is the de facto typesetting system of choice for all mathematicians. Basically all papers and math books are written in it, and its “math mode” syntax has been widely adopted by the most widely-used web-based math display systems such as MathJAX.

TeX was written by Donald Knuth in 1978, and LaTeX (a system of macros written on top of TeX) was written in the early 1980’s. Development has continued on LaTeX, and the underlying TeX system has been effectively been marked as “complete.”

I wanted to care about software estimation

I’m the kind of person who hears about a good idea and immediately wants to try it. So when I read about reference class forecasting, and coincidentally heard some managers in my org discussing the problem of software projects going way over (time) budget, I eagerly suggested we try reference class forecasting.

Reference class forecasting is a familiar principle: you should use data to inform your estimate. People tend to underestimate the time it takes to deliver a project when they’re deep in the details of the project. Due to optimism or overconfidence, “everyone thinks they’re above average.” I know I often underestimate. Reference class forecasting suggests you instead predict a project’s delivery time based on the actual delivery time of past “similar” projects. In mathy-softwarey lingo, collect a dataset of project completion time estimates and their actual completion times, define a probability distribution over the past data, and compare your estimate for a new project to the distribution. This will cause you to adjust your estimate to hopefully better match the truth.

The value of math is at the margins

I used to work at a Bitcoin startup. It ended up being a waste of my time, but I was lured there with promises of working on interesting mathy problems in the cryptography space. What actually happened was that the interesting mathy work didn’t materialize. Instead, we rushed to launch a product, a website, a mobile app, fix bugs, deal with customer feedback, and promote it. I spent almost all of my time on those things, and almost no time on anything mathematical. In hindsight, this was all too obvious. Startups need a thousand basic things before they need anything close to mathematical sophistication.

Google doesn’t seem to be too much different, at least in my position. I work on optimization pipelines that automate the planning of Google’s data centers. It sounds cool, but once the basic system is in place almost all of the work is decidedly unmathematical. Data is dirty, documentation is missing, outages happen, and alignment and communication are hard. Not to mention that mathematical deliberation is slower than hacking. If you take too long, organizational pressures will mean someone else comes up with a patchwork solution that becomes the established normal leaving your work fighting against a general fear of change. Add to that the all-too-common organizational shock, like a re-org, important engineers leaving, or a global pandemic, nuanced mathy work can get deprioritized until death. That’s software and business working as intended, as my managers remind me.

The Spiritual Nature of Software Tools

At the risk of waxing philosophical and boring you, I want to discuss one other aspect of Simone Weil’s “The Need for Roots”. A persistent theme throughout her writing is how society must recognize the spiritual nature of work.

I read it on multiple levels. First, that people need to work to feel useful, one of the fundamental needs of a human soul. Second, how physical labor forces a normally active human mind, to slow down, becoming “subject to time in the same way as inert matter.” In this way it has an impact on the human experience unlike aesthetics or logic. Couple that with the fact that we spend so much of our lives working, and it makes sense to say that work stands above “all other human activities” in spiritual significance.

Future book ideas

My first book, A Programmer’s Introduction to Mathematics (PIM), was, in my view, a success. Self-publishing provided enough royalties to make it financially worthwhile. And I enjoyed writing it. This has made me consider writing more books. Maybe even become a full-time writer. In exploring this option, I’ve written down a few book ideas, which I’d like to share with you. I’ve already started drafting some of them, albeit erratically.

The Needs of the Soul

Last issue I waxed poetic about some informal communities I was involved in. I’ve been reading a few books about what makes a good community.¹

What makes a community?

Recently I’ve been thinking a lot about community and culture.

I’ve participated in many communities over time, centered around a school or workplace, a shared living space, organized activities like Boy Scouts and sports, or intellectual activities like running a graduate seminar.

One thing I noticed in all this time is how ephemeral a community can be, and how much they rely on the persistence of one individual to keep it going. One example is when I ran a graduate seminar as a PhD student. Once a week someone would give a talk on a topic of their choice. If nobody wanted to, or I couldn’t find an invited speaker, I would be forced to present or the seminar would simply not happen.

What is Actually Used?

Everyone wants their work to be useful. Being useful to others is a fundamental need of the human soul. That often manifests as the need to feel useful to others, even if it is fulfilled only superficially. Or worse, one tries to accommodate pressure to make their work seem useful in ways that it isn’t, or to groups for whom it isn’t, because those ways of being useful are considered “better” by someone in a position of power. These manifestations are ultimately harmful.

This has implications for mathematics, when we ask the question, “What parts of mathematics have useful applications?” (The mature version of “When are we ever going to use this?”) But first I want to talk about software, and a simpler question, namely, “What is in production right now?”