Open-sourcing our Python-to-Typescript route generator

Buttondown&#39;s contribution to open source software is long overdue; we, like every other piece of software out there, stand on the shoulders of giants.
Many parts of Buttondown do not make sense to open source: they are overfitted, finnicky, or otherwise engineered in a manner that makes them a poor choice of software for community usage and/or stewardship.
But there are many parts which I suspect would be useful as open source projects, and I&#39;m excited to announce the first of many such initiatives: the dryly (but descriptively!) named <code>django-typescript-routes</code>.
Check out the repository <a href="https://github.com/buttondown-email/django-typescript-routes">here</a>.
<code>django-typescript-routes</code> is a way to connect the URLs of your Django application to a Typescript app. That&#39;s all it does: you provide a <code>urlconf</code>, it spits out a <code>URL</code> object.
<img src="/next-assets/img/urls.png" alt="">
This is tremendously useful to get some of the benefits of strong typing even when you&#39;re using Python to power your SPA. It&#39;s a small utility, but one that has prevented countless bugs in production — I hope you find it useful.

Announcing django-typescript-routes

We recently added pre-filled templates for your archived emails' share images, using Vercel's @vercel/og and Satori libraries.

Buttondown has the ability to add <a href="https://docs.buttondown.email/advanced-features/share-images">custom share images</a> (&quot;og:images&quot;) to your emails, which show up on social media sites when you share the link to your archived email.
We recently added a couple pre-filled templates that you can choose, as an alternative to designing your own image and uploading it:
<img src="/next-assets/img/image-generator.png" alt="The new templated share images available in Buttondown&#39;s settings. They&#39;re prefilled with your newsletter&#39;s name and email&#39;s subject.">
We built this using Vercel&#39;s new <a href="https://www.npmjs.com/package/@vercel/og">@vercel/og</a> package, built on <a href="https://github.com/vercel/satori">Satori</a>.
It generates the images directly, instead of the previously ubiquitous method of creating a screenshot using <a href="https://pptr.dev">Puppeteer</a>. That method was effective, but it takes a lot of infrastructure to run an entire Chrome browser just to take screenshots. Vercel&#39;s solution is lightweight and works beautifully with <a href="https://nextjs.org">Next.js</a> and <a href="https://vercel.com">Vercel&#39;s deployment platform</a> (as expected, given that they made both).
The problem with a constrained environment like images is that a lot of possible edge cases can arise — most of them involving text that&#39;s longer than expected or not in the format you expected.
To mitigate this, I built a big page of test case so I could visually confirm that nothing&#39;s falling apart. I had a bit of fun with the test cases as well:
<img src="/next-assets/img/image-generator-debug.jpg" alt="A page with 16 test og:images, each testing different edge cases such as &quot;long author name.&quot;">
I also built a randomized tester, which takes the most common English words and generates a nonsensical newsletter name and subject line. By clicking &quot;randomize&quot; over and over, I could get some additional confidence that the designs were resilient.
<img src="/next-assets/img/image-generator-randomize.jpg" alt="A button labelled &quot;Randomize!&quot; next to two share images, which contain randomly generated text.">
(Of course, I&#39;m sure there are edge cases that I didn&#39;t catch. If you find a combination of text that looks off, please let me know — <a href="mailto:support@buttondown.email">support@buttondown.email</a>!)
Now, the templates are piped into Buttondown&#39;s share image picker (which comes up when you click &quot;Web archive&quot; in the toolbar at the bottom). We use query string parameters to pass the newsletter&#39;s name and subject to render these dynamically.
<img src="/next-assets/img/image-generator.png" alt="The new templated share images available in Buttondown&#39;s settings. They&#39;re prefilled with your newsletter&#39;s name and email&#39;s subject.">
Since Buttondown simply stores the URL of the share image, this required minimal changes to Buttondown&#39;s app. The custom-generated images are now available as an option, if you choose to use them.

How we built made-for-you og:images

Step by step insights into cutting the email view's TTFB by 70%

<h2 id="introduction">Introduction</h2>
Buttondown is a Django app for sending emails, mostly beloved by technical users who like its REST API and Markdown support. Like most other email platforms, it also hosts archives of sent emails, and these archives process a millions of pageviews every month in traffic.
These archive pages are classical Django detail views: <a href="https://buttondown.email/hillelwayne/archive/i-really-like-powershell/">here</a> is an example of one. You can see from the URL that the contract is pretty simple: given a username and a slug, return the email. Simple enough!
I noticed that Google Search Console was dinging Buttondown for a slow TTFB: Buttondown&#39;s archive pages were averaging around 1.2s, and the guidance is to be below one second. These views are pretty uncomplicated, so I dove in with Django Debug Toolbar (ol&#39; faithful) to see what was happening:
<img src="https://pbs.twimg.com/media/FugNtpxWAAATluU?format=jpg&name=medium" alt="">
Nine queries! Yikes! DDT is kind enough to provide all of the information you really need: what queries are being run, which ones are repeated, and in what order. Let&#39;s dive in, shall we?
<h2 id="removing-unnecessary-joins">Removing unnecessary joins</h2>
This is obvious when you say it out loud: the easiest queries to optimize are the ones that you can straight up remove because they&#39;re entirely unnecessary. In this case, I had two such queries:
<ol>
<li>A query on <code>monetization_stripeaccount</code></li>
<li>A query on <code>monetization_stripe_payment_link</code></li>
</ol>
Both of these pull data from Buttondown&#39;s mirrored store of Stripe data, and are entirely unnecessary on this page: we don&#39;t surface the payment link directly to users on this route, nor do we need information about the author&#39;s Stripe account. So we remove them from the serializer that was pulling them in:
<pre><code class="language-diff">class PublicNewsletterSerializer(serializers.ModelSerializer):
- stripe_account = serializers.SerializerMethodField()
- stripe_payment_link = serializers.SerializerMethodField()

...

- def get_stripe_payment_link(self, obj: Newsletter) -&gt; str:
- if not obj.stripe_payment_link:
- return &quot;&quot;
- return obj.stripe_payment_link.data[&quot;url&quot;]

- def get_stripe_account(self, obj: Newsletter) -&gt; str:
- if not obj.stripe_account:
- return &quot;&quot;
- return obj.stripe_account.account_id

...

 class Meta:
 fields = (
 ...
- &quot;stripe_account&quot;,
- &quot;stripe_payment_link&quot;,
 ...
 )
</code></pre>
Excellent! We&#39;re already down to seven queries, as confirmed by DDT:
<img src="https://pbs.twimg.com/media/FugOPoGWIAUEBG7?format=jpg&name=small" alt="">
<h2 id="caching-the-object-on-the-view">Caching the object on the view</h2>
DDT helpfully informs us that some of these queries are duplicated: we grab the email, newsletter, and account twice. This is because of some janky logic I have in the permissions-checking for a given email: we show a slightly different version of the email depending on who&#39;s logged in (a premium subscriber may see different data compared to an unpaid one, for instance, or the author may be able to see posts that aren&#39;t published yet.)
Rather than fetching the email twice, then, let&#39;s just cache the email the first time it&#39;s fetched:
<pre><code class="language-diff">@method_decorator(xframe_options_exempt, name=&quot;dispatch&quot;)
class EmailView(TemplateView):
 def get_template_names(self) -&gt; List[str]:
- return determine_template_path(self.request, self.get_object())
+ return determine_template_path(self.request, self.email)

 def get_username(self) -&gt; str:
 if hasattr(self.request, &quot;newsletter_for_subdomain&quot;):
@@ -89,7 +89,8 @@ def get_object(self) -&gt; Optional[Email]:
 email_slug = self.kwargs.get(&quot;slug&quot;)
 username = self.get_username()

- return find_email(email_id or email_slug or &quot;&quot;, username)
+ self.email = find_email(email_id or email_slug or &quot;&quot;, username)
+ return self.email

 def get_context_data(self, **kwargs: Any) -&gt; Dict:
 context = super().get_context_data(**kwargs)
</code></pre>
That brings us down to a mere four queries:
<img src="https://pbs.twimg.com/media/FugOtC_XsAIErub?format=jpg&name=small" alt="">
<h2 id="denorming-with-select_related">Denorming with <code>select_related</code></h2>
If you look at those top three queries, it sounds fairly obvious what they&#39;re doing:
<ol>
<li>Grab the email for a given slug and username.</li>
<li>Grab the newsletter for that email.</li>
<li>Grab the account for that newsletter.</li>
</ol>
These are all part of the same object, so we can use Django&#39;s <code>select_related</code> method to pull in
those foreign keys as part of the initial query.
<pre><code class="language-diff">- return emails.get(**kwargs)
+ return emails.select_related(&quot;newsletter__owning_account&quot;).get(**kwargs)
</code></pre>
And with that one-liner, we&#39;re down to 385ms across only two queries:
<img src="https://pbs.twimg.com/media/FugPDEhWYAMyF2F?format=jpg&name=small" alt="">
<h2 id="using-defer">Using <code>defer</code></h2>
We&#39;re down to just two queries, but one of them is fairly large — 300ms for a simple join on an index seems awfully large, right?
Well, here&#39;s a fun bit of esoterica about Buttondown&#39;s implementation: <code>Email</code> objects are large. They contain a bunch of different cached versions of themselves, like:
<ul>
<li>How the email should look as a &#39;teaser&#39;</li>
<li>How the email should look on email</li>
<li>How the email should look in Markdown</li>
</ul>
and so on, to make sending emails as quick as possible. But — especially for larger emails — these add up in size, and the simple act of serializing them from Postgres to Python is non-trivially slow.
This is compounded by the fact that Django&#39;s ORM defaults to the equivalent of a <code>select *</code> — if you fetch a row, you&#39;re getting all of the columns included in that row.
So, as a final touch, we use Django&#39;s <code>defer</code> command which allows us to selectively exclude certain fields that we know we don&#39;t need (in this case, it&#39;s various versions of the email that are rendered for other views):
<pre><code class="language-diff">- return emails.select_related(&quot;newsletter__owning_account&quot;).get(**kwargs)
+ return emails.select_related(&quot;newsletter__owning_account&quot;).defer(
+ &quot;rendered_html_for_email&quot;,
+ &quot;rendered_html_for_web&quot;,
+ &quot;body&quot;,
+ ).get(**kwargs)
</code></pre>
That saves us another 100ms, bringing us down to around 300ms!
<img src="https://pbs.twimg.com/media/Fuj7QLoX0AEoh8O?format=png&name=small" alt="">
<h2 id="conclusion">Conclusion</h2>
We started with a view that took 1050ms across nine queries and we ended up with one that took 300ms across two queries. Pretty excellent for an hour&#39;s work and ~thirty lines of code, considering this view gets hit a few million times a month.
It bears repeating that none of the techniques described above are particularly novel. None
required extensive tooling beyond <code>django-debug-toolbar</code>; exactly zero rocket surgery was
performed in the course of this optimization.
And yet! This code stuck around, unoptimized, for quite some time. There is so much performance
work out there in so many applications that looks like the above: twenty-dollar bills, sitting there
crumpled on the ground, waiting to be picked up.
To wit, there are other things I could probably do to get this endpoint even faster:
<ul>
<li>That second query is checking to see if the newsletter being fetched has any &quot;metadata fields&quot; associated with it. Metadata fields are a bit of an edge case feature, and only around 15% of newsletters use them; I could add a field like <code>Newsletter.has_metadata_fields</code> and check that before making the query, to eliminate it for the 85% of newsletters who don&#39;t use it at all.</li>
<li>The primary query to pull out an email and a newsletter would be slightly faster (~30ms) if I queried the newsletter by ID instead of username. I could keep a cache mapping usernames to IDs to take advantage of that.</li>
</ul>
And all of those are good ideas for different days — but I&#39;m a sucker for performance work that ends up with negative lines of code added.

Speeding up a Django view

I&#39;m not sure exactly what to call this fairly common pattern, but there are a bunch
of strings in Buttondown that I want to be able to pretty-print in a fallback-friendly way.
UTM campaigns are a great example: 95% of the time this value is <code>facebook</code> or <code>twitter</code> or <code>blog</code> or <code>email</code> but it technically can be anything. So you round up
the most common keys and throw them in a map that you can reference down the line:
<pre><code>const UTM_CAMPAIGN_TO_LABEL = {
 &#39;facebook&#39;: &quot;Facebook&quot;,
 &#39;reddit&#39;: &quot;Reddit&quot;,
 &#39;linkedin&#39;: &quot;LinkedIn&quot;
 // ... and so on
};

[
 &quot;facebook&quot;,
 &quot;reddit&quot;,
 &quot;linkedin&quot;,
 &quot;yandex&quot;
].map(campaign =&gt; UTM_CAMPAIGN_TO_LABEL[campaign] || campaign)
</code></pre>
The problem is that, by default, <a href="https://www.typescriptlang.org/play?#code/MYewdgzgLgBAqgFQLIH0DCBBJAFDBJAcQDkUEB5FAGQwCEBRSmAXhgG8AoGLmAcgDMAhsACmAIxAgA1jwBcMAEQAxIWImT5AGk7ceAJ2EATAwEsoshQCVDJqJu1ceAG2NhJhl+fmUXbg3jDy9jAA9MEwAHSRMAJgBjAQIDDg7AC+ANzs7ADa7ACQ8oIi4lJ2+fpGpqXyzq7uAVr5AJ4xBsIAHoEAuuEAtgIADgAUwAI9-QLGAOZgzAB88MjoWLiEJORUtAxZI2MT050wAD6HMDvjU2AAlOxAA">Typescript doesn&#39;t like that</a> — it knows what&#39;s in your map, and unless the strings you pass in are a strict subset of those keys it&#39;s sad.
All you need to do is ease the type inference by expanding the key type to string, <a href="https://www.typescriptlang.org/play?#code/MYewdgzgLgBAqgFQLIH0DCBBJAFDBJAcQDkUEB5FAGQwCEBRSgLhgG8BtAawFMBPGASzAxoAJ0EBzALrNREgL4wAvKwBQMdTADkAMwCGwLgCMQIDpuYAiAGL6jJjhYA0ajZpFcAJh-5RzMCwBKnt5QTi7qmgA2gtzeYH4WlDGeeGAW4TAA9JkwAHT5MLpgHsIgMOAqcgDcKipsKgCQFnoGxqZhTe5ePh0W0WCxgr08RR5cAB7pkrkAtroADgAUwLoz87r84kKKAHzwyOhYuIQk5FS0DGwraxtbkjAAPg8w1+ubYACUKkA">like so</a>:
<pre><code>const UTM_CAMPAIGN_TO_LABEL: {[key in string]: string} = {
 &#39;facebook&#39;: &quot;Facebook&quot;,
 &#39;reddit&#39;: &quot;Reddit&quot;,
 &#39;linkedin&#39;: &quot;LinkedIn&quot;
 // ... and so on
};

[
 &quot;facebook&quot;,
 &quot;reddit&quot;,
 &quot;linkedin&quot;,
 &quot;yandex&quot;
].map(campaign =&gt; UTM_CAMPAIGN_TO_LABEL[campaign] || campaign)
</code></pre>
This is trivial, and despite its triviality I forget how to do this around 40% of time. My hope is that you, too, may discover this if you google &quot;typescript-friendly label fallbacks&quot; (because what else are you going to call this pattern? I honestly have no idea.)

Typescript-friendly label fallbacks

In which I tell you that you're overcomplicating things and you should throw everything behind a single git repository.

<img src="/next-assets/img/monorepo.png" alt="A before-and-after of my setup.">
I wrote <a href="https://buttondown.email/blog/on-monorepos">last month</a> about having migrated Buttondown from a number of separate microrepositories to a single monorepo. I&#39;ve since completed the migration, and the slight irritation I felt from having to unwind a bunch of technical choices with deployment + continuous integration has faded.
I am here to tell you: if you are running a software business and you aren&#39;t at, like, Google-tier scale, just throw it all in a monorepo. I am kicking myself for not having done this to begin with.
Here is a list of I think reasonable drawbacks of having a monorepo if you are not a Google-tier company:
<ol>
<li>You cannot apply role-based access controls to certain parts of the repository for security/compliance reasons.</li>
<li>You cannot open-source certain parts of your repository. (This was the biggest sticking point for me for a while — I want, ideally, everything in Buttondown to be open-source except the core plumbing and author-facing application.)</li>
</ol>
Conversely, here are a list of things that became an order of magnitude easier now that my entire codebase is in a single repository:
<ol>
<li>Rolling out API changes concomitantly with downstream changes to the documentation or the OpenAPI spec.</li>
<li>Introducing feature-level changes and the blog post announcing those changes.</li>
<li>Context-switching between different workspaces with subtly different configurations.</li>
</ol>
You may be reading this short essay and thinking: &quot;Yeah, no shit! Of course you do not need a monorepo when you&#39;re talking about, like, 150K lines of code.&quot; Congratulations — you are sensible, and well adjusted, and certainly wiser than I was six months ago.
But I know there&#39;s a few of you out there who are clinging to something — some irrational reason why you have your welterweight project split across four separate repositories.
Maybe it&#39;s some abstract sense of purity; maybe it&#39;s because you didn&#39;t realize most IaaS let you deploy from a subfolder. Whatever the reason is: trust me. Move it into a monorepo. If you do and you end up regretting it, [email me] and I&#39;ll donate to the charity of your choice.
To wrap up, I will deputize two people who are smarter than I am who have also opined on the subject:
<ul>
<li><a href="https://danluu.com/monorepo/">Dan Luu</a></li>
<li><a href="https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repositories/">Gregory Szorc</a></li>
</ul>

Just use a monorepo

I opine, perhaps for too long, on the idea of monorepos.

I wanted to write about annual planning, and how this year is going to differ from previous ones, but I have (in true fashion) procrastinated finalizing my annual plan a little bit. So that&#39;ll be next week, and this week I would like to write about... monorepos.
<img src="https://pbs.twimg.com/media/Fj9FMhEXwAgxQyP?format=jpg&name=large" alt="">
Buttondown is now in a monorepo. There&#39;s two big advantages of this approach (and these advantages are not particularly new to Buttondown):
<ol>
<li>Code sharing is much easier. I now have a lovely OpenAPI spec that powers the documentation page (and is published to its own repository, though I&#39;ll get to that in a minute); rather than having to perform a bit of a shell game keeping that spec in sync across a number of repositories, I can have a single source of truth.</li>
<li>Concomitant changes are much easier. The ideal feature launch involves not just changing some stuff in the main application but writing a release note, adding a changelog entry, adding documentation — timing the release of all of these PRs was frustrating.</li>
</ol>
There are a couple downsides:
<ul>
<li>A lot of deployment setups assume that you&#39;re trying to deploy an application at root, meaning I got to spend some time mucking around with Vercel (easy), Netlify (slightly less easy), Heroku (kind of annoying) to configure their build steps to hit the relevant subdirectory.</li>
<li>A lot of tooling assumes that your configuration is stored at root. I use a lot of VS Code plugins that all helpfully assume you have a single tsconfig.json or pyproject.toml at root, which... I do not.</li>
<li>My platonic ideal for Buttondown&#39;s code visibility is something like &quot;literally everything is open source except the core application&quot;, and this is a step in the opposite direction. Even taking a fairly practical thing that I want to be open for obvious reasons, the OpenAPI spec, I now have to manually cut new PRs from the monorepo against the specific OpenAPI repository (or, more likely, set up a GitHub Action to perform the same thing). This is kind of annoying.</li>
</ul>
As you can probably tell from the benefits and drawbacks, this is...marginal. It took around five hours to finish everything (less than I expected) and I&#39;ve probably recouped all of fifteen minutes&#39; worth of effort in doing so. (Which, frankly, isn&#39;t a bad calculus if you extrapolate from a week to a year or so.)
But this is the time of year for hastily undertaking such projects! I&#39;m trying to keep the backlog down low, but all of the &quot;this seems nice but when am I going to have the time&quot; type of work happens right now, in that lovely December twilight, where the roadmap doesn&#39;t matter and the sales pipeline slows down a bit.

On monorepos

Pre-announcing an alternative to django-js-reverse

(Nerd alert! This post is a small technical snippet for Python and Django programmers. If you don&#39;t know what those proper nouns mean, don&#39;t worry — feel free to skip this post.)
Buttondown has used <a href="https://github.com/ierror/django-js-reverse">django-js-reverse</a> since its inception to route backend API urls from the Django backend to the Vue frontend.
This has been (not to damn with faint praise!) better than nothing: some kind of syncing mechanism certainly beats having to manually correlate backend routes with frontend definitions. But there are two nits I&#39;ve had with the package:
<ol>
<li>The package isn&#39;t TypeScript-friendly; errors in routing or invocation (such as calling <code>Urls.email_detail()</code> instead of <code>Urls.email_detail(&quot;foo&quot;)</code>) are not caught until runtime.</li>
<li>The package does some light metaprogramming to allow you to refer to a route not just as its name (<code>email-detail</code>) but via camel or snake case (<code>Urls.emailDetail()</code> or <code>Urls.email_detail()</code>). This attempt at ergonomics makes it trickier to actually find all invocations of a route at a glance.</li>
</ol>
Naturally, I thought to myself: why not try my hand at building a better version? I&#39;m happy to share that it was fairly painless: while I&#39;m still a little bit away from formally open-sourcing this as a package (there are a number of administrative tasks like registering with PEP and settling on a name that I need to take care of), here is the <a href="https://gist.github.com/jmduke/434739ebc4e2112397becfd403f2ecfd">total implementation</a> — 34 lines of Python and 4 lines of templated TypeScript.
I am confident this implementation is missing many, many, many edge cases. It does not handle URL namespacing nor non-string parameters, nor I&#39;m sure many other things. But it is a step change improvement to Buttondown&#39;s codebase, and I&#39;m excited to have it. Please let me know if you find it useful, or if you have any feature requests/suggestions — I promise to update this blog post with a link to the repository once it&#39;s published to PEP.

Generating a Typescript router in Django

A quick hit nerd post for all you Pythonistas out there.

(Nerd alert! This post is a small technical snippet for Python and Django programmers. If you don&#39;t know what those proper nouns mean, don&#39;t worry — feel free to skip this post.)
I have been a fan of the asynchronous programming library <a href="https://python-rq.org/">RQ</a> for a long time. It&#39;s been a classic &quot;sweet spot&quot; library: more powerful and futureproof than rolling my own solution from scratch, but more aggressively minimalist and ergonomic than something like <a href="https://docs.celeryq.dev/en/stable/getting-started/introduction.html">Celery</a> which has felt the accumulated weight of a decade of bloat and featureq requests.
And Buttondown uses RQ a lot. Every API request, Stripe event, email sent, or webhook event created (plus around seventeen dozen other things) goes through RQ in order to keep the interface nice and snappy and the concomitant backend requests short and stout.
One of the things that is not so great about RQ is the out-of-the-box observability; it comes with an admin dashboard that does little else than tell you the number of items in each queue, and you&#39;re left to build out your own solution from scratch.
Here is what I did. This ten line <code>Job</code> subclass is easy to plop in and replace with the default one that RQ provides. I&#39;m using <a href="https://www.structlog.org/">structlog</a> as the logger mechanism, but that&#39;s more of an implementation detail than anything (you can use whatever you want).
<pre><code>class JobWithBetterLogging(Job):
 def perform(self) -&gt; Any:
 start = time.time()
 result = super().perform()
 end = time.time()
 logger.info(
 &quot;job.finished&quot;,
 key=self.key,
 seconds=(end - start),
 method=self.func_name,
 data_size=len(self._data),
 queue=self.origin,
 enqueued_at=self.enqueued_at.timestamp() if self.enqueued_at else None,
 started_at=self.started_at.timestamp() if self.started_at else None,
 )
 return result
</code></pre>
You get a lot of information here: your logs now tell you exactly what keys are being logged, how long each function took, how long they&#39;ve been sitting in the queue, the size of the payload you placed within Redis, and what queue was being used. This, combined with offline log analysis (or a structured query engine like <a href="https://logtail.com/">Logtail</a>) will let you construct dashboards that can trivially answer questions like:
<ol>
<li>What functions are taking the most aggregate time?</li>
<li>What functions are bloating up Redis?</li>
<li>What queues are accruing the most demand?</li>
</ol>
And so on.

Better logging in RQ

I nerd out about structuring a medium-sized Django project

<div class="notice">Heads up! This is a pretty technical post with absolutely no new information about Buttondown's features! If you're not interested in Django (or don't know what Django is), feel free to skip this one.</div>

<h2 id="wait-whats-buttondown">Wait, what&#39;s Buttondown?</h2>
I run <a href="https://buttondown.email">Buttondown</a>, a newsletter tool. It&#39;s a pretty nice tool, in my opinion! Please check it out if you get a chance.
This application is around 45,000 lines of Django code [^2], with commits dating back to 2016. It&#39;s grown and mutated a lot over the past six years.
I&#39;ve been meaning to write more about how my approach to structuring and developing Buttondown has changed over the years, and I think a good jumping off point would be the biggest conceptual tool Django offers to organize your code: apps.
<h2 id="wait-what-are-apps">Wait, what are apps?</h2>
The official <a href="https://docs.djangoproject.com/en/4.0/ref/applications/">Django docs</a> define applications as:
<blockquote>
A Django application is a Python package that is specifically intended for use in a Django project. An application may use common Django conventions, such as having models, tests, urls, and views submodules.
</blockquote>
Okay, that&#39;s not particularly useful. Two Scoops of Django is more useful:
<blockquote>
Django apps are small libraries designed to represent a single aspect of a project. A Django project is made up of many Django apps. Some of those apps are internal to the project and will never be reused; others are third-party Django packages.
</blockquote>
Or to go further still, to quote <a href="https://www.b-list.org">James Bennett</a>:
<blockquote>
The art of creating and maintaining a good Django app is that it should follow the truncated Unix philosophy according to Douglas McIlroy: ‘Write programs that do one thing and do it well.”’
</blockquote>
Apps are essentially a DSL around modules: they provide a level of namespacing and structure around logically disjoint pieces of functionality &amp; business logic.
<h2 id="my-general-philosophy-with-apps-design">My general philosophy with apps design</h2>
As far as I can tell, there are two main competing visions for how to use apps in Django:
<ol>
<li>Apps should be self-contained groups of primitives (models &amp; business logic) that are logically independent, as quoted above.</li>
<li>Apps are a dumb idea for the vast majority of codebases and introduce a bunch of unnecessary friction, to be avoided at all costs. [^1]</li>
</ol>
I think I land somewhere in the middle, but leaning towards the former camp. My heuristic is roughly: is this code something that I would want to break out into its own codebase or deploy target at some point?
Here&#39;s a top level overview of the apps Buttondown has with some high-level metadata; keep scrolling for more in-depth information.
<table>
<thead>
<tr>
<th>app</th>
<th>lines of code</th>
<th>number of models</th>
<th>year introduced</th>
</tr>
</thead>
<tbody><tr>
<td>api</td>
<td>2000</td>
<td>1</td>
<td>2018</td>
</tr>
<tr>
<td>checker</td>
<td>1300</td>
<td>4</td>
<td>2021</td>
</tr>
<tr>
<td>email_address_validation</td>
<td>1500</td>
<td>3</td>
<td>2018</td>
</tr>
<tr>
<td>emails</td>
<td>30000</td>
<td>37</td>
<td>2016</td>
</tr>
<tr>
<td>events</td>
<td>1400</td>
<td>1</td>
<td>2017</td>
</tr>
<tr>
<td>flags</td>
<td>500</td>
<td>1</td>
<td>2021</td>
</tr>
<tr>
<td>marketing</td>
<td>500</td>
<td>2</td>
<td>2016</td>
</tr>
<tr>
<td>markdown_rendering</td>
<td>1000</td>
<td>5</td>
<td>2019</td>
</tr>
<tr>
<td>monetization</td>
<td>3500</td>
<td>11</td>
<td>2020</td>
</tr>
</tbody></table>
<h2 id="api"><code>api</code></h2>
Buttondown has an <a href="https://docs.buttondown.email/api-reference/introduction">external-facing API</a> with the long-term goal of exposing all important parts of the app&#39;s functionality for programmatic access. (The somewhat silly Platonic ideal I strive towards: someone should be able to build out their own Buttondown client just with the API surfaces I provide.)
As such, I carve out that API as its own app. This app doesn&#39;t own many models in of itself — except a fairly generic <code>APIRequest</code> model which tracks incoming requests and outgoing responses — but acts as a superstructure for various primitives owned by <code>emails</code>, owns the schema validation &amp; <a href="https://www.django-rest-framework.org/api-guide/views/">DRF views</a>, that sort of thing.
If I had a magic wand and/or infinite time, this app would probably be shaped a little differently: more like a series of middlewares and decorators that, when applied to other apps, could expose those apps to the external API.
This is a nice theoretical exercise, but the ROI on such work feels very low. I know apps are mostly about model separation, but I think &quot;separation of concerns&quot; (mushy of a phrase as that might be ) is particularly valid, and has served me well here.
<h2 id="checkers"><code>checkers</code></h2>
One of the internal frameworks I really loved at <a href="https://stripe.com">Stripe</a> was &quot;checker&quot;, which was a very pleasant DSL for declaring programmatic scheduled invariant checks in your code. This is my shamelessly reappropriated version of that framework, and its proven so invaluable I&#39;m surprised that it&#39;s not more ubiquitous.
The core of this app is a decorator, <code>@register_checker</code>, which takes a function that returns <code>CheckerFailure</code> objects and does a bunch of metaprogramming to email (or page) me whenever a list returns something.
Sometimes I use this for administrative tasks, like manually auditing new accounts who connect Stripe accounts with pre-existing customers &amp; charges:
<pre><code class="language-python">@register_checker
def no_stripe_accounts_need_auditing() -&gt; Iterable[CheckerFailure]:
 for newsletter in Newsletter.objects.filter(
 paid_subscriptions_status=Newsletter.PaidSubscriptionsStatus.NEEDS_AUDITING
 ):
 if newsletter.stripe_account:
 yield CheckerFailure(
 text=f&quot;Stripe account {newsletter.stripe_account.account_id} needs auditing&quot;,
 subtext=f&quot;&quot;&quot;
 Newsletter: {newsletter}
 Link to Stripe: https://dashboard.stripe.com/connect/accounts/{newsletter.stripe_account.account_id}
 Admin URL: https://admin.buttondown.email/admin/emails/newsletter/{newsletter.id}/change/
 &quot;&quot;&quot;,
 data={&quot;newsletter_id&quot;: str(newsletter.id)},
 )
</code></pre>
Other times, I use it for checking to ensure that no emails are in a problematic state space that necessitate re-driving or SMTP shenanigans:
<pre><code class="language-python">@register_checker(
 severity=Checker.Severity.HIGH, cadence=Checker.Cadence.EVERY_TEN_MINUTES
)
def no_emails_stuck_in_flight() -&gt; Iterable[CheckerFailure]:
 for email in fetch_relevant_emails():
 text = f&quot;Email {email.id} (from {email.newsletter.username}) is stuck in flight&quot;

 expected_receipts = calculate_expected_receipts(email)
 if not expected_receipts:
 continue

 actual_receipts = EmailDeliveryReceipt.objects.filter(email=email)
 # &#39;high&#39; is the queue used for asynchronously_send_email_to_recipients.
 queue = django_rq.get_queue(&quot;high&quot;)
 current_backlog_size = queue.count

 subtext = (
 f&quot;Expected {expected_receipts.count()} receipts &quot;
 f&quot;but only received {actual_receipts.count()}. &quot;
 f&quot;Current backlog: {current_backlog_size}&quot;
 )
 yield CheckerFailure(
 text=text, subtext=subtext, data={&quot;email_id&quot;: str(email.id)}
 )
</code></pre>
(I really, really want to open-source this, and probably will at some point this year.)
<h2 id="email_address_validation"><code>email_address_validation</code></h2>
The <code>email_address_validation</code> app acts as a simple interface that takes strings and returns a <code>ValidationResult</code> object associated with them. That result is compiled from a number of heuristics, from regular expressions to internal data to external services like CleanTalk and Mailgun.
I&#39;m pretty happy with this interface! I originally came to the approach with the concept of one-day deploying this as a stand-alone SaaS, and while my appetite in doing so has largely abated (feels like a lot of boilerplate for a fairly small incremental bump in revenue) it&#39;s made the rats&#39; nest of logic well-encapsulated.
<h2 id="emails"><code>emails</code></h2>
<code>emails</code> is, as one might guess from the name, the oldest and largest app in the codebase. It contains the three most important models in Buttondown — <code>Subscribers</code>, <code>Newsletters</code>, and <code>Emails</code> — plus another 34 to boot. By default, most logic ends up here; it is the sun of Buttondown&#39;s little heliocentric universe, as you might have surmised from that summary table above.
I&#39;ve idly mused on what splitting up <code>emails</code> would look like, and haven&#39;t come up with any satisfactory answers. The logic here is tightly coupled, and decoupling feels very low-ROI.
<h2 id="events"><code>events</code></h2>
Buttondown generates a lot of events from third-party applications. I store data points for every attempted (and successful) email delivery in order to both surface analytics to newsletter authors and to track internal heuristics like particularly spammy authors or slow-to-respond domains. That&#39;s not even getting into opt-in functionality like click or open tracking which leads to even more events.
Plus, Buttondown connects to a swath of ESPs (Mailgun, AWS, and Postmark to name a few) which means the interface for events between providers has to be relatively consistent. Enter this app, which stands up a bunch of webhook routes and munges responses into a generic <code>EmailEvent</code> object that contains pointers to other models.
(Also, shout out to <a href="https://github.com/anymail/django-anymail">django-anymail</a>, which makes this process much easier.)
A not so fun fact: I&#39;ve titled this section <code>events</code>, but the name of the app is actually <code>mailgun_events</code> — Mailgun was Buttondown&#39;s first ESP, so it made sense at the time. Mailgun&#39;s now only processing around 5% of Buttondown&#39;s email traffic. Let this be a lesson to you all!
<h2 id="flags"><code>flags</code></h2>
For a decently long amount of time, I used <a href="https://github.com/django-waffle/django-waffle"><code>django-waffle</code></a> to manage flags and switches for Buttondown. I try to avoid phased rollouts in the &quot;1% of traffic, then 10% of traffic, then 25% of traffic&quot; vein except for very fraught situations, but I find these primitives useful in two scenarios:
<ol>
<li>Pieces of functionality like &quot;make sure this specific newsletter returns a 301 instead of a 300 because I signed a lucrative annual contract with them&quot; which I really don&#39;t want to enshrine at the database level</li>
<li>Circuit breakers like &quot;turn off all access to <a href="http://cleantalk.org">CleanTalk</a> because they started returning a bunch of timeouts and I don&#39;t want it gumming up the works of synchronous API requests.&quot;</li>
</ol>
I decided to end up ditching <code>django-waffle</code> and focus on a lighter-weight flags system to reduce my third-party dependency surface area a bit and, more importantly, to reduce the number of database queries I&#39;d need for some particularly chokepoint-y areas like email rendering or address validation.
I&#39;d like to open source this app at some point: it&#39;s very lightweight and conceptually simple.
<h2 id="marketing"><code>marketing</code></h2>
If I could really wave a magic wand, I&#39;d have this app not exist at all: <a href="https://twitter.com/tylertringas/status/1250521285630836741?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1250521285630836741%7Ctwgr%5E%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.placemark.io%2Fpost%2Fwhy-placemark-switched-the-blog-marketing-to-webflow">conventional wisdom</a> these days is to have your marketing site be in a different codebase entirely, and to have your core application be on <code>app.domain.com</code>.
Unfortunately, I was not conventionally wise in 2016, and at this point it&#39;s not so trivial for me, since <code>buttondown.email/&lt;username&gt;</code> is where most folks&#39; archives live and I&#39;d need to do some tricky routing shenanigans.
So what do I do instead? I serve the marketing pages in boring ol&#39; Django, and store the code for them in a standalone app so the views and routes are at least in their own little world. There are a couple models here — namely <code>Updates</code> (which powers a little widget in the login page letting folks know about new features) — but data and business logic is thin-to-non-existent here.
<h2 id="markdown_rendering"><code>markdown_rendering</code></h2>
Buttondown is a <a href="https://buttondown.email/features/markdown">huge fan of Markdown</a>, and has quite a bit of dedicated logic to make it play nice: I&#39;ve got custom extensions for tables, footnotes, highlighting, smart embeds (for things like Instagram and YouTube, whose oembeds are not email-friendly due to their use of iframes), and much more.
This all culminates in a single method meant to be the external &#39;interface&#39;: <code>render</code>, which takes a string and a <code>RenderingTarget</code> (either &quot;for the web&quot; or &quot;for email&quot;).
This has been a really nice encapsulation of business logic &amp; function. While I don&#39;t see myself open-sourcing or spinning off a dedicated rendering instance or anything like that, it makes it very easy to dive into rendering esoterica and it leads to very maintainable code.
<h2 id="monetization"><code>monetization</code></h2>
This app is a bit of a misnomer, since it sounds like it might pertain to either managing paid newsletters or managing paying users. It&#39;s actually even more simple than that: it&#39;s a series of webhooks and models meant explicitly to replicate Stripe&#39;s state space in my own database.
This app explicitly doesn&#39;t have any business logic; any operations like &quot;mark a subscriber as churned once a <code>StripeSubscription</code> is updated&quot; or &quot;create a new subscriber when a new <code>StripeCustomer</code> is created&quot; are handled within <code>emails</code>.
(I&#39;m aware of the existence of <a href="https://github.com/dj-stripe/dj-stripe"><code>dj-stripe</code></a>, which seemed...a little cumbersome and confusing the two times I tried to onboard to it. Building out my own equivalent probably took more time than it was worth, but I appreciate the gradually revealed complexity and it felt like a particularly important part of the database to treat with care.)
<a href="https://twitter.com/thesephist/status/1534629659144597505?s=21&t=A3to3cCL6mhCoCjaeXvHrA">Linus asked</a> to hear more about why do this at all, let alone with a bespoke app. A couple reasons, in descending order of magnitude:
<ol>
<li>Complexity. Buttondown cares about Stripe a great deal more than the median SaaS app. While its billing model is <a href="https://blog.buttondown.email/2022/05/13/repricing">fairly simple</a>, it also uses <a href="http://stripe.com/connect">Stripe Connect</a> to manage paid subscriptions for hundreds of other newsletters, meaning that there are a lot of subscriptions, plans, customers, and so on.</li>
<li>Speed. Not only does Buttondown care about Stripe a great deal, Stripe is often in the critical path of customer-facing transactions. Billing flows in general can afford to be slow: it doesn’t really matter if it takes ten seconds to generate an invoice view that’s hit by some back-of-office administrators a couple times a day. However, it’s important that Buttondown can answer questions like “when did this paid subscriber pause their subscription?” or “how many new customers did this newsletter get last week?” relatively expediently, because these are core flows! And where core flows are concerned, network calls to <code>api.stripe.com</code> are painful, let alone having to deal with some of the pagination and eventual consistency tradeoffs Stripe has enshrined.</li>
<li>Connect. Certain queries I want to keep track of from an operational perspective are literally impossible in Stripe at the moment, such as “give me all disputes across all connected accounts in the past 24 hours.”</li>
</ol>
In terms of how I actually model this within Postgres: I try to respect foreign key relationships, and plop the rest of the data in <code>JSONB</code>. For example, here’s the <code>StripeSubscription</code> model:
<pre><code>class StripeSubscription(BaseStripeModel):
 # Core information.
 id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
 data = models.JSONField(default=dict)

 # Relevant dates.
 creation_date = models.DateTimeField(auto_now_add=True)
 modification_date = models.DateTimeField(auto_now=True)

 # Stripe-side primary key.
 subscription_id = models.CharField(max_length=100, unique=True)

 # Foreign keys.
 customer = models.ForeignKey(
 &quot;StripeCustomer&quot;,
 on_delete=models.CASCADE,
 related_name=&quot;subscriptions&quot;,
 )
 account = models.ForeignKey(
 &quot;StripeAccount&quot;,
 on_delete=models.CASCADE,
 related_name=&quot;subscriptions&quot;,
 )
 plan = models.ForeignKey(
 &quot;StripePlan&quot;,
 on_delete=models.CASCADE,
 related_name=&quot;subscriptions&quot;,
 )
</code></pre>
<h2 id="conclusion">Conclusion</h2>
That&#39;s the full list! I subscribe to a pattern that I suspect is fairly common amongst codebases of Buttondown&#39;s size: one overstuffed, gross &quot;primary&quot; application and a bunch of orbiting, comparatively better-designed ones.
I like using apps: they make my brain happy in the same way cleaning up test code makes me happy (which is to say, sometimes the happiness is worth the lack of obvious business value.) That being said, I&#39;d be warier of being over-aggressive with app usage as opposed to being over-conservative; I&#39;ve lost afternoons to chasing down cross-app migration issues whereas besides some very long <code>models</code> lists I can&#39;t point to any specific footguns I&#39;ve stumbled upon from having a Very Big Django App.
Please let me know if you have any questions — I&#39;m <a href="mailto:justin@buttondown.email">all ears</a>!
[^1]: As someone who&#39;s lost five hours of their life to shenanigans with cross-app migration squashing, I can certainly sympathize with some of the histrionics of this world-view.
[^2]: Well, and a bunch of front-end code, but that&#39;s out of scope for this post. Also, my front-end code is much more poorly organized than my back-end code.

How I organize Buttondown's Django applications

The journey of 37,000 lines of static types begins with a single annotation.

Blog — Engineering

Buttondown's engineers, writers, and biggest fans write about the whole thing: engineering challenges, useful tips and tricks, new features, and more.

Feast your eyes upon the latest post:

Announcing django-typescript-routes

Open-sourcing our Python-to-Typescript route generator

Or perhaps peruse the archive:

Ready for a better newsletter?