123dev #41: Skills, stories, and software every dev should know
Hi,
This will be the last issue that goes out through Buttondown. After this week I will have completed my transfer of all posts over to Revue and I will be sending future issues through their service. You do not need to re-subscribe or change anything. The only difference you should notice is the sender information will change. I will also take down the current static site for 123dev.email and redirect it to the Revue landing page.
via reddit
Comments
BGP
I heard the GIF came from what happened when Facebook ran the following command.
ip link set fb0 down
Thankfully they were able to fix it with
sudo reboot
Thinking
If you’re overthinking you should write. If you’re underthinking you should read.
Written word is much slower than spoken word. It helps to slow down and be deliberate when you have lots of ideas in your head.
When I’m “underthinking” I like to ask people questions, lots of questions. Reading lets me dwell on words longer. I can really slow down and take my time to have more thoughts about fewer words which often will cause me to overthink.
Links
I try not to make this newsletter about current events, but it’s hard to avoid some of the lessons learned from this week. The first sentence in the last paragraph says so much (emphasis mine)
We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making.
I think this may be a lesson in the fact that once your application and organization gets to a sufficient size the only way to properly test failures is through practices like chaos engineering. Unknown unknowns are the things you can’t plan for.
More details about the October 4 outage - Facebook Engineering
Now that our platforms are up and running after yesterday’s outage, we are sharing more detail on what happened and what we’ve learned.
If you were one of the many people who learned about BGP this week I liked Julia’s practical list of tools and links anyone can use to explore BGP. It’s a good set of commands to know because there tends to be a large BGP inspired outage once every 5 years or so.
https://jvns.ca/blog/2021/10/05/tools-to-look-at-bgp-routes/One of the fallacies of distributed systems is that the network is reliable. This article has lots of examples of networks being unreliable even without human intervention.
https://queue.acm.org/detail.cfm?id=2655736