Hey folks 🎉
what a week! The anniversary issue last week was a blast, looking through a year of serving this community. The giveaway is going strong as well. Some exciting things are going on in machine learning. Let's dive right in!
The Latest Fashion
Scikit-Learn is approaching v1.0! There's a release in progress that you can sneak peek even.
I regularly read Morningbrew, no surprise. But the Emerging Tech newsletter they publish had a fascinating piece about the recent Foundational Model debate and AI ethics.
I have a special place in my heart for forensics. How about forensic data science to uncover fraudulent data in a 2012 paper? It's about honesty when signing a statement before or after providing information in the influential journal PNAS. This article goes into so much detail about their investigation. Trust me, it's incredibly interesting to read.
My Current Obsession
Look... I'm obsessed with the Newsletter giveaway. I've always wanted to do a giveaway to the data science community, and now I have the funds and a reason. Welcome to all the new joiners, and thanks again to all the veteran readers! More info below!
My talk proposal to Pydata Global 2021 got accepted! The title is: How to guarantee no one will understand your machine learning project. So I will prepare a talk on communicating and delivering your machine learning projects. So excited!
The deadline to send in my keynote is also approaching. Still trying to improve the script at this point. The main idea is to relate how we could advance AI in geophysics by looking at self-driving cars. That should be fun.
Thing I Like
I just got a new bed and mattress. It's the best. I feel like I'm finally getting some restful sleep. It's a hybrid mattress from Simba, similar to this one. As they say, spend money on the things that separate you from the ground:
I am giving away $165 in AWS credit, five books of your choice, and 15 access codes to one of my Skillshare courses. I was honestly nervous that no one would be interested, but that was obviously unfounded. There are now over 1800 tickets entered, and it's not too late to join for you:
Ken Jee was nice enough to share the giveaway. Let me know if you came from there!
Hot off the Press
I spent some time building a page for the books I was involved in publishing. Seems like this list is slowly big enough to warrant its own place on the internet.
I wrote a small piece about personal finances with wise.com, where you can keep budget categories in low-cost index funds now. I love it.
I don't typically include Tweets, but my tweet about Hugging Face now playing well with online automation tool Zapier has picked up a bit of attention. That was nice!
Machine Learning Insights
Two weeks ago, I shared the Stanford paper on Foundational Models. Let's talk about them!
The idea of Foundational Models is essentially large language models that are used and reused again and again. It has become effortless to use large pre-trained models from BERT to GPT-J trained on datasets and with hardware that would not be available to us mere mortals. These models have not only been attractive in Natural Language Processing but have found wide adoption across machine learning applications, including satellite imaging, Visual Transformers for computer vision, and of course, biology. Basically, if your work in a domain of expertise, there's probably a transformer applied somewhere.
Transformers are eating the world.
The big problem with these foundational models is ethics. GPT-3 was trained on half a trillion words mainly gathered from the internet. GPT-2 before that was very obviously trained on Reddit data and in question-answer sessions on Earth Science would regularly venture into Flat Earth territory. In social justice issues, these models often inherited hateful biases from the unfiltered input. This type of bias gets reiterated and usually amplified if we use a model like this as a foundation for almost every application possible.
(I talked about transformers before, here's my favourite guide to understanding them.)
Critics of the foundational model have said that it's a publicity stunt, which honestly it kind of was. I like the notion of foundational models. However, merely a week after, Standford announced their new group that will focus their research around foundational models. When I shared the paper with 100 researchers only from Stanford across different groups, I wondered what this would bring along. Collaborations like these are seldom altruistic at their core, especially when it's so obviously only Stanford researchers.
The paper has more problems, including citation farming, especially since 100 authors are on this 200-page paper. This should have rather been a book where chapters can be cited (like my book chapter). These models were formally defined as large pre-trained models that are then used in transfer learning, a common technique in machine learning applications, so not a novel concept per sé. Missing actual foundational models like ResNet-50 is another problem. A model that is the foundation of almost any modern computer vision application. Professor Malik from Berkeley had a pretty strong response, which critiques that these transformers are not grounded in causality. Especially in language, this is a problem, and I've often heard actual linguists like Rachael Tatman talk about the importance of causality in language over the pure correlational approach of current NLP.
Interesting developments overall. Foundational models seem interesting but need to include broader definitions of models. The ethical complications with either type of model have been discussed widely and are worth investigating, e.g. hate speech in language models and non-consensual pornography in vision data sets.
Question of the Week
- What is your opinion on Foundational Models?
Send me your answers or post them on Twitter and Tag me. I'd love to see what you come up with. Then I can include them in the next issue!
Tidbits from the Web