Develop faster with Python 3.12, multi-process speed profiling, polars-business and sequencing sourdough DNA

Further below are 5 jobs including: Data Scientist Insights (Customer Experience & Product) at Catawiki, Permanent, Amsterdam, Senior machine learning engineer - NLP/DL, Senior Data Engineer at Hackney Council, ML Researcher at Velo (stealth startup), Senior Data Engineer at Hertility Ltd, Permanent, Remote

Python 3.10 and the just-released 3.12 help pinpoint errors with more useful exception message and this can improve your software development process. I give an example below. If you're tackling multi-core programming and you want to understand which bit is slow - check out VizTracer below.

I also give an example of polars-business for date/time logic and, totally off topic, I talk a tiny bit about DNA sequencing in sourdough yeast starters and custom mechanical keycaps.

Develop faster in Polars via better error messages and a couple of leadership resources

Further below are 5 jobs including: Senior machine learning engineer - NLP/DL, Senior Data Engineer at Hackney Council, ML Researcher at Velo (stealth startup), Senior Data Engineer at Hertility Ltd, Permanent, Remote, Software Engineer at Qualis Flow Ltd

I've started work on a Polars course, I'd love to hear if and why you'd like to learn Polars. I'm going to document some of my findings and I start below with a to_datetime observation plus a Pandas helper. I got to discuss some of this during a recent Software Engineering for Data Scientists course, I share a few insights below, notably on getting ChatGPT to write unit tests (not my recommendation, but it certainly can get you started).

I've also kicked off my RebelAI leadership group with an initial group of excellent data science leaders. I've posted a couple of resources that came from our kick-off chat below. If you're in a leadership role and joining a peer-support leadership group might help you, just reply to this and we'll have a chat.

Completing our charity car drive, LLM lies, teaching logic to kids, why commenting on GitHub is a Good Idea

Further below are 5 jobs including: Senior Data Engineer at Hertility Ltd, Permanent, Remote, Software Engineer at Qualis Flow Ltd, Scientist/Engineer for Machine Learning (multiple vacancies), Senior Data Scientist at CEFAS, Permanent, Manager Data Science - Business Analytics at Catawiki, Permanent, Amsterdam, The Netherlands

So I'm coming up to the end of my several-month break and shortly will be back onto some proper data science work. Taking a head-clearing break has been amazing. I've shared some notes and photos below on the charity fund-raising old-car rally I was on recently (we raised a ton of cash for charity - thanks for your support!), currently I'm surfing in Cornwall and then over the next couple of months I've got updated forms of my training courses to run privately and in public. The Rebel AI leadership group also kicks off in a month.

Further below I talk on a couple of LLM and open source topics that I've been keeping an eye on.

Some more thoughts on Polars and Dask

Further below are 7 jobs including: Software Engineer at Qualis Flow Ltd, Scientist/Engineer for Machine Learning (multiple vacancies), Senior Data Scientist at CEFAS, Permanent, Data Scientist at Arena.Online, Permanent, Droitwich (Worcestershire) (Ian's note - Arena sent me a flower bouquet as a thank-you and they were nice and safely delivered, cheers!), Manager Data Science - Business Analytics at Catawiki, Permanent, Amsterdam, The Netherlands, Analytics & Digital Data Scientist at JLR, Permanent, Gaydon UK, 3-6 month Applied AI Residency at Gradient Labs

Recently I ran a private Higher Performance Python course, my talk with Giles Weaver (LI) at PyDataLondon 2023 on Polars and Pandas 2 and Dask was on my mind, I share some further observations on these below. I've also released new dates for my public courses for November, early bird tickets are available for each of these if you're quick.

In just 10 days I'm off on my charity car adventure to Venice and back, there's another photo below (and no, this car still hasn't tried to melt itself, unlike the last one - phew) and I'd love it if you considered donating please to our Parkinson's Research charity. Details at the end. Many thanks to those of you who have generously donated already, we're well over our initial target and that's down to you kind souls - many thanks.

Reflections on Polars and Pandas 2

Further below are 7 jobs including: Scientist/Engineer for Machine Learning, Senior Data Scientist at CEFAS, Data Scientist at Arena.Online, Manager Data Science - Business Analytics at Catawiki, Senior Software Engineer at Qualis Flow Limited, Analytics & Digital Data Scientist at JLR, 3-6 month Applied AI Residency at Gradient Labs

I'll share some thoughts on Polars and Pandas 2 (with Dask to follow in the next newsletter) based on a talk I gave at a couple of conferences recently. Further below I share an update on the charity car drive (including the engine-melted Volvo) several of us are doing in September in aid of Parkinsons Research. At the end I've got a new Data Science/Eng Salary Guide, if you're looking for a job or hiring there's some useful detail there - that's right before this edition's 7 jobs.

New public training courses in November

Lessones learned at the Executives at PyData discussion session from PyDataLondon 2023

Further below are 3 job roles including Senior roles in DS and generative DNNs at organisations like QualisFlow, Jaguar Landrover and Gradient Labs

This issue is rather delayed. I've switched gears, taking time to focus on other projects for a few months including a charity banger-car driving event (MotoScape Car Rally - below) before tackling the new RebelAI leadership community that I've mentioned in past issues.

The car below is our chosen "banger" for the charity car drive this September (this is the one I didn't blow up), details below if you want to hear a silly story.

You might want to pay attention to Polars - our first thoughts

Further below are 4 jobs including Lead and Senior Data Science positions at companies like JustEat and ECMWF…

Below I talk on my Rebel AI leadership group (could this support you? I’d like to hear!), some schedule details for PyDataLondon 2023, my first thoughts on Polars over Pandas 2 (Arrow + Query planner == great!), Martin’s article on “the Alan Turing institute has failed”, my JustGiving endeavour for later in the year and updates on DirtyCat, DTreeViz, SpaCy and Numba.

Rebel AI leadership group (a forthcoming endeavour)

New Rebel AI leadership group, PyDataLondon conference and SW Engineering tools for you

Further below are 5 jobs including Senior/Lead/Head Data Science and Data Engineering positions at companies like JustEat, CataWiki and MadeWithIntent

I'll talk below on a new endeavour called "Rebel AI" that I'm building for excellent data science leaders, give you some reasons to attend our PyDataLondon conference, list some more tools for the software eng side of data science and share some updates on useful open source tools.

Rebel AI leadership group (a forthcoming endeavour)

Observations from last week's Software Engineering course and a pile of new Open Source tools for you

Further below are 7 jobs including Senior/Lead/Head Data Science and Data Engineering positions at companies like Number 10 (Downing Street), JustEat, CataWiki, ECMWF and MadeWithIntent, Coefficent Systems

Thoughts from my recent Software Engineering for Data Scientists course

Last week I ran a public iteration of my Software Engineering for Data Scientists course - it focuses on people who use Notebooks and don't yet write tests, moving everyone towards a collaborative critique mindset to diagnose "bad" Notebooks, add unit tests and DataFrame tests, extract useful code into a folder structure and finally to review a bunch of "weird" Pandas outcomes. As usual this led to spirited conversation, feedback on good process and lots of new tools and techniques to take back to the office. I don't tend to write-up the discussion that occurs in these sessions and I figured I'd try that experiment. If you'd like to attend the next one, or to have a private iteration in your group, just reply to this.

A new book on feature engineering, imminent Pandas 2 and more

Further below are 4 jobs including Senior Data Science positions at companies like Made with Intent and Coefficient Systems...

Soledad Galli has written the 2nd edition of her book on feature engineering - it is aimed at anyone looking for new inspiration with a focus on tabular data. I give a review below. I've also noted several new open source packages including the imminent Pandas 2, a new version of Modin (to accelerate Pandas), Redframes for simpler Pandas work and an update to Altair.

Software Engineering for Data Science course in a month

“Asking good questions about data science projects”, ChatGPT hallucinations, ArcticDB, and Pandas in the browser

Further below are 7 jobs including Senior/Lead/Head Data Science and Data Engineering positions at companies like Coefficient AI, MadeWithIntent, ECMWF, Global Canopy and Ocado…

Apologies to be slow on this issue - once again (infant == MASSIVE_TIME_SUCK) # True.

I see on The Guardian that our UK government is investing 900M GBP in a new supercomputer plus annual 1M GBP awards for groundbreaking AI research. It is nice to see us starting to play catch-up, after so much dismal news in the UK (thankfully this week’s Budget statement didn’t send the market into a dive, unlike the last one) seeing some focus on AI innovation in the UK feels welcome.

Notes on Python 3.12, new matplotlib features and some new open source tools

Further below are 7 jobs including Senior and Research Data Science and Data Engineering positions at companies like (the very lovely) Coefficient Systems, Ocado and the Metropolitan Police

We had a great PyDataLondon meetup last week including a talk on deep learning for tabular data (it still isn’t beating GBTs) and a talk by my friend Nick on building cross-functional teams.

He talked about turning ML project design sessions from defining hard (and possibly-impossible) questions like “we shall decrease churn by 5%” into derisking questions like “can we understand what drives churn such that we might decrease it in a later iteration”, to remove the risk of being pinned to a hard metric before any work is done. He also spoke on defining both “progression criteria” (to figure out if you’ll keep iterating) and “completion criteria” (I have a “definition of done” in my Success course) so you know when you can stop and add value to the organisation.

Tips for fixing Data Science projects from our Global conference, valuation, rubber ducking

Further below are 9 jobs including Senior and Research Data Science and Data Engineering positions at companies like (the very lovely) Coefficient Systems, Ocado and the Metropolitan Police

Slowly I catch up on my silly backlog. Infants truly suck all of your time (I'm finding the 2+4+6am interruptions to be the most costly right now, sigh). But they're also so much fun! Watching my child try to reason through whether the thing up ahead is a train or a tram (pantograph? 2 rails? carriages? what's the difference!) is wonderful.

Our monthly PyDataLondon meetup (RT it) is on next Tuesday, my buddy Nick will be speaking on "functional team embedding" (he's got a ton of experience between running teams at Google, Facebook and startups) - you should attend.

Faster Pandas, Dask, compilation and dataset annotation improvement

Further below are 6 job roles including Senior roles in DS and DEng and Research at organisations like Ocado along with a tender offer for a consultancy collaboration with Global Canopy.

Apologies for the gap since the last newsletter. Christmas, illness and an infant makes for a potent combination. I’m now catching up on a host of backlogged reading which I look forward to sharing. I’ll keep this issue a bit lighter.

Open Source updates

Patterns for successful projects and a bumper load of text-fixing OCR tools

Further below are 6 job roles including Senior roles in DS and DEng at organisations like Causaly and Medidata.

PyDataGlobal ran a week back with several thousand online attendees and several days of long-tracks. If you weren’t a ticket holder then you’ll have to wait until the start of January for the general video release (ticket holders got an email for video access for December). The schedule was pretty varied and talks were of high quality.

“Data Science Project Patterns that Work” at PyDataLondon 2022

Further below are 3 job roles including Senior roles in Data Science and Data Engineering at organisations like Causaly and MDSol.

Tomorrow at 4-5pm (Wednesday, UK hours) I give an Expert Briefing for PyDataGlobal on The State of Higher Performance Python. Please join if you’d like an open q&a around higher performance topics - you’ll need a ticket to attend. Next week I talk on Data Science Patterns that Work a couple of hours before I run another Executives at PyData discussion session for leaders.

Below I talk some more on a worrying trend with deep language models and share some nice visualisations for correlation that’ll help your team understand the pitfalls of “just using one number to explain a rich space”.

PyDataGlobal’s just starting

Specialist explains why your Python 3.11 run-times are faster

Further below are 3 job roles including Senior roles in Data Science and Data Engineering at organisations like Causaly and MDSol.

Below I have notes on the reasons and a diagnostic for speed-ups in Python 3.11.

Executives at PyDataGlobal for Dec 1st

Do you want your Python code to run faster? Good news! You’ll soon get easier debugging in sklearn too

Further below are 5 job roles including Senior roles in Data Science and Data Engineering at organisations like Causaly, Cultivate and MDSol.

Two issues back I spoke a little on the upcoming Python 3.11 release, that release is now out and I give a little benchmark below - the speedups for no work on your behalf are very nice.

Do you use scikit-learn with Pandas? If so there’s news of an upcoming nice change below where Transformers (e.g. the preprocessing tools) will maintain DataFrames rather than converting down to numpy array objects which will ease debugging. I’m also experimenting with mamba in place of conda, notes further below.

Learn the scaling secrets of a 1-person ML stock picking engine

Further below are 6 job roles including Senior roles in Data Science and Data Engineering at organisations like Causaly, Cultivate and Trust Power.

Further below I pull out some insights from the a review article on a 1-person ML stock picking engine (which self-reportedly was good) for tools for scaling and share some notes on the latest HoloViz visualisation tool.

I'm looking forward to getting along to the next PyDataLondon meetup on Tues 1st November at the usual venue, it'll be good to talk to folk face to face again (well, I'll be masked, but talking).

Big speed-ups for pure Python code coming soon with Python 3.11

Further below are 5 job roles including Senior and Lead roles in DS and DEng at organisations like Causaly, a gambling start-up and an eco power firm

Below I talk on some significant speed-ups and parallelisation opportunities we’ll see in new versions of Python, refer to a nice faster-CSV-solution for Pandas and share some new tools which let you embed Python in Excel and tidy up your older-style Python code.

Python 3.11 will introduce some significant speed-ups, 3.12 has even more changes

Sourdough by Experiment, Squirrel Squadron

Further below are 3 job roles including Senior roles in DS and DEng at organisations - if you’ve got a job to advertise, just reply to this newsletter

I’m on holiday, surfing down in Cornwall. For this issue I’m taking a lighthearted reflection on a couple of years of acquiring a new skill - sourdough baking - with some observations on how it applies to data science (‘natch):

How do you build a solid backlog and derisk your novel projects (Part 2)?

Further below are 7 job roles including Senior, Staff and Lead roles in DS and DEng at organisations like Deliveroo and Trust Power and Kew Gardens

I’ve got the second part of the write-up from my recent Success call for DS leaders on “backlogs, derisking and estimation”. The first half of the write-up is in the last issue. I discuss the updated nbdev v2 further below for potentially faster Jupyter based R&D.

I’ve had to rejig my training dates, only the Higher Performance Python course is available at the end of October for sale right now due to increased client demands for strategic support (I help teams build and execute a plan when they’re under pressure to deliver). Sorry if that’s a pain, I plan to run my Successful Data Science Projects course a few months later, possibly in January. Please reply to this email if you’d like a notification about this.

How do you build a solid backlog and derisk your novel projects?

Further below are 7 job roles including Senior and Staff roles in DS and DEng at organisations like Deliveroo and Regulatory Genome and Kew Gardens

In the last issue I talked on Building Successful Data Science Projects - my well-received talk from PyDataLondon 2022.

Last week I ran a related call for DS leaders on “backlogs, derisking and estimation”. I’ve got a summary of the first half of that call below, the rest will follow in the next issue. Building a solid backlog is critical to success and there’s a ton of tips in the write-up below.

Talking about derisking, building a backlog and estimating Data Science projects and PyDataLondon 2022 Conf videos

PyDataLondon2022 happened 6 weeks back ( schedule ), all the videos are now online alongside all the videos from all the conferences on the PyDataTV channel. Go fill your boots with great content. I’ll talk on my video further below.

Further below are 5 job roles including Senior and Staff roles in DS and DEng at organisations like Deliveroo and Regulatory Genome

In the last issue I talked about the Zoom call I ran recently (thanks to those who joined!) discussing “DS Team Structure and Growth”. On Monday 8th Aug (next week) I’ll run another on making a Backlog, Derisking and Estimation. Reply to this if you’d like an invite, we’ll have open discussion for an hour on a prioritised list of topics. I’ll write-up the notes for the next newsletter plus I’ll share that recording but if you have questions on this topic, you’re best off joining the call (even if just to lurk).

Observations from my Team Structure and Growth Call last week

I noted last issue that I’d dig further into building data science teams with a Zoom call - the notes and YouTube link for that are below, there were 12 of us on the call for an hour representing a range of larger UK companies with mature data science outfits.

Further below are 6 job roles including Senior and Staff roles in DS and DEng at organisations like Deliveroo and Regulatory Genome.

On August 8th I’ll be running the next open call on Backlog, Derisking and Estimation - see details here and sign-up here or reply to this email and ask for an invite (note - this is open to leaders, but not to recruiters).

More lessons from the PyData London "Executives at PyData" session on Team Structure

In the last issue I summarised the main discussion at the Executives at PyData session I co-ran at our PyDataLondon 2022 conference. I'm extending these topics into two free Zoom sessions, the first is this Monday (details below) on team structure, the next is in a month on topics including how-to-derisk-and-build-a-backlog. You're welcome to sign-up and I can send you a GCal+Zoom invite. Below I give some notes on Team Structure and link to some Python 3.11 benchmarks showing off the new speed-up work that we'll enjoy in a few months.

At the conference there was a lot of continued discussion on Team Structure and some of the original participants will be joining for the Monday call. I'll share a link to that recorded call afterwards. If you have questions you'll want to attend, it is free and may help you unlock blockers for your team.

Looking back at my notes from the conference John Sandall (thanks John!) shared this short story on how not to grow a data science team which captures many of the situations I've found myself in during strategy sessions. There's a bullet point list near the top and the last item - "no data leadership" - I'd argue is the root cause. You can't comfortably invest in all the other problems if leadership don't have good experience in what it means to be a data-driven company (i.e. - get the data right, get the questions right, build the simplest things that make change, do the right testing, then add complexity incrementally).

Lessons from the Executives at PyData session at PyDataLondon 2022

We ran our first post-pandemic in-person PyDataLondon conference a week back in central London, bringing circa 400 data scientists and engineers together for a packed and really solid schedule. We had 3 days of events, a social with a very generous bar, spontaneous break-out sessions and lots of lively conversation. It was lovely to see folk again for the first time in 3 years. Personally I've been cautious having become a father during lockdown, so this was my first "big" event since lockdown...it was a bit overwhelming and I attended in chunks, avoided the crowded sections and tried to keep my mask on. I'm really glad I attended, thanks to all those who came, presented and volunteered.

I ran a session for "Executives at PyData" aimed at anyone with a leadership role. We discussed a whole pile of issues, I've written up some notes below. If you're a leader, you'll want to take a look. I'm also scheduling two follow-up Zoom calls (details below) to continue these discussions, one on Team Structure and one on Backlogs & Derisking. Reply if you'd like to attend to get the calendar invite for the Zoom call.

Further below you'll find 8 listings for jobs at the likes of DeepMind, Deliveroo up to Staff Data Science roles.

PyDataLondon - talks aimed at leaders, our great schedule and volunteering

Next week sees the return of our PyDataLondon conference after a 2 year Covid gap. The volunteers and NumFOCUS have put in a huge amount of work to bring this event back and I'm looking forward to attending, speaking and running some sessions. Will you be attending?

The schedule is packed with strong talks including a couple on "getting it right" (one by yours truly), ASR, processing bank data, fuzzy string matching at scale (beyond fuzzywuzzy!), a DataFrame showdown, faking data for testing, measurements and fairness, feature stores, NLP, plenty of Bayesian and talks on MLOps.

I'm very happy to give an updated run of my Building Successful Data Science Projects talk on Sunday, based on several years of my popular course (email me if you'd like a notification for the next course?). We'll look at a mix of successful and "rather less successful" project outcomes from my portfolio and I'll draw out lessons which, if they fit the patterns you see, will help you succeed more frequently. The prior talk in the schedule on Lessons in transitioning from academia to business looks interesting to me, it has a focus on looking beyond models and scores and considering the needs of the business.

Thoughts - PyDataLondon meetup and conference

Next week we have our first PyDataLondon meetup in a couple of years! It’ll comprise a set of lightning talks to get folk ready for the post-meetup discussion. The venue is the same as we used to have (thanks to the Man Group!). Sadly I can’t make it, I hope to be along to the next.

A month later we have our PyDataLondon conference. I’ll be talking on “Successful Data Science Projects” (aka a list of some select past failures, with lessons learned, plus a happy ending) and running another “Executives at PyData” session aimed at leaders. If you’ve got a ticket for the conference you can attend both, I’d love to say hi in person.

I’ve been thinking a lot about past project failures in my consulting - so much of it comes down to human factors (not “algorithmic issues”) and I’m keen to hear more from attendees. Figuring out how to communicate these issues and to bake them into project plans to lessen their impact seems critical to getting more success for data teams (IMHO).

Thoughts

Thanks for the good wishes some of you have sent. We’re all much better now - sleep deprivation is crap (and nasty infant illness is just awful), but everyone is on the mend. So much so that I’ve just been off surfing - w00t! Never is repeatedly falling off of a board so much fun.

Covid in the house passed without consequence, my wife got over hers and by isolating in my roof office she didn’t pass it on (my boy and I were negative all week). We used the 3M mask I’d noted a couple of issues back plus fabric gloves for plate passing. It was a pain. I then got myself an antigen test and it seems I’ve never had it (it states pretty good sensitivity & specificity) which is annoying, I’d hoped I’m been asymptomatic and had just missed it at some point.

So - I’m still wearing masks when out and about and still trying to be sensible. I recommend that you stay safe, especially when in busy places - the long Covid stats are still very unpleasant, see the Independent for ONS stats - 1.2M people in the UK (a non-trivial fraction of folk who had Covid, got 3mo+ long-Covid debilitating symptoms).

Thoughts

This will be a short issue, my infant has been very ill in recent weeks and I confess that sleep deprivation is catching up. I had some great replies to the Coronavirus masks question that I need to summarise, that'll happen in the next issue (when I've had some sleep). I also took an antigen test which says I've never had it (go great masks!), more next time.

There's 15 jobs below including full stack devs, quants, research and senior roles at orgs including greentech, healthcare and DeepMind.

Open source - memray, Ibis, an awful bug report

Thoughts - metrics used whilst “making $1M for my client”

Below I talk on system profiling before trying to make things faster, Covid masks (maybe you have better recommendations?), metrics to monitor research progress to build on my comments in the last issue, extracting Pandas dataframes from PDF tables, Python 3.10 and lowering one’s energy bill by recording data in the house.

Further below there are 13 job ads for data science and data engineering including with DeepMind, Risilience (real interesting!), Monzo, BAMFunds and more.

PyDataLondon 2022 is coming for June 17th. The Call for Proposals is open, we’re looking for sponsors (totally useful if you’re hiring) and tickets are on early-bird for <1 week so get them cheaper if you buy in the next few days. This conference is in-person, with a reduced capacity to make it safer. Reply directly to me if you’re interested in sponsoring.

Thoughts - on “making $1M for a client” and speeding-up Dask

Recently with one of my clients we’ve celebrated finding over $1M in recoverable overbilling and fraud at a large fintech. I’ve reflected on the “data science” (deliberate double-quotes!) journey that we’ve undertaken that quickly got us to this high-value success on a green field project. I’d love to hear if you’ve got anecdotes to share for this sort of process?

Monzo (and others!) are hiring for DS and DEng positions, the job ads are down below as usual. I asked Neal Lathia (Dir. of ML at Monzo) about their hiring process and he pointed me at a public link that details the process. If you want to see a good example of how you might setup your own hiring process - read that link. If you want to work at Monzo then, obviously, do the same (and check the ads below).

For the last issue my link to the Metaculus prediction site was the most clicked link (13% of clicks). A couple of you asked how to go about learning to forecast and for me I’m enjoying see how others reason their way to an answer - I discussed a few in the last issue. On the Will Russia control Kyiv by June challenge (current forecast - 3% likely) there’s a nice comment by quant_speaking reasoning between the June and April challenge to look at the implied daily probability of success as estimated by the community, noting that the implied probability between the two challenges reasonably aligned. I had to spend a little while thinking about the math and found that a fun challenge (I’ve not tried winding a long-forecast to a daily-implied, then pushing it forwards again). This is the sort of comment I’d suggest reading if you want to get your head into these challenges.

Thoughts

Given some of the big events happening in the world I’ve focused more of my time on “estimation”, I’ve made notes on this and the Metaculus site below. I’ve added a brief word on the Ukraine (in short - if you want to do something useful, make a donation to a reputable organisation). I’ve noted Simon Willison’s suggestion to help open source maintainers get support from companies and list the PyDataLondon and PyDataBerlin conference dates at the end.

There are 8 roles further below including Senior DS, data engineers, Principal roles and more.

Ukraine

Thoughts

Below I talk on leading vs lagging indicators to track progress, the lux auto-interesting-relationship discovery tool and share Jesper’s thoughts on the Unshcedule that I recently wrote about.

With one of my insurance clients we’ve been talking about using metrics on R&D projects to signal progress. Whilst we’re doing well on our projects the path is inevitably convoluted. In this keynote talk Bandon Sanderson (prolific author) talks on avoiding “lagging indicators” and using “leading indicators”. In our case “number of cases that need follow-up investigation” would be a lagging indicator (and might not be resolved for many months) but “analyses made this week” is easy to count and communicates progress.

What indicators do you use to measure and communicate progress?

Thoughts

I finished another Successful Data Science Projects course last week, we had some very productive discussions about the whole early-stage through to deployment process and common problems along the way. Not having the right sponsors, or sponsors being unavailable, or not knowing the “bigger” project that other teams are working on were common issues.

We got talking on the ways we lose productivity and how we might get some of it back, I’m sharing a couple of links below on this. If you’d like a notification for my next Success class or any of my training courses just reply to this, I can send you a single email when I have the next date.

Good process

Did you know that Signal AI need a Senior Data Scientist and Aflorithmic need a Data Engineer and Software Engineer. Details for all of these and more are down below.

Thoughts

I reflect on how much useful work I get done in a day, my recent Software Engineering class with a pile of tips for you and share a though on start new high-risk anomaly projects successfully.

David McIver recently wrote about how People Don’t Work as Much as You Think on his newsletter, also hn discussion. David is a smart cookie, author of the hypothesis testing library. He gives an honest opinion of how much “quality work” he gets done in a day. I’ve been using an Unschedule from The Now Habit (see pdf link to print), that shows me that I get 2-3 hours productive work a day around “everything else”. If you’re at all worried about how much time you do or don’t get to focus it is worth a read.

Did you know that Netacea are hiring for a Head of DS + Data Engineer, Signal AI need a Senior Data Scientist and Aflorithmic need a Data Engineer and Software Engineer. Details for all of these and more are down below.

Better data science code and derisking projects

I’ve asked about your processes that improve your workflow. Further below I reflect on my use of Test Driven Development and linting tools and I’ll have some more pytest thoughts to share soon. I also share thoughts for derisking anomaly projects plus I’ve been asked to share the AIQC library - all further below.

Dave Kirby has kindly shared his observations on Obsidian for note-taking - thanks Dave:

Did you know that Netacea are hiring for a Head of DS + Data Engineer, Signal AI need a Senior Data Scientist, Aflorithmic need a Data Engineer and Software Engineer and VivacityLabs have a Special Projects role. Details for all of these and more are down below.

Thoughts and an interview with the author of Polars

Below I have the second part of the interview with Ritchie Vink on his "faster than Pandas" Polars dataframe library - part one came last issue. It is only 2 years old and beats Pandas (sometimes significantly) in many benchmarks. Previously I interviewed Kaggle Competition Expert Mani Sarkar which contained lots of great ML tips.

Building on the "beginner mind" note from the last issue I'll add that I keep a GDoc with high-level notes I've taken from books I've read. I read a lot and whilst lots of little ideas get integrated into my thinking, it is so easy to forget other useful points, especially "easy stuff". I've spent over a year building this doc - it contains bullets of points and is deliberately light. Looking back on it always reminds me of something useful. Do you have any practices to build a second long-term memory that you've found valuable?

Did you know that Netacea are hiring for a Head of DS + Data Engineer, Signal AI need a Senior Data Scientist, Aflorithmic need a Data Eng and 2iQresearch are hiring for a Senior Dev and and a Quant Dev? VivacityLabs have a Special Projects role and Inawisdom need a Software Engineer.

Details for all of these and more are down below.

Thoughts and an interview with the author of Polars

Down below I have my next interview - this one with Ritchie Vink on his "faster than Pandas" Polars dataframe library. It is only 2 years old, beats Pandas (sometimes significantly) in many benchmarks and seems to be gaining traction. I think his developmental style is interesting and I dig into this in some of the interview, it'll air over this and the next issue. Previously I interviewed Kaggle Competition Expert Mani Sarkar which contained lots of great ML tips.

Did you know that Netacea are hiring for a Head of DSci + Data Engineer, Signal need a Head of DSci, Aflorithmic need a Data Eng and 2iQresearch are hiring for a Senior Dev and and a Quant Dev? VivacityLabs have a Special Projects role and Inawisdom need a Software Engineer. Details for all of these and more are down below.

Thoughts

I spent a really interesting hour with Richard Pelgrim of Coiled last week talking on the state of Higher Performance Python and Dask. Richard helps folk on-board to Dask so he had some great observations about blockers and diagnostics that help new users.

I spoke on the usual suspects - line_profiler, memory_profiler, ipython_memory_usage and VizTracer, also Numba, installing bottleneck & numexpr to speed-up Pandas and I briefly mentioned Polars and Vaex. Richard gave a little live demo of some of the latest changes in Dask’s dashboard diagnostics. The video is available if you fill in the access form on Coiled’s blog in the link above.

This newsletter is now NotANumber

I’ve renamed my newsletter from “Ian’s Thoughts and Jobs List” to “NotANumber”. I’m building up this newsletter, recently I’ve started to interview smart people e.g. Kaggle-Mani last issue and I want to get some guest editors involved over time. Hence I’ve taken my name out - I’ll be the editor in chief. Why this name? Lots of reasons, I might reveal them in the future.

Did you know that 2iQresearch are hiring for a Senior Dev and and a Quant Dev? VivacityLabs have a Special Projects role and Inawisdom need a Software Engineer. Details for all of these and more are down below.

NaN==NaN is False

Thoughts

Did you know that 2iQresearch are hiring for a Senior Dev and and a Quant Dev? See their ad and many more below.

We had PyDataGlobal a week back, we had over 2,000 attendees! The schedule was great, videos are available for attendees and the videos will be public in a couple of months.

I ran an Executives at PyData session to discuss pressing issues for team leaders. We discussed how a cross-functional embedded DS team was generally more productive than an isolated team. A thornier topic was "how do we build trust that open src tools will be around in 10 years - we can't buy a service contract on sklearn!". Hopefully NumFOCUS will have a growing part to play in this trust issue.

Thoughts

Did you know that Holiday Extras are hiring a Senior Data Engineer? See their ad and many more below.

runs from tomorrow, the is online. I’m running another session on Thursday afternoon - please join and bring your boss!

Expert Briefings at PyDataGlobal, the GIL and DirtyCat

Did you know that Holiday Extras are hiring a Senior Data Engineer? See their ad and many more below.

I’m starting to experiment more with this newsletter. Some of you know that I have an infant - wee Kai is now 14 months (and we’re just starting nursery). He’s brill, I want to spend more time with him so - I’m going to work on growing this community.

line_profiler fixed and CI help required

Did you now that Kindred are looking for a Lead Data Scientist along with quants and python developers? Details are below along with other Senior roles in the job adverts.

black on Notebooks, Pandas warts, Multi-task RL and Successful Data Science Projects

Did you know that M&S are hiring for several "Senior & Lead Data Science vacancies"? See the jobs listings below for details.

Last issue I noted that scikit-learn v1 was coming but I didn't know how soon - it turns out it is being released in "release candidate" mode right now. On hackernews there's a discussion about this and the general DS ecosystem in Python which somewhat inevitably devolves into a critique of Pandas. If you're at all frustrated with Pandas (not sklearn!) you might find the discussion and alternate libraries interesting - both polars (non-Numpy Pandas-like DataFrame library) and proplot (matplotlib sane-defaults wrapper) get useful mentions.

Do you have any critiques of Pandas to share with me? I'm always collecting war stories with a plan to create an intermediate course to make working with Pandas more efficient and easier to understand.

Jupyter graphical debugger, a bug in line_profiler, a new timeseries forecaster

Did you now that Kindred are looking for a Lead Data Scientist along with quants and python developers? Details are below along with other Senior roles in 12 job adverts.

One thing I teach in my Higher Performance Python class is using in a Notebook to figure out what’s slow in a cell (it runs at the cmd line too). Just recently this , thankfully a bug was reported and a PR is on the way. There’s a simple patch you can apply if this affects you before the PR gets released (it is due to updates in Jupyter and how filenames were being reported which changed the parsing needs for external tools).

Scikit-learn v1.0 upcoming

Did you now that Kindred are looking for a Lead Data Scientist along with quants and python developers? Details are below along with other Senior roles in 11+ job adverts.

Whilst working on a client project last week I started to see some sklearn warnings about named arguments for the upcoming version 1.0. Intrigued I dug a bit further and indeed there's an in-progess changelog noting this change.

The new HistGradientBoostingRegressor, inspired by LightGBM, is indeed very fast. XGBoost has the same trick for precomputed bins so all 3 approaches are a little less accurate but significantly faster (5-10x faster than the default exact method in XGBoost, and the sklearn method is much faster than the equivalent GradientBoostingRegressor). It also handles NaN data and will internally encode categoricals.

Loading parquet data quickly, James Powell's training, Human-Learn

Did you now that Kindred are looking for a Lead Data Scientist along with quants and python developers? Details are below along with other Senior roles in 14 job adverts.

Last week I ran my last public Higher Performance Python training for the summer. We covered profiling and spoke a bunch about getting Pandas to go faster and stretching beyond plain Pandas with Dask and Vaex. I'm actually using a bunch of these ideas on a current Kaggle competition (Optiver Volatility) as we're limited to a 16GB dual core VM which is painfully slow. I'll report more on "what is working well" in the future.

One trick I've learned is that Pandas' read_parquet is substantially more memory efficient if you pass the path= argument (at least when using pyarrow, I haven't tested fastparquet) to read a set of partitions in 1 go vs reading them iteratively and glueing them together with a concat. A correctly-sized DataFrame is built ahead of time, rather than needing temporaries which increase RAM usage if you iterate. It also works out to be faster to load data this way too. Sometimes it is the little things that add up. Given that this dataset fitted into my laptop's RAM I was not surprised to see that the overhead of trying Dask negated any benefit of parallelism.