Transcript: Inverting the Testing Pyramid - Brian Okken

Whether it's scaling an application, deriving insights from data, or getting results from AI, Six Feet Up helps you move forward faster.

See what's possible at sixfeetup.com.

Hi, welcome to another episode of Django Chat.

I'm Will Vincent, joined by Carlton Gibson.

Hey Carlton.

And we're very pleased to have Brian Aachen back on the show.

Welcome, Brian.

I mean, welcome.

I'm glad to be back.

That's really well.

So for those who don't know, Brian hosts two different pod uh podcasts and books, courses, we're going to get into all of it.

But I'm really excited to lean on you, Brian, to find out about the broader Python ecosystem.

So maybe I'll lead with that.

An open-ended question.

What do you see in 2026 in Python?

Again, you sit much broader perch than we do.

So um I uh we in 2024 we got UV came out, and I can't believe it's only been that long.

It's it just seems like that's the world now.

Um, and uh probably you guys, but uh definitely Michael and I and a lot of early adopters grabbed a hold of it.

Like I probably I not right away.

I I played with it right away, but um using UV even professionally, like in a corporate setting, uh, probably the start of 2025, but by the end of 2025, I'm I'm seeing like my whole company is like interested in it.

So um I do see uh I think I see see more corporations just uh and and corporate settings using the UV tooling because it just makes sense.

But Brian, what you tell us about your day job.

Uh it's always been fun.

But I I work for good, good, good, good recovery there.

Um I work for a rodent schwartz.

It's a um uh test and measurement um well they do a lot of stuff, but I'm in the test and measurement side.

And uh so that's like electronic test equipment.

And uh most of my career has been in cellular and uh and wireless and Wi-Fi test equipment.

Um and uh these are these are fun boxes, but they're very complex instruments.

And the uh what am I going at?

Oh, um a lot of the tooling around it, we don't there's I don't think there's I think there might be some Python inside some of the boxes, but not the ones that I know of that I work with.

Uh but we do a lot, like we do the testing with Python, we do and a lot of the the utilities and all the day-to-day stuff.

A lot of there's a lot of stuff that needs done and we use Python.

And there's um there um and be because of that, the the we try to, you know, maintain best practices and uh try to not jump on too many bandwagons, like because it we could be changing directions too quickly.

Um but um UV just seems like a direction there.

And because of this, the time savings.

I mean, savings saving time is faster.

But I I also see that like actually the corporate pickup of new stuff is happening faster than like say five, ten years ago.

Um it used to take it took used to take like a year or two or more, you know, or even cut a few years for like best practices to really meet reach corporations because they kind of move slower.

Um, but um, but I'm seeing that rapid uh that uptick go quicker.

So anyhow.

Is there a what what's changed in the world such that corporations are able to move faster?

I like it.

I think it's um a common um credibility and information, but also I actually you know, it's as awful as it is, I think AI is bringing a lot uh because uh, you know, there's there's a lot of people that like to just do their job and don't they they use what they learned in college or something and and don't don't pick up a lot of new stuff.

Um, but um uh and there's you know I was always reading blog posts and listening to podcasts and stuff, but some people don't do that.

And so the people that just sort of now they're they're asking chat bots and the chat bots are paying attention to stuff.

So um, you know, the the the the fear I get though is there's less and less of us blogging about it because there's no money in it anymore.

So uh um um yeah, I I don't know where the new information is gonna come from.

Or are you familiar with that term and the academic papers around that?

Um I

Um so there was an Oxford paper uh 2024 showing both uh training it and the maths behind it that in the absence of other things, um the models get worse over time.

So then there's the argument of okay, so we know that's the case.

Maybe somehow the training gets better, maybe synthetic data is better than we think it is.

Maybe at scale those things don't apply.

I mean, nobody knows, but I I'm in exactly in the same boat, right?

Like I could plug in things and the LM will spit out something I wrote or you wrote if we prompt it correctly, but like the search is dead, the business model's dead.

I don't feel any imperative to create that content, nor do many other people.

So how can they how can they auto-complete the next word when they're training on worse and worse data?

Maybe we go back to books.

I don't know.

Um uh

Oh, you know, like Amazon, like some big percentage of it is just AI generated stuff there too.

So I mean I I half wonder if the snapshot of like mid-2022 internet is like this canonical uh corpus, and then we like fill in recent things, but we rely on that corpus of largely human content for the variants that we need.

Um Well,

But you you just said maybe something of the opposite.

Maybe it's actually easier for people who aren't quite as in love with it to AI.

I just money.

Well uh I don't buy it.

Like coding's the only game in town.

Maybe it's okay to get rid of those people.

Uh but uh but I think that there's I mean, there's just get you gotta find your passion somewhere.

I don't know.

I I I I couldn't work a job where I didn't enjoy it.

Um I really enjoy what I do.

So uh I don't know.

Uh but the and I think that like I'm writing a book, right?

We're gonna talk about that later, but I'm not doing it to get money because like that's that's like what I could if I wanted a few extra dollars, I could just go work at McDonald's up the street for a side job, I guess.

That'd be uh more cost effective than uh than writing a book right now.

Uh but but I'm gonna do it anyway.

But anyway, uh I don't need to transition into that anymore.

But the one thing I do while we're talking about AI, I want to say I think that um one of the neat things is that uh people code is using AI code assistance can go back to some of the old school good processes of come up with some ideas, generate a plan for how we're gonna do that, generate a few designs, um, look at alternative solutions.

Um, should we use that database or a different database?

Should we use that front end or a different front end?

Um, come up, bounce around those ideas and what are the trade-offs and why, and then uh iteratively build it up and put in some uh top level API level testing for the system and uh build it up.

That's what we thought good practices were, but it was too expensive.

So a lot of um, a lot of you know scrum houses and stuff and and and agile sl it was agile slop that was churning out garbage.

And I I think it's cool that we can maybe do the old school good practices, but we can do it faster.

It doesn't take months anymore, it takes days or weeks to go through that whole cycle.

Whereas now it's like no I will I will write it out not because I'm going to implement it because I'm going to feed it to a machine but at least it's like yes the explain what you're after before you touch to the keyboard.

That's progress.

And um and now we can go oh well we're like 10 days into it and it sort of seems like a dumb idea now.

Let's change direction.

And and we didn't do that before.

We would just be like well we spent like two months rewriting this thing we're gonna have to build it now.

And I think that I think that the the we're actually getting the agile part the agility part of agile I guess maybe okay interesting anyway optimistic good good good good good and I'm having fun still uh yeah I I you know what uh that's sounds smug though uh as one of the what uh people working software that still has a job um I I think that I think I'm still having fun that's good

I see lots I mean it's it's more interesting than even I thought it would be without AI.

More more tumultuous but more interesting.

I just saw it on social media somewhere who's talking about AI um and having it be an averager like it it turns out garbage but it's average garbage.

Average is fine for me because I'm yeah that's a classic level.

Yeah um so if I can have the AI bot generate average that's way better than I can do by myself.

So um for skills that I don't have I'm gonna lean on it uh and uh and maybe that's okay um yeah uh and it's especially for projects that don't have a budget to hire experts in every field why not um so I don't know where I was going with that but

because I'm still making money I'm doing it because it's keeping me from burning out so that's keeps

So anyway

development and um and that was that was great those two those two books are great um especially for lean uh the lean software development especially like the first actually the introduction the introduction is all you need to read it but like the rest of it no

be it works for some people maybe it doesn't and and to be fair they said that they said the principles are what you need to know the practices that we're gonna fill out the last two thirds of the book are our interpretation of how you could fulfill the principles.

So fair enough um and I like those but I wanted to take that so one of one of the things that Lean gives you is an idea of how to analyze processes for waste and waste being anything that doesn't directly add value to your customer not waste is in garbage but waste is in like like tests I think all tests are waste which is gonna tick a lot of people off but that's true.

Does a customer really care how how you're testing I don't know um but think of it as like what if you doubled it if I doubled the number of tests it's if the the product's already perfect and I spent ten thousand dollars more on testing do you want me to?

And they're like no it it's already good.

So uh incremental value sort of a thing um whereas like an extra feature value uh an extra test if it doesn't test anything I care about not value so anyway

onna be more reliable it's gonna be it's gonna last longer it's gonna be something I can put my trust in and rely on whereas the thing that was put together in a sloppy manner is probably gonna break within you know a small period of time and it'll better break at a crucial moment and software testing's that kind of quality control right it improves it gives you a better product in that sense so I as a customer might

ally gonna do those things that's the actual value how we get there is we put in place testing um um so I guess what I'm saying is like the uh uh we used to say it's cheaper just to have people manually test stuff than to automate it.

Now the cycles are so fast that we can't rely on that because I want to be able to change a feature and ship it today.

So we have to have automated tests.

But it isn't the automated test that's the value it's the the the value that those tests create I guess.

It let's like management activities like that's one of the things that Lean Software Development throws in there and I think it's hilarious um is all management activities waste.

It's like the benefits of the world unite.

It's talking about that there's there's essential work that you have to be done.

I guess think about it in a silly example I'm gonna cut circles out of a piece of paper.

If I slow down and and like do the circles really close together I can reduce the number of wasted paper so I can get the most circles out of a piece of paper.

But I have to slow down to do it.

So is it worth it?

the customer facing tests.

The does this feature actually do what I want it to do those are valuable tests.

So those are the ones we keep the unit test does it do the thing that the developer wanted it to do uh only if the customer cares about it.

If it's just that a a developer thinks that a unit should do this because you they're gonna reuse the the something on a new product less valuable um those sorts of things.

So but

Right.

And I'm sure you you talked about this right there's some like what but I we can't have infinite tests but like I want to have some tests even if the customer never sees them just so I can move faster without breaking stuff inadvertently all the time.

Um so I I do find the value in tests but it's the higher level ones.

I think um the API level tests and the component level tests are way more valuable than uh do my getters and setters work for instance.

Like that's what I'm picking up which is fair right like maybe wasted time.

Um and it and because the test pyramid said gave the unit developers the permission to focus on unit tests and the rest of the pyramid is somebody else's problem.

Right.

There's no dedicated QA team anymore.

Uh there's who's gonna write the component tests it's it's a bunch of people building trying to build houses with very thoroughly tested nails and pieces of wood and nobody to make sure that the house is to spec.

And that's just the truth of it.

And that's where we see like we still see a lot of people um focusing too much on does my little widget work?

And I'm not and I and and there's there's I write a lot of unit tests too but it's focused ones of like this algorithmic bit that's confusing let's throw tests around it to make sure that it it always works the way we think it does.

Or if we're gonna throw a corner case in because we need it or there's a bug fix put a test around it so that nobody looks at it and goes oh there's a weird algorithmic our algorithmic bit in the middle here that that it'd be a lot faster if we took that out but if there's a reason for it there throw a test around it so that it doesn't break later.

Maybe it's an application that won't scale or an AI initiative that just isn't delivering but that's where Six Feet Up comes in.

With deep expertise in Python Django and AI they solve hard software problems modernize platforms and get teams to market faster.

Their impact speaks for itself automating healthcare pipelines for hospitals helping NASA explore Pluto building severe weather prediction tools and applying AI to connect farmers with vital crop data.

When the stakes are high and the problems are hard six feet up is the partner that delivers see what's possible at six feet up is the partner that delivers

end.

They're not really they're not adding the value as you say.

Yeah pretty um because the scaffolding around your implementation doesn't help you if you try to take um so that like the a lot of the the the thing with test driven development was red green refactor but is the unit test again in the way of refactoring yeah um exactly so if a unit test if the behavior stays working and your unit test breaks then it's the pro the problem is you've got a unit test that shouldn't be there.

So that and there's I've seen too many projects not change things because they don't want to change a bunch of unit tests.

And that's

Yeah.

Because you need so the you need the agent that's doing its thing to have some way of knowing if it's if it's successful and without tests.

So what are you so you want tests that you're actually willing to review.

So just like you you'd have it like maybe you'd work with Claude to uh come up with your your uh your design plan you'd review it to make sure that you agree with it what what it's gonna do before you build it.

Same with your tests if you're not willing to review the tests then you shouldn't have the AI build it um whereas um and I think at the high level API level tests is is where we I'm I'm willing to review those tests to make sure that the product I'm building actually does the thing I want it to but if it's got some weird funky implementation down at the bottom I'm not gonna review the unit test for that um so uh I wouldn't even know what to review

and that's that's the that's the hard part as well even with people like not just not just AI stuff but people if you're if you're how do you review somebody else's unit tests um whereas you can review the API level test pretty easy you can go oh yeah that's testing the behavior of our system uh whereas lower level component lower level stuff is harder to review somebody else's project I

view.

That's hard work right that it's the first thing to acknowledge is this is not going to be easy.

You're not gonna get you're not gonna glance at the tests suddenly understand them and be able to say whether they work or not.

You've got to go through them

Yeah okay

So that's super

Do we get those in or did you have other things you wanted to mention there?

I don't I don't know how you have a company valued so largely slow so large that it isn't making enough money to cover its bills.

That's whatever.

Well I mean not my problem I guess

It's not that AI is like another chunk of revenue it's just a necessary piece of the puzzle of our offering.

I don't know.

Um ads that's that's the joke that was going around right like they finally got AGI ad generated income.

By the way check out Pepsi no thanks

Okay.

And um the and it's it's um it's like it's actually still doing okay.

It's um the it's down probably with the rest of the industry, um, as far as sales go.

But I I keep up on it and keep I I keep getting questions.

So one of the great things about um I guess I don't know, was it last year?

Maybe it was last year or a year before.

Uh anyway, um I think it was 2024 where I went through the whole book and did the course.

So there's a the complete pie test course, just I walk through the entire thing.

And it's there's a lot because it's uh it's it's a book.

Um it's not a big book, but to to go to walk through everything is takes a while.

Like, you know, the first couple of chapters fly by easy, but then it's like, oh, this goes on.

There's more, there's depth.

It's a substantial poem.

So uh I think the book is still valuable now.

The the the I I am I don't I don't know if I'll do a third edition.

The the only problem right now isn't the content of how to use PyTest.

The problem right now is that the project that I used there, it it was it uses dependencies.

So those dependencies are out of date.

And I have to go, I should go back and uh update the project to with the new um updates from the dependencies.

That that would probably be the if I if I were to do a third edition, um the main thing I would do is take out the dependencies in the project um and uh just just do it uh not have any dependencies so that I don't I have complete control over it and it'll always work.

It just does I see stuff like that.

I feel your pain.

Like I use what?

I use um a uh a a database layer.

Um the the that it's just a silly it's like mini db or I can't even remember what I use.

Tiny db, that's it.

Um and it's uh it's a it's like Mongo, but like small scale.

It's a single file uh document database.

And actually for it's kind of fun if if you're using it, but um but it's it's got an old version in there.

And if I would uh if I would redo it, I'd probably just go with like what's that the built-in SQL?

Uh SQLite.

SQLite.

Yeah.

I probably just have SQLite because that's that's always going to be supported.

Um and then uh and then I wasn't I used like I think I used click in the first edition and then typer for the CLI interface.

And um again, both of those are great projects, but they're dependencies.

So I probably just re-implemented in um in whatever the Arcbars.

Yeah, Arc Bars.

That's it.

Um and I'm also better at I mean I'm I'm better at testing with ArcPars applications right now, uh, than I used than I was when I did the 20 the second edition.

So um, but um I still I it's it was a good good idea to write that book.

Um and I I'm I'm glad that a lot of people have gotten value from it.

It's you know, it it we think through the you know the classic books, the fluent pythons and things, and you know, the PyTest book, the PyTest book is gonna be a good thing.

Like everyone has like that's always the thing.

You're like, is it is it a tiny pond or am I a big fish?

Right?

It's yeah, a bit of both, I'd say.

Um so

I want to mention projects and books, but I want to get your t hot take on what do you see as the Python web framework space?

Because you're not a biased insider, you know.

So there's Django, there's Fast, there's Flask.

I think I'm

too much of an email.

What what do you make of what's going on?

I I have a project that's that's that I'm a side project that's using Django.

Um and so I'm hope I'm glad that Django's staying is keeping up to date.

And I I actually think Django's a safe bet now.

Um it just looks like you you don't have it's kind of you don't have to defend yourself if you're gonna say I'm gonna use Django or Fast API.

Um everything else I think you'd have to defend yourself as to why you were using using that.

Um but that's just it.

Uh um I I do it's I think it's interesting that Michael's rewriting a bunch of his stuff in court instead of something else.

Uh so

So yeah.

PhD's gone wild.

Something like that.

So I

Because last time you came on the show, you just started and you were like you've still exploring your way in Django, and that was a little while ago.

So you found your found your feet and oh

no, no, no.

Um so uh it's a perpetual concept project.

Uh but uh like a painting.

That's the plan.

Um to to like Django's great

That that's the thing I always have to remember with my blinders of Python Django, like that's as good as AI gets because it's so well documented, both are so well documented, right?

Some other fringe language or framework, like you're not gonna get as good results.

So the Lean TDD book is a book I've been wanting to write for 20 years.

Um, and so I'm finally doing it so that it's out of my head and it and it can it's living rent-free and it needs to pay its own way.

Um, and this side project that I want to do is the same sort of thing.

It's a project I've wanted to do for probably 10 years, and I want to get it done so that it and it just it doesn't have to make me a ton of money, but it pro if I need to either see that it sinks or swims, and by swimming it just has to pay its own like hosting fees, yeah, exactly.

Um, and that's about it.

Um, but yeah, um yeah, I'm a newbie there.

Um yeah, we are running short on time.

What do we do next?

So Carlton and I are happy to go.

And if you feel inspired, um I'll I'll start.

So projects, I'm gonna shout out DjangoPackages.org, which is not a new project that Jeff Triplett has taken over, but he just r uh re-redesigned the whole thing.

So this is like the best place to go if you want to search through the thousands of Django uh third-party packages.

He used AI, he's talking, he's talked about it.

Um definitely check it out, give it a new look.

Carlton, you have one.

So it's uh it's a JSON field subclass that uses the message spec serialization um tool or library um to impose um schemas and validation on the data you store in a JSON field.

Um what I like about it, there's several of these um packages out there that are using different options.

What I like is it's another example of people in the Django community jumping onto these newer serialization options rather than you know just REST framework serializers that we've had forever and a day.

So there's a PyDantic version, now it's a message spec version.

And I think the more people in the Django community start using these things, the more they'll go, oh wow, this is actually a step up from what we're used to.

So um Django message spec field is my recommendation for the week.

Do you have something top

One of the things I've been using a lot lately was is kind of an old tool, is uh is from uh Rough.

Um so it's another astral tool.

So uh we like I mean we like linters in in software uh to to check stuff.

And linters really are just um static analysis tools.

Um so static analysis tools are very powerful as a starting point before you even run things to test them.

But um the rough checks are amazingly thorough.

It's not just it's not just uh simple things like, you know, did you you did, you know, if you use a variable or you have a you have an import that you're not using, it'll check for that.

But there's so many checks.

So you should go check out the rules at um at uh at Astral for the rough rules.

And and there's a lot of stuff you can turn on.

There's bug bear, there's uh there's even I've been using a a PyTest uh uh Flake 8 PyTest extension that you can turn you can just turn it on with Rough Check.

Um it's already built in.

It's it's amazing.

So uh if you if you even if you've played with Rough before, go check it out.

It's keeps keeps getting better.

So books.

Yeah, I'll go first.

So Polars is like a a data frame library, a bit like Pandas, right?

But it's folks.

It's so it's written in Rust and it's got Python bindings around it, and it's got an expressive, lazy API, and all sorts of things.

And so it's worth investigating.

Um Pandas has got a new version coming out, and Pandas is not by any means done and dusted, but Polars is, you know, another thing in that space.

And it's a good book.

It introduces everything very well and I it's going to live on my desk for a long time now because it's got a very good reference section and it kind of as you dig into it explains the kind of methods and then you're going to keep it on the desk to dig into the details as you need them as you're using them.

So um Python Polar's the definitive guide.

Cool.

I will mention not a programming book but relevant which is How to Hide an Empire, which I actually read several years ago and I'm rereading.

So it's a history of the greater United States.

So without getting too political uh the US has a long history of acting like an empire.

And this book dives into that and talks about disabusing common myths that Americans uh tell ourselves about what we're doing and it's timely unfortunately so really well really well written and I don't say that lightly I'm a super snob with writing and this is excellent.

So strongly recommend it.

Good one.

can do this but you can keep going

So I think just find out what you're still having fun with and double down on that.

I like Boston.

Carlton came to visit we walked a lot of it so

Bye bye see you next time bye bye thanks again to SixfeetUp the Python Django and AI experts you call for the hardest software problems.

From scaling applications to simplifying data complexity and unlocking AI outcomes they help you move forward faster.

See what's possible at sixfeetup.com