Transcript: Inverting the Testing Pyramid - Brian Okken

software problems. Whether it's scaling an application, deriving insights from data,

or getting results from AI, Six Feet Up helps you move forward faster.

See what's possible at sixfeetup.com.

Hi, welcome to another episode of Django Chat. I'm Will Vincent, joined by Carlton Gibson. Hey,

Carlton. Hello, Will. Hello.

And we're very pleased to have Brian Auchin back on the show. Welcome, Brian.

Hey, welcome back. I mean, I'm glad to be back.

I probably not right away. I played with it right away, but I'm using UV even professionally, like in a corporate setting, probably the start of 2025. But by the end of 2025, I'm seeing like my whole company is like interested in it. So I do see, I think I see more corporations just in corporate settings using the UV tooling because it just makes sense.

setting so it's important to prefix any discussion of my my choices with that but brian what you tell

it's fun now it's always been fun but i i work for good good recovery um i work for a road in

schwartz it's a um uh test and measurement um well they do a lot of stuff but i'm in the test

measurement side and uh so that's like electronic test equipment and most of my career has been in

cellular and uh and wireless and wi-fi test equipment um and uh these are these are fun

boxes but they're very complex instruments and the uh what am i going at oh um a lot of the tooling

around it we don't there's i don't think there's i think there might be some python inside some of

the boxes but not the ones that i know of that i work with uh but we do a lot like we do the

testing with python we do and a lot of the the utilities and all the day-to-day stuff a lot of

there's a lot of stuff that needs done and we use python and there's um there um and because of that

that we try to you know maintain best practices and uh try to not jump on too many bandwagons

It's like because we could be changing directions too quickly.

But UV just seems like a direction there.

And because of this, the time savings, I mean, saving time is faster.

But I also see that, like, actually the corporate pickup of new stuff is happening faster than, like, say, five, ten years ago.

It used to take like a year or two or more, you know, or even a few years for the best practices to really reach corporations because they kind of move slower.

But I'm seeing that rapid, that uptick go quicker.

Is there, what's changed in the world such that corporations are able to move faster?

I think Python Bytes brings a lot of...

Ooh, good, good.

A lot of credibility or information?

I think it's credibility and information, but also actually, you know, it's as awful as it is.

I think AI is bringing a lot because, you know, there's a lot of people that like to just do their job and they use what they learned in college or something and don't pick up a lot of new stuff.

But, and there's, you know, I was always reading blog posts and listen to podcasts and stuff, but some people don't do that. And so the people that just sort of now they're, they're asking chat bots and the chat bots are paying attention to stuff. So, you know, the, the, the, the fear I get though is there's less and less of us blogging about it because there's no money in it anymore.

So, yeah, I don't know where the new information is going to come from.

Are you familiar with that term in the academic papers around that?

I can guess.

I mean, yeah, basically you train on AI slop and it just degrades and becomes, you know, the variance shrinks so much.

So there was an Oxford paper, 2024, showing both training it and the maths behind it that in the absence of other things, the models get worse over time. So then there's the argument of, okay, so we know that's the case. Maybe somehow the training gets better. Maybe synthetic data is better than we think it is. Maybe at scale, those things don't apply.

I mean, nobody knows, but I'm in exactly the same boat, right?

Like I could plug in things and the LM will spit out something I wrote or you wrote if

we prompt it correctly.

But like the search is dead.

The business model is dead.

I don't feel any imperative to create that content, nor do many other people.

So how can they how can they autocomplete the next word when they're training on worse

and worse data?

I don't know.

Maybe we go back to books.

I don't know.

Well, like books, I mean, don't get me started on book. Oh, you know, like Amazon, like some big percentage of it is just AI generated stuff there too. So, I mean, I half wonder if the snapshot of like mid 2022 internet is like this canonical corpus. And then we like fill in recent things, but we rely on that corpus of largely human content for the variants that we need.

if you didn't love coding why would you get a cs degree i just money well i don't buy it

doctor like coding's the only game in town okay well maybe it's okay to get rid of those people

Because if I wanted a few extra dollars, I could just go work at McDonald's up the street for a side job, I guess.

It'd be more cost effective than writing a book right now.

But I'm going to do it anyway.

But anyway, I don't need to transition into that anymore.

But the one thing I do, while we're talking about AI, I want to say I think that one of the neat things is that people using AI code assistance can go back to some of the old school good processes of come up with some ideas, generate a plan for how we're going to do that, generate a few designs, look at alternative solutions.

Should we use that database or a different database?

Should we use that front end or a different front end?

Come up, bounce around those ideas and what are the tradeoffs and why?

And then iteratively build it up and put in some top level API level testing for the system and build it up.

That's what we thought good practices were, but it was too expensive.

So a lot of, um, a lot of, you know, scrum houses and stuff and, and, and agile, it was

agile slop that was churning out garbage.

And I, I think it's cool that we can maybe do the old school good practices, but we can

do it faster.

It doesn't take months anymore.

It takes days or weeks to go through that whole cycle.

and that is people were now actually keen to write the specs that you've been asking them to write

for you know so can we have feature x or what do you mean by feature so how dare you ask me let's

just implement it was the sort of attitude before whereas now it's like no i will i will write it

out not because i'm going to implement it because i'm going to feed it to a machine but at least

it's like yes explain what you're after before you talk to the keyboard that's progress exactly

the big mistakes was the spec was golden and we can't touch it and um and now we can go oh well

we're like 10 days into it and it sort of seems like a dumb idea now let's change direction um

and and we didn't do that before we would just be like well we spent like two months rewriting this

thing, we're gonna have to build it now. And I think that I think

that the we're actually getting the agile part, the agility part

of agile, I guess, maybe.

okay anyway optimistic good good good good good and i'm having fun still uh yeah i you know what

that's sounds smug though uh as one of the what uh people working in software that still has a job

um i i think that i think i'm still having fun that's good well we are i mean it's why we do

time right like with all the ai stuff and it's nice to kind of know a little bit what a little

bit about what you're doing and then see how ai fits in for work and i don't know for me like i

work at an ide company we're still sorting it out i see lots i mean it's more interesting than even

i thought it would be without ai more more tumultuous but more interesting now one of the

a link i just saw it on social media somewhere he's talking about ai um and having it be an

averager like it it turns out garbage but it's average garbage um and if you are if you're if

you're new at something like if i was going to write some css average is fine for me because i'm

yeah that's a classic level uh yeah um so if i can have the ai bot generate average that's way

better than i can do by myself so um for skills that i don't have i'm gonna lean on it uh and uh

and maybe that's okay um yeah and especially for projects that don't have a budget to hire

experts in every field why not um so i don't know where i was going with that but well it's just a

general ai rambling that we have each week now yeah yeah um but let's yeah okay one more thing

is we talked about the podcast a little bit. I started blogging about code and podcasting

and writing, not because I wanted to, I mean, the little extra money is nice, but that is what

I was burning out in CS and, and in coding. And that involvement in the community is what

turned it around. And I, and I still do it not because I'm still making money. I'm doing it

because it's keeping me from burning out so that keeps keeps the connection no it's gonna be quite

setting where you really think you really want to love what you're doing but you're not working

with anybody that has any sort of uh like heated feelings about it then you got to find the people

that too, so that you can talk with them. So anyway, definitely. Well, let's, um, we talked

Um, there's, I'm going to probably throw in some AI in there. There's no mention of AI in it right

now, but I've, I've got the first draft on. So lean test-driven development. I was way back in

the early 2000s, uh, a few books really helped me out. Um, uh, pragmatic programmer. Uh, and then

there was a book called, uh, lean software development. And, um, and that was, that was

great. Those two, those two books are great. Um, especially for lean, uh, the lean software

development, especially like the first, actually the introduction, the introduction is all you need

to read it, but like, I mean, it's a lean, it's a lean book, right? Yeah. No, it's pretty beefy,

but that's the sort of the thing in the book industry is like, oh, your idea is only like 40

pages. No, you're going to have to make it bigger. And they put, there's a bunch of stuff in the back

that I don't, maybe it works for some people, maybe it doesn't. And to be fair, they said that

they said the principles are what you need to know. The practices that we're going to fill out

the last two thirds of the book are our interpretation of how you could fulfill the principles. So fair

enough. And I like those, but I wanted to take that. So one of the things that lean gives you

is an idea of how to analyze processes for waste and waste being anything that doesn't directly

add value to your customer. Not waste is in garbage, but waste is in like, like tests. I

think all tests are waste, which is going to take a lot of people off, but that's true. Does a

customer really care how you're testing? I don't know. But think of it as like, if you doubled it,

if I doubled the number of tests, if the product's already perfect, and I spent $10,000 more on

testing, do you want me to? And they're like, no, it's already good. So incremental value sort of a

thing. Whereas like an extra feature, value. An extra test, if it doesn't test anything I care

the bit because like if i buy um you know i buy a product and two products they look identical on

the shelf but one has come from a a reputable fabric of reptile factory who's got quality

control and all those things it's going to be more reliable it's going to be it's going to last

longer it's going to be something i can put my trust in and rely on whereas the thing that was

put together in a sloppy manner is probably going to break within you know a small period of time

and it'll better break at a crucial moment and software testing's that kind of quality control

right it improves it gives you a better product in that sense so as a customer i'm like yes those

you advertise is doing it's actually going to do those things that's the actual value how we get

there is we put in place testing um um so i guess what i'm saying is like the uh we used to say it's

cheaper just to have people manually test stuff than to automate it now the cycles are so fast

that we can't rely on that because i want to be able to change a feature and ship it today

um um so we have to have automated tests but it isn't the automated test that's the value it's the

the the value that those tests create i guess um it looks like management activities like that's

one of the things that lean software development throws in there and i think it's hilarious

um is all management activities waste um it's like

the benefits of the world unite i was a manager for 10 years and um and i got i mean there's a

lot of activity that we do that is just dumb but that's not what it's talking about it's talking

about that. There's essential work that you have to be done. I guess think about it in a silly

example. I'm going to cut circles out of a piece of paper. If I slow down and do the circles really

close together, I can reduce the number of wasted paper so I can get the most circles out of a piece

of paper. But I have to slow down to do it. So is it worth it? You have to do trade-offs. Do I slow

down or do i speed up and it there's there's either wasted time or wasted paper and you have

to do those trade-offs and that's where testing and a lot of other stuff comes in almost everything

right yeah there's trade-offs everywhere so um really what i did is i'm using all those lean

principles and focusing on the test-driven development process and looking at where the

value comes from and i think the real value in a lot of automated tests are in the customer facing

tests. Does this feature actually do what I want it to do? Those are valuable tests. So those are

the ones we keep. The unit test, does it do the thing that the developer wanted it to do? Only

if the customer cares about it. If it's just that a developer thinks that a unit should do this

because they're going to reuse something on a new product, less valuable, those sorts of things.

And because the test pyramid said, gave the unit developers the permission to focus on unit tests and the rest of the pyramid is somebody else's problem.

And there is, in most companies, there's nobody else to take the rest of the problem.

There's no dedicated QA team anymore.

there's who's going to write the component tests it's it's a bunch of people building

trying to build houses with very thoroughly tested nails and pieces of wood and nobody to make sure

that the house is to spec and that's just the truth of it and that's where we see like we still

see a lot of people focusing too much on does my little widget work and I'm not and I and and

there's, there's, I write a lot of unit tests too, but it's focused ones of like this algorithmic

bit that's confusing. Let's throw tests around it to make sure that it, it always works the way

we think it does. Or if we're going to throw a corner case in because we need it, or there's

a bug fix, put a test around it so that nobody looks at it and goes, Oh, this is a weird

algorithmic, our algorithmic bit in the middle here that, that it'd be a lot faster if we took

that out. But if there's a reason for it there, throw a test around it so that it doesn't break

it's an application that won't scale or an AI initiative that just isn't delivering. That's

where Six Feet Up comes in. With deep expertise in Python, Django, and AI, they solve hard software

problems, modernize platforms, and get teams to market faster. Their impact speaks for itself.

Automating healthcare pipelines for hospitals, helping NASA explore

Pluto, building severe weather prediction tools and applying AI to connect farmers with vital

crop data. When the stakes are high and the problems are hard, Six Feet Up is the partner

that delivers. See what's possible at sixfeetup.com. I use unit tests as kind of scaffolding for

build and I'll have broken it down into logical steps. And I need a test for step one. And okay,

i've got step one's passing i've got step built from one built and that test remains useful while

i'm building step two and step three and step four all the way to the end but kind of once i've built

the feature and i've got a test around the whole feature like an integration test around the whole

feature it's questionable whether i need to maintain those unit tests or and sometimes they

can become a barrier because later on you want to come back and refactor the that that implementation

and you're kind of scared to because you've got all these tests that you're going to throw it

away it's hang on but they were just there for scaffolding while i was building it they're not

really they're not adding the value as you say well that's where the big the rule of thumb is

the scaffolding around your implementation doesn't help you if you try to dig um so that like a lot

of the the thing with test server development was red green refactor but is the unit test again in

the way of refactoring. Yeah, exactly. So if a unit test, if the behavior stays working and

your unit test breaks, then it's the, the problem is you've got a unit test that shouldn't be there.

Um, uh, so that, and there's, I've seen too many projects not change things because they don't

want to change a bunch of unit tests and that's exactly, or we can't change that. The tests will

back to the ai question because i think test-driven development so almost comes back into its own if

you've got a you know a claude code or a co-pilot cli or a gemini cli or whatever your favorite one

is yeah because you need some you need the agent that's doing its thing to have some way of knowing

if it's if it's successful and without tests exactly you're going to eyeball it and the last

review. So just like you'd have it, like maybe you'd work with Claude to come up with your design

plan. You'd review it to make sure that you agree with it, what it's going to do before you build

it. Same with your tests. If you're not willing to review the tests, then you shouldn't have the AI

build it. Whereas, and I think at the high level API level test is where I'm willing to review

those tests to make sure that the product i'm building actually does the thing i want it to

but if it's got some weird funky implementation down at the bottom i'm not going to review the

unit test for that um yeah so uh i wouldn't even know what to review and that's that's the that's

the hard part as well even with people like not just not just ai stuff but people if you're if

How do you review somebody else's unit tests?

Whereas you can review the API-level tests pretty easy.

You can go, oh, yeah, that's testing the behavior of our system.

Whereas lower-level component, lower-level stuff is harder to review somebody else's project, I guess.

That's hard work, right?

It's the first thing to acknowledge is this is not going to be easy.

you're not going to glance at the test,

suddenly understand and will be able to say

whether they work or not.

You've got to go through them.

Well, hopefully.

you should write readable tests.

And if you're going to have somebody else

or some other bot write it,

they need to be readable tests.

I think that's a nice thing to aim for, certainly.

of what I do for my job.

Right now, I'm in a role to try to make sure

all of the system level tests are at a highly readable level so that everybody in the project

ecosystem in 2024, you mentioned UV. Did we talk about 2025 and 2026? Did we get those in or did

you have other things you wanted to mention there? Just, I think, I think we're going to see the,

corner case even in corporation settings and the and then not and i think 2025 was the year of

people uh five coding and i think 2026 is people actually using it as an assistant to make

developers um uh do the right thing maybe yeah or or maybe we'll see how the business model holds up

too yeah well um i have no idea about that one that one just seems crazy i don't i don't know

I don't know how you have a company valued so largely, so large that isn't making enough money

on and think about what I can say publicly, I think there was a sense of a lot of companies

were like, oh, we're going to build tools on top of LLMs and we're going to make money in, you know,

extra money. Whereas now the sense is, no, just to like keep, to be competitive, we have to integrate

ai but we have our existing business model it's not that ai is like another chunk of revenue it's

just a necessary piece of the puzzle of our offering well if you're already charging every

how do you make more money i don't know um ads that's that's the joke that was going around

generate by the way check out pepsi uh no thanks i i'm i'm going back oh sorry carlton you look

your new book but we should talk about the old book i i was doing so thinking to myself oh we

should talk about the second edition but and then i thought that was just last year and then i looked

it was actually 2022 and i was like oh wow how has it been three years since the second edition of

um it's like it's actually still doing okay it's um the it's down probably with the rest of the

industry, um, as far as sales go, but I, I keep up on it and then keep, I keep getting questions.

So one of the great things about, um, I guess, I don't know, was it last year? Maybe it was last

year or a year before. Uh, anyway, um, I think it was 2024 where I went through the whole book and

did the course. So there's a, the complete pie test course, just, I walk through the entire thing

and it's, there's a lot because it's, it's, it's a book. Um, it's not a big book, but to, to go,

to walk through everything is, takes a while.

Like, you know, the first couple of chapters fly by easy,

but then it's like, oh, this goes on.

There's more, there's depth.

It's a substantial film.

especially with the second edition of keeping it

to things that are going to stay relevant

even in future editions.

So I think the book is still valuable now.

The, I am, I don't know if I'll do a third edition.

The only problem right now isn't the content of how to use PyTest.

The problem right now is that the project that I used there, it uses dependencies.

So those dependencies are out of date, and I have to go.

I should go back and update the project with the new updates from the dependencies.

That would probably be the—if I were to do a third edition,

the main thing I would do is take out the dependencies in the project

um and uh just just do it uh not have any dependencies so that i don't i have complete

control over it and it'll always work it just do i see stuff like that i see will

nodding in the background oh i'm just i'm yeah i'm not i i feel your pain i feel your pain

and the dependencies are like not big deals like i use what i use um a a database layer

um the the that it's just a silly it's like mini db or i can't even remember what i use tiny db

that's it um and it's it's a it's like mongo but like small scale it's a single file uh document

database and actually for it's kind of fun if if you're using it but um but it's it's got an old

version in there and if i would uh if i would redo it i'd probably just go with like what's that the

built-in sql uh sqlite yeah i probably just have sqlite because that's that's always going to be

supported um and then uh and then i wasn't i used like i think i used click in the first edition and

then typer for the cli interface and um again both of those are great projects but they're

dependencies so i'd probably just re-implement it in um in whatever the oh boss yeah arc bars that's

it um and i'm also better at i mean i'm i'm better at testing with arc parse applications right now

than i used than i was when i did the 20 the second edition so um but um i still i it's it

was a good good idea to write that book um and i i'm i'm glad that a lot of people have gotten

know the classic books the fluent pythons and things and you know the pi test but the pi test

it's a and we know it's not a small area right like everyone has that's always the thing you're

like is it is a tiny pond or am i a big fish right it's yeah a bit of both i'd say one of the core

You've seen the wolf.

projects and books, but I want to get your hot take on what do you see as the Python

web framework space? Because you're not a biased insider, you know? So there's Django,

there's Fast, there's Flask, there's newer things.

I think I'm too much of an outsider. I think...

Well, I'm still curious.

what you think what do you make of what's going on well i like that okay i have a project that's

is keeping up to date and i i actually think django is a safe bet now um it just looks like

you don't have it's kind of you don't have to defend yourself if you're going to say i'm going

to use django or fast api um everything else i think you'd have to defend yourself as to why

you were using using that um but that's just it um i i do it's i think it's interesting that

michael's rewriting a bunch of his stuff in court instead of yeah something else uh so yeah

take my advice from it because i'm not really a web web developer so i do want to ask how was

like you're still exploring your way in django that was a little while ago so you found your

project uh but uh i could paint you could spec it out with ai and you know that's the that's the

my blinders of python jango like that's as good as ai gets because it's so well documented both

are so well documented right some other fringe language or framework like you're not gonna get

write for 20 years um and so i'm finally doing it so that it's out of my head and it and it can

it's living rent free and it needs to pay its own way. Um, and this side project that I want to do

is the same sort of thing. It's a project I've wanted to do for probably 10 years and I want to

get it done so that it, and it just, it doesn't have to make me a ton of money, but if I need to

either see that it sinks or swims and by swimming, it just has to pay its own like hosting, hosting

costs yeah exactly uh and twenty dollars a year for like or whatever for its uh url renewal um

and that's about it um but yeah um yeah i'm a newbie there um yeah we are running short on time

we sprung on you last minute so carlton and i are happy to go and if you feel inspired um i'll i'll

start so projects i'm going to shout out jango packages.org which is not a new project that

jeff triplett has taken over but he just uh re redesigned the whole thing so this is like the

best place to go if you want to search through the thousands of jango uh third-party packages

nice so the new redesign is really nice he used ai he's talking he's talked about it um

definitely check it out give it a new look carlton you have one okay so the package i want to talk

message spec serialization um tool or library um to impose um schemas and validation on the data

you store in a json field and what i like about it there's several of these um packages out there

using different options what i like is it's another example of people in the django community

jumping onto these newer serialization options rather than you know just rest framework serializers

that we've had forever and a day.

So there's a Pydantic version.

Now it's a message spec version.

And I think the more people

in the Django community

start using these things,

the more they'll go,

oh, wow, this is actually a step up

from what we're used to.

So Django message spec field

is my recommendation for the week.

yeah, recommending things.

Do you have something top of mind?

I've been using a lot lately

is kind of an old tool

is from Ruff.

So it's another astral tool.

Yeah, I was going to ask you

about that, actually.

But the check stuff is amazing.

So we like, I mean, we like linters in software to check stuff.

And linters really are just static analysis tools.

So static analysis tools are very powerful as a starting point before you even run things to test them.

But the rough checks are amazingly thorough.

It's not just simple things like, you know, if you use a variable or you have an import that you're not using, it'll check for that.

But there's so many checks.

So you should go check out the rules at Astral for the rough rules.

And there's a lot of stuff you can turn on.

There's Bugbear.

There's even, I've been using a PyTest,

a Flake 8 PyTest extension that you can turn,

you can just turn it on with Rough Check.

It's already built in.

It's amazing.

So if you, even if you've played with Rough before,

go check it out.

It keeps getting better.

Yeah, I'll go first.

So I'm going to recommend the Python Polars,

the definitive guide.

So Polars is like a data frame library,

a bit like Pandas, right?

but it's Polar.

So it's written in Rust

and it's got Python bindings around it

and it's got an expressive lazy API

and all sorts of things.

And so it's worth investigating.

Pandas has got a new version coming out

and Pandas is not by any means done and dusted,

but Polar is, you know, another thing in that space.

And it's a good book.

It introduces everything very well

and it's going to live on my desk for a long time now

because it's got a very good reference section

And then kind of as you dig in, it explains the kind of methods and then you can keep it on the desk to dig into the details as you need them, as you're using them.

So Python Polar is the definitive guide.

So it's a history of the greater United States.

So without getting too political, the US has a long history of acting like an empire. And this book dives into that and talks about disabusing common myths that Americans tell ourselves about what we're doing. And it's timely, unfortunately. So really well, really well written. And I don't say that lightly. I'm a super snob with writing. And this is excellent. So strongly recommend it.

so little brother must be a play on big the idea of big brother yeah yeah okay um the other and

it's it's a targeted towards cs people so like or like there's a lot of programmer that the heroes

are all programmers which is cool as it should be right yeah um another one uh that i just sorry

we're only just supposed to do one but um no you can do as many you can keep going my daughter had

to how to read a book she's having to read a book for um uh for english called they're there by

tommy orange oh yeah yeah yeah and i i listened to the audiobook for that and it was phenomenally

um it was very hard there's a lot of it's not an easy thing to get through because it's talking

about um the way we've uh treated the native population in america um but from their perspective

kind of um yeah so it's but it's it's very very relevant as well and i i highly recommend it

those are my recommendations any as we go out or any things you wanted to mention we didn't ask

you about you want to shout out i mean of course people i think will have links to all your various

And I assume people got into software to have fun and not just to make money.

So I think just find out what you're still having fun with and double down on that.

Yeah, there's a lot in that.

That's great advice.

Thank you for waking up early to accommodate our schedule.

It's very early for you.

Yeah.

Now I get to go to work.

Yeah, yeah.

Well, same.

Carlton's done.

He's going to go make dinner.

Yeah.

getting dark now and well it's been dark all day because the weather's miserable but okay and i'm

have a car you gotta walk that's what i was told afterwards they're like why did you try to take a

car into boston that's yeah and boston drivers it's like the aggressiveness of new york with

a byzantine one-way streets that are not laid out in a grid so if you just see boston through

can't get anywhere without a car over here so yeah yeah try it again without a car i think

and it was it was i always miss it when it's on a hiatus so it's just over the summer you know we

have to keep our energy up. I appreciate it. And I understand that. Well, again, thank you for

Bye-bye. See you next time. Bye-bye. Thanks again to Six Feet Up, the Python, Django, and AI experts

you call for the hardest software problems. From scaling applications to simplifying data complexity

and unlocking AI outcomes, they help you move forward faster. See what's possible at SixFeetUp.com.