Transcript: Speed Up Your Django Tests - Adam Johnson

Hi, welcome to another episode of Django Chat, a weekly podcast on the Django web framework.

I'm Carlton Gibson, joined as ever by Will Vincent. Hello, Will, how are you?

I'm good. Hi, Carlton.

Hello, Will. And today we have Adam Johnson back with us, who's a long-time contributor,

MySQL expert, came on the show before, just a technical board member, and just released

a new book called Speeding Up Your Django Test, so we're going to talk about that.

Hello, Adam. How are you?

hello hello from sunny london sunny london is it sunny today no no well it's evening so what's the

difference right it's allowed to rain at night so you're you're on for a second time we'll try to

focus the discussion on your new book on testing since that's a topic that we can always talk more

about do you want to quickly mention the book and why you wrote it sure so the book is speed up your

Django test the title is like it sounds quite niche but I guess I've ended up writing quite a

lot of other stuff in there and just like through the lens of like you've got tests but you want

them to be fast and so it covers a load of like easy easy-ish techniques to like flip some

configuration or like run your tests in parallel that will make them faster but that also dives

into like what is a test how can you write them to be fast and um how can you write them to be

accurate and we should mention your quick bona fides i mean you've worked with django for what

over eight years you've worked on the django testing library as well as a number of pi test

plugins is that correct exactly um nearly eight years with django so you don't need to be unduly

modest when you talk about talking about testing and django just want to get that out there well

Well, maybe just a quick question. So why performance? At what point does performance

become an issue? Because for a lot of people, just having tests is an issue. So it seems a

little bit secondary to focus on performance. And I know that you talk about testing in general

through that lens. But when do people realize their tests are slow? At what point in the process

or the size of the code base does that happen for people?

Yeah, I think this is something I didn't touch upon much in the book. I'm going to add an update

about like how fast should tests be um i think like maybe one reason people don't write tests

is because they're perceived as slow and normally more like it's slower right like oh i don't know

what i'm doing it's a whole different paradigm of coding but also it can it can get quite tiresome

to run this thing if it takes one minute even each time and that can be quite a noticeable

difference to your flow but i know a lot of companies like they wait until their tests take

half an hour before they even consider something um this is what happened to me uh in my previous

position a few years ago why plan um tests were taking 20 minutes 25 minutes that's like a very

slow feedback cycle to make a typo or realize that the future doesn't work in this situation

and not get the feedback can i just ask that so okay the whole test suite takes 20 minutes okay

that's too long but you don't run the whole test suite when you're writing some new code right you

run one or two tests individually and they only take you know a couple of seconds and then that's

part of it carlton not everyone yeah i guess that's one thing but yeah and that's definitely

something you should do um but it's also like generally um pull requests are gated on the

whole test suite passing yes so if you have like a bug in production you don't want to be waiting

an extra 25 minutes before you get that fix out it can easily become the thing that everyone

skips even like oh the tests are being slow let's just merge the code and then you just

and it's friday and it's lunchtime ah yes uh so in terms of writing tests python has

the built-in unit test framework django has its own test um was it that's based off that test i

should know off the top of my head we call it django testing framework it's not really yeah

something creative but you know you and basically everyone i know who works with python or django

uses pi test could you just talk about why that is and i know you've noted that you know flask

also recommends pi test um how do you think of those stages of unit test to django test to

pi test well i guess unit test i see is like something slightly historical in the python

context. As Python was getting started, unit testing was also getting popularized through

JUnit, which was this Java framework, and then its clones. So unit test is effectively

a port of the Java testing framework. And that's why you might consider it quite un-Pythonic

in the way it names things. Like there's all these assert functions and they're all in

camel case. And you have to put everything in classes, which is something that's not

necessarily Pythonic. And so we've got unit tests, Django just built on top of what was there in

Python. And there've been like a bunch of attempts to make a more Pythonic testing framework. And I

think PyTest is the one that has stuck with the community, had a lot of effort put behind it.

So yeah, I think that's why we tend to stick with it. That said, like when it comes to Django,

like there's really a balance we've got all these testing features in the django testing framework

and like database control like extra assertion functions and things like the setup test data

which i have like a whole section on because that's a real speed boost for your tests that

you can't easily reproduce in pytest or at least haven't been and so i think you really have to

mix them at this point to get the best of both worlds so you use like the django test case classes

um with pi test i mean one of the thing i really liked about pi test when i first came across it

was the plain asserts right because even now i have every time i'm like self assert what if it's

not assert equal i'm like assert raises message reduce all the time i have to go and look it up

because it's just, I can't remember those methods.

But Assert, yeah, yeah, it's fine.

Is it AssertEqual or AssertEquals?

That was even something that I knew.

Oh, I know, yeah, I still have that, yeah.

I still have that issue.

Well, honestly, off the top of my head,

I can't recall which of the one

is the one you're supposed to use now,

except it's one of them, not the other.

Exactly, it's AssertEqual.

And you just, you get a warning if you use AssertEquals,

but it still works.

You just add extra warnings to your test suite.

Yeah, I think I've stripped,

i think i've updated all three of my books because they've been brought to my attention that

you know past ones wasn't always like that uh well so i did want to um so speaking of these

three different testing things one thing i've seen uh with total beginners is if they're new

to python they're new to django they're new to unit test and new to django tests a lot of times

you know everything is magic so where the fact that the django testing things mimics unit test

uh often they don't know what's python versus what's django let alone what's pytest so i really

i'm glad that you have your book to you know go further steps because i see people i mean probably

i myself too have confused you know what is a python thing versus what is a django specific

thing until i look it up um so that i find it interesting that there's it's convenient and

makes sense that it follows those maxims but also means if you just kind of drop into a project or

blindly follow along with some tutorials you really don't know what's going on with these

different layers of of testing things that are used in a modern django stack yeah it can be quite

hard to figure out which piece of documentation i'm going to go look at even well so so separate

from you know the official docs and and your book how how do you think about you know so if i came

to you with a django project that you know maybe didn't have tests how would you where would you

start how would you structure that and how would you educate a client who maybe you know has a code

base and their engineer quit or something and they hire you how do you do that um well i guess with

testing it always depends upon what the aim is um i think we know in general like you can test

exhaustively but it's not always worth it um like in the hardware world they draw the they have a

higher bar. They draw the line way further out. They will test things exhaustively.

Every CPU design will just be stress tested for days at a time. It's not something we tend to do

in the web community because it's quite easy to push a fix. But I like to try and keep a higher

bar, I guess, than most projects and aim for something like 100% coverage. But it really

really depends on what we're aiming to do because if you've got a website that doesn't support much

of importance like if it's a gif sharing site for you and a few friends and maybe you don't even

need tests you can just fix it when it has problems if it's like a voting website then

i hope it has like very good tests something i'd ask you about coverage because like there's one

way to get 100 coverage quite easily is if you test do right big broad um broad integration tests

that sort of i know they test the the total outside interface of the project and by doing

that they call through every layer and you can get quite close to 100 really quickly but they're not

good tests right no yeah that's another thing i think the branch option a coverage like restricts

you a bit more from doing that because that means that every if has to branch both ways so it can

be very hard from the outside of the project to run exhaustive tests before they become slow.

Yeah, in Coverage 5, there's the new format and ability to build plugins that would like assert

that this file is covered by that test file. So, hopefully, we'll see something for Django where,

you know, you've got your models file that must be covered exact 100% in a test models file. You've

got your views file must be covered exactly by the test views yeah that that's a real helper right

because well it's easy to cheat and and not always a bad thing right like if you just have that one

integration test that's way better than nothing hey and like those tests those tests work right

like so you need to check i don't know your login flow like that kind of high level integration

check yeah it does this does that does that that brilliant and it'll catch things but um for me

one of the advantages of one of the things i really why do i tell the reason i really love

testing is because it enables me to make changes i know i didn't break anything and if if i've got

small unit tests covering little bits of functionality i can make those changes knowing

that i didn't break didn't break anything in adjusting stuff and without that ah you can't do

it yeah if you have those things that just test your login flow you have no idea if this particular

thing in the login view that you're tweaking is actually working or not so the book is on

performance wins but in terms of talking about it there's a number of kind of easy ones or things

that you have whole chapters on what what comes to mind if someone says well adam what are some

performance things i can do to improve my tests um so like i have this easy wins chapter that is

basically a bunch of configurational things to install and some of them are quite context

dependent um but some of them apply to nearly every project so you've got like changing

authorization password hashes that's something that's in the django docs i just try and explain

it a little bit better basically they're like django's auth module uses a deliberately slow

password hasher so every time you check a password which in tests will be every time you fake a

log in through the test client, it

takes a few hundred milliseconds of processing power.

This adds up.

So in test, you can swap to one that basically does nothing

and is very, very insecure and should never

be deployed in production, but it makes it test fast.

MARK MANDELMANN- Well, sort of like the built-in server

Among other things.

Exactly, yeah.

And then there's like a bunch of packages you could install

or configuration to change because Django has some built-in

for in-memory usage.

So like disk is still like 40 times slower than memory.

Even with a fast SSD, you're not going to get a memory-level performance.

So just swapping your tests to use in-memory, that's normally a big win.

And I guess the final thing that would be quite an easy thing to switch over to,

especially when your product is young,

is parallelizing your tests.

So this is built into Django's test framework,

and there's a very good plugin for PyTest.

But, you know, your computer has many cores these days.

You should use all of them when running tests.

It's no good just burning through one of them

when you could be using 12.

I want to come back just a sec there,

because you called this, like, easy quick wins,

and I was just using memory,

But like that chapter is amazing, right?

Because you're like, oh, well, you know, you can use an in-memory, think of an obvious one, file storage or something.

I don't know.

Okay, that's, you know, that's not it.

But then you're like, oh, but, you know, you could configure MySQL and Postgres to use in-memory, you know, in-memory file system.

And then, you know, you've got the in-memory performance that you get out of SQLite.

But for, you know, Postgres or MySQL, you're like, oh, wow, now that, hang on, that's pretty cool down there.

Yeah, that was when I also was like, whoa.

i hadn't thought of that one yeah there's just like one line for docker if you're using docker

for your database you can tell it to mount the database on a temp fs and uh like i put it in

the django mysql test suite and it took 20 off the total runtime so and also for those playing

a django chat you know drinking game docker was mentioned so drink drink docker docker docker

yeah carlton please i have a whole bunch of questions but carlton i tend to okay so one

you know one thing that comes up with tests all the time is um and this is slightly orthogonal

to the topic of the book but is um database steep because that's you know database is going to be

one of the slowest aspects but no matter what you do and django you know very much tied to the

database through the orm so i mean what's the question now i guess it's really important to

optimized there for your Django project, because the nature of Django, the way you built, you

develop with Django, because we're model-based, because the ORM's there, optimizing that's

important, I guess. What would you say? I mean... Well, that also ties into migrations, too, which

I know you have a whole thing about in the book. Yeah, let's tackle just, like, using the ORM first.

So, the ORM is basically a translator for objects to SQL, and then calling the database with that

SQL. There have been a number of attempts to build like a mock ORM, which keeps the objects

in Python. I think like this, this is something that could work for like a very limited subset

of the API, but SQL has so many oddities to it, especially the variant of SQL you're using. Like

there's so many differences between Postgres and SQLite even that I wouldn't want to test against

like a fake version of my database and it's just too much of a mismatch and so I think it's really

important there to focus on like how can we optimize the actual database we're connecting to

like in memory is one way parallelization just uses multiple databases that can use

multiple cores on your computer that's really a good way of doing it and upgrading to the latest

version. Also generally you'll see a small or notable improvement in

performance. Migrations are a bit of a different kettle of fish because they're

like this whole bunch of SQL and Python operations that you have to run at the

start of tests to get a fresh database that your project expects. There's not

really like a way around that. You have to get that database into the state that

your program expects it.

As soon as you deploy your app to production,

that's the state the database is going to be in.

So there's the keep DB flag in Django's test runner

and an equivalent in PyTest Django.

That keeps the database around between test runs.

So if you're rerunning the same set of tests

against the same thing, you can avoid that.

Offer massive overhead.

And the other thing is squash your migrations regularly.

So combine, like, all these long sets of operations into one slimmer set.

Yeah, so in the book, you follow the party line.

You're like, you know, it's the squash thing.

I mean, what's your off-the-record view on, you know,

just sort of deleting all your migrations

and clearing out the migration history table and starting again with one initial?

I've never actually done that, I guess,

because the biggest Django project I worked on

was at YPlan, and

it didn't

have Django migrations.

Well, it did

for third-party apps, because you can't

get away from it for third-party apps,

even like Django ContribWorth.

But it

kind of predated even SouthBeanGood,

so there was a custom migration

runner that just ran a bunch of SQL

files in a directory.

It wasn't

the easiest to work with.

But I guess it was faster than Django migrations, especially at the time.

As to actually just deleting all your migrations and replacing them,

yeah, I think it works.

I think there could be even some documentation on how to do that

because there's so many caveats.

If you're deleting run SQL operations,

you're not going to have a database in the same state.

Run Python operations.

But the key point is that all your databases need to be migrated

to the exact same point, and, you know, then, okay, maybe, maybe.

Maybe, yeah.

I mean, maybe it's safer at that point to just take a schema dump of production

and load that and then write new migrations on top of that.

So the other chapter I really liked was about profiling

because I think profiling is kind of something that we all know

in theory we should do but that you very rarely see in a good guide to it and i think your chapter

on profiling is just amazing it's like yeah this is this is here it is here are the tools here how

they fit together here's the uis that you felt until i'm like yeah that's that's almost perfect

that could be a book in itself oh maybe i'll open it up yeah um yeah profiling is something that

has helped me spot a bunch of stuff in Django even. The two tools I really cover in the book

are Cprofile, which is built into Python, and PySpy, which is this Rust-based profiler

that basically knows how to read Python's memory layout. It's quite magical. It works with Python

2.7 up to 3.8, maybe 3.9 now. It can just read what is happening inside of Python, and it does

this like 100 times a second. It pauses your program, checks what functions are running,

records that and keeps going. And I've been able to spot like, oh, this Django function is in the

top 20 functions in my test suite. It could definitely be cached inside of Django and things

like that. I think we should be doing more profiling in general as a community. I've

definitely had some optimization pull requests made to my projects where I have a sense that

this is in no way a bottleneck if you read the code you might suspect it's a bottleneck but

if you do a profile you might not even be able to spot it so yeah i mean that's one of the sort of

golden rules right is don't don't guess what you're spending your time measuring yes i did

i originally start that chapter with the old donald newth quote premature optimization is the

root of all evil although i changed it when i was like rebalancing the genders of the quotes

fair enough so so actually so caching we just mentioned you talk about that um in your

continuous integration um chapter and i wanted to ask more broadly so there's a lot of options

for that and i hear anecdotally almost everyone kind of uses something different for that like

what what is your preferred choice and kind of what is the landscape of good and bad options

you've seen for CI and Django projects.

Right, so what CI should you be using?

Yeah.

I don't know.

That's something of a moral question

alongside a technical question.

Well, you have a Greenfield project.

It's all Atom.

What are you going to use?

Well, if there are repos on GitHub these days,

I would use GitHub Actions.

Okay.

I heard a number of people very much in favor of it.

Yeah, it works great.

It's fast.

I mean, Microsoft have thrown a lot of money

at getting a lot of runners there

ready to run your open source projects

or your closed source projects,

a lot of community work.

But I moved to it because Travis CI was being slow.

And then I discovered that

it's because GitHub introduced a new rate limit.

This was after I'd moved to GitHub Actions,

but it's because GitHub introduced a rate limit

on the way Travis was integrated legacy-wise.

So Travis just couldn't push the results back

and you needed to go and reintegrate your app

with the new like GitHub apps

rather than GitHub legacy integrations.

So maybe I wish I was still on Travis.

It had some benefits.

Yeah, because that was, I mean, Travis and Circle

and Travis in particular was kind of the,

had like a good default for open source packages.

So are you saying, so does that,

Is that something malicious on Microsoft's part

or more just a change and they didn't decide to update it?

I mean, I guess I don't really have a read there.

I never received a notification as a Travis user,

and probably Travis could have detected that,

but then I never received a notification from GitHub either,

so GitHub could have detected that.

I mean, it could be all of the above too.

Hard to know.

My little two cents on the same question is that GitHub Actions is nice

because it's right there, and it's awesome.

you know whether you recirculate ci that's great yeah that's fine travis got good as well you know

i've been using that for a long time with the open source stuff use a hosted one right don't don't

unless you've got a real reason don't spin up a jenkins and try and manage that because that's

hard work okay yeah that would be my one rule i like the last time i touched jenkins i was like

i'm never ever going to use jenkins again it seems like it's the most common legacy thing right

because it's free as in not developer time free but free but i i see that the most i mean i've

used that and when i i ask people about this i often if they're not using a hosted one they're

you know they're some version of jenkins it's not that it's not good as content it's just it's just

so much time keeping it up to date and running and in sync with what stuff you need and yeah

i remember the first time i came to a hosted ci and you know it was just like the scales falling

for my eyes it was like this is the future i've been a you know i was like we've got to do this

i say at the time i'd go around the company companies go around clients and be like yeah

we're going to use hosted ci that was like you know my hammer of the year yeah like any time you

save managing your ci environment it's time you get for testing or feature development i have

another question but carlton get yours in okay well the other thing that um like so what i really

like about the book is there's so many things that aren't testing related at all so we've talked

about profiling, we're talking about CI now.

I mean, CI is testing, but you know, kind of also isn't.

It's not speeding up your unit tests,

it's the whole development process.

But then there's a whole little section on queues.

Which, like, you know, configuring the queues,

and it's like, you might be using this queue,

and here's how you configure that.

And it's like a little.

configuring for tests but yes yeah i know but it's like you know it's like a sort of mini essay on

you know the different q options that are available and i'm like yes and all my favorites

are all in there and it's like yeah so that's what it's kind of what i like i like like how

you've gone over the entire spectrum of stuff and it's not just you know it's not just tests in the

narrow sense like i've read a lot of books on unit testing it's like this the whole process

so you know that's not really a question it's just i guess it's a bit discursive in that way

but i hope the titles let you like skim the bits that you don't need to know about right now and

come back to them when you think you do yeah but it's not i mean it's anything but it's like a

massively awesome resource it's only 160 pages right so it's not like it's like this um distracting

but it's just massively dense and what else is in there that i really liked um like advanced trip

um tips on configuring the test case subclasses and you know using particular test case subclasses

and i'm trying to think particular one um but you know that's that's hard knowledge to gain

um so you know thank you for putting it all down thank you well maybe this is the the segue i did

want to ask at one point about the writing process for you so i know that you've you've been writing

almost weekly what for months now and so the book kind of kind of came out of that um could you just

talk about upping your you know output as a blogger and then what the book process was like

because i think a lot of people think about potentially doing a book but most people don't

actually do it right and well i guess like it looks like i write every week and but i don't

like I queue up unpublished posts and then when it comes time to publish them I look at the ones

that like I'm like oh this is this is nearly ready and then I push it and I guess that is one thing

that I've learned is that like you don't need to be like perfectly ready to publish and like take

that MVP approach that we put in startups and put in writing because normally it's better that

something is out there then it never gets out and you learn more by doing the whole cycle not just

by like writing one complete perfect post a month but by writing like four different tutorials of

different kinds and things like that and so the book came out of me like sitting down to write a

blog post which was like gonna be like top 10 tips to speed up your Django test suite and like I

write a bit of this a bit of that and then i had like a 4000 word draft or something and that wasn't

even like in prose that was just like bullet points and like do this thing and like copied

some text from iosware um yeah and i guess all that stuff was like already in my head and my

notes from like years of working with django tests and i'd you know made drafts like this could be a

blog post on this one little thing and i realized like just bringing them all together would actually

be more powerful um yeah i guess lockdown was really the biggest motivating factor for getting

this done because i'm just sitting at home all the time only so much netflix i can watch i'm

gonna have to write my book now yes yeah well and kudos for writing it and publishing it it is

you know it's kind of like until you're on the other side it's easy to hear people say that but

then you realize it though i will say if you i don't know if you're planning to put it into

paperback but that's a whole another fun thing to deal with doing and updating that's maybe another

future project yeah um i've got a bunch of updates to put in the book because like i published a

library this week for mocking time because it was a bit of a yak shave i wanted a section on how to

mock the current time but i can't find a library that would be like perfect to recommend i knew

of the trade-offs between two of them so i wrote the third one um so yeah i i think the ebook form

will be me for a little while whilst i like iterate and add a few more sections and maybe

rearrange stuff um yeah i think because it's technical and you know i mean like

to take examples of github actions or whatever that you know that that's liable to change and

so to have it in the ebook format where you can update it and the good thing is like i have all

these django projects for the code they get included so i can like upgrade my requirements

to django 3.1 in october rerun all of those tests although some of them are meant to fail

for examples and then yeah it will definitely be an update process i don't know how often you

update will um too often um it's it can be very burnout inducing for me uh at least once a month

um you know some version of something i did wrong or django changed or docker changed or python

changed or heroku changed or stripe changed uh it is important to do it is also important not to

jump on everything that comes in because i have it's an endless supply of stuff um i mean i'm

currently updating my django for professionals book because there's stuff around stripe and then

actually well we could talk about this um i asked on the jingle forum there's a thing around

what is the thing i asked the question of you you helped you helped answer this both of you um

give us a clue the thing that um what what oh one of the in the deployment checklist something

was added for 3.0 that was flipped and then in 3.1 i think you did it adam is going to be switched

again i forget which secure referrer so people noted for me that that had changed because in

uh 2.2 the 2.2 version there were nine things that if you just ran the deployment checklist

failed and then it's now seven and then yes secure refer is one that uh changed and then also the

default will change so they're currently what isn't a default but then for 3.1 it's going to

be set to something else anyways it's stuff like that times a thousand where it's educational and

i'm really glad people do it but it's uh yeah so i i i try to get better about devoting specific

time to updates rather than just having it constantly weigh on me yeah i was i'm not great

i was reading um harry percival's book obey the testing goat and yeah during this and he actually

like deliberately pins at janga 111 and a specific python version and says use these throughout the

book i'm not going to support the others if you see random errors it's probably because of that

come back here and and install the versions i'm working with um yeah and and he'll update it at

some point but he's like very tightly integrated to specific versions i think that could work

we have to pin it i guess i would the two things i would say is one is beginners need it to be up

to date or need and want it to be up to date and and they will still install the latest version of

Django, even though you tell them not to, and they will get tripped up. So maybe Harry's book

is a little more advanced. You can get away with that for a beginner book. I think you have to.

The other thing is anything that's non-Django can and will change a million more times. I mentioned

the Stripe API. I would love to have nothing but Django and Python in my books. I'm kind of getting

there instead of waving my hands and saying, yeah, Heroku, yeah, Stripe. I think I'm probably

actually going to remove the Stripe chapter, for example, because it just changed. It's going to

change again. It's more than I can handle. I probably will keep up Heroku because I do want

to show easy, in quotes, deployment. But anytime you step outside of Python Django, you're just

opening yourself up. SendGrid changed in April, all of a sudden changed the config. So chapter 12

and my beginner's book was broken.

So that was fun.

You know, so, you know, I'm hoping to,

I think either you have to charge a lot

or just not do it to show integrations

with non-core Django things,

even if it's things that a modern Django stack

probably uses.

As a content creator and updater,

it's really, really hard to do.

Well, one of the things I deliberately dropped

from my book was, well, didn't put in,

was Docker.

There it is again.

like jeff triplett fan posted about it there's like a new build kit integration in docker like

this new way of building images and it's like i don't know a bazillion times faster i was thinking

okay i'll put this and then i was like no this is going to change in like one month's time because

they'll make it the default or something yeah yeah well i didn't even mention docker in that list

they yeah there was some there was another bug like a month ago where people you know a dozen

people are writing me and it's like oh i don't know i gotta put on my thinking hat and there

was something specific to windows docker not mac docker not linux that was breaking um which you

know the whole point of docker is to not have to worry about systems and honestly in my professionals

book which i know you you've gone through with with someone you were teaching yep i've and i may

well remove docker from it in future editions in part because it's just another way of complexity

even though it is quite common but i don't know it's a trade-off because it you know because these

are modern tools right like you know pi test right pi test is not django i mean it kind of is but it

really isn't um your ci uh it's it's hard to strike that balance i think it's a little bit easier

maybe in more advanced things or it's maybe more easier for things that are you charge a lot for

but you to update stuff for you know docker and whomever it's just makes people not update their

stuff yeah with darker i've i've tended now towards just using it for running the database

and like redis or memcache and um don't use an image for the python stuff i feel like it's

it's quite a bit easier just to have python on your system with pyenv or something and

you've got it there you know python is working on your terminal and you don't need to build

images again and again update them in a team setting as well that works out okay um yeah it

has for like one client where i've put them in what were you gonna say carlton well i i'm kind

of in the same sort of boat like you know for running i don't know some postgis enabled you

know database so i'm not installing that locally because it will destroy my entire system when i

get it wrong um yes to have to have you know various versions of python set up locally and

you know the the core set of dynamic libraries i need to compile the the core dependencies

that's not difficult to do and this idea that you you don't need to have a known environment you

know a reproducible containerized environment for that kind of local development yeah brilliant when

other cases yes totally but yeah i think when you hit a certain size there's definitely savings like

if you're using five different external libraries and you've got 20 team members

you're gonna have a real pain of a time getting 100 of those installs in sync over time so you

one of the things you have that i want to ask about is this triple a test structure

could you explain what what that is sure like i know i invented this um

uh it came to me to a friend james cook at his pycon presentation one year pycon uk that is

and and he's written a blog post on it which i find myself pointing people to quite a lot

um i guess it's something that's implicit in testing is that this is the general structure

of how you write a test and but often tests like when they tend away from it they become more

difficult to use it or run or less performant and so the triple a um stands for arrange act

assert. So you always think of your test as like arranging the environment. So putting that

pre-existing user in the database or calculating the URL to go and fetch or something like that.

The act is like actually using the system under test, the component that you're trying to test.

and storing its result or results.

And then the assert is when you make some assertions

on what happened, like did it actually update that user

or did it return the correct response?

And sometimes you have multiple assertions,

like did it both send an email

and return the correct response from the view?

But yeah, I tried to cover that in the book

as a way of like, here's how you should be thinking

tests and structuring them better what I liked about your discussion there was

the way you but this bought into though well you know can I group asserts

together well you know think about it if you're arranging the same stuff and then

you're doing the same actions then you can make the set of related asserts

there's no you don't have to duplicate that because that's obviously gonna run

storm into time but you know this is why I think there's so much more than just

speeding up your tests in this book it's it's like yeah okay you've you've sort

of tied it back to speeding up your test there but it's general principles for

how you arrange your unit tests or how you might think about the structure of your code

as well i think yeah there's that perhaps dogmatically applied idea of like a single

assertion per test and then you can end up with tests that like one fetches a view and checks

it returns a 200 status code another fetches a view and checks that it has a certain header

another fetches a view again and you've just trebled the work for like fetching the view

whereas you could really think of all those asserts as like they're just asserting different

parts of the http response which is ultimately like a text file one one thing i'd like to do in

that um circumstance or a combined assertions is i'd like to be able to sort of conditionally

assert you know even if that one fails run the other assertions tell me what else went wrong but

the tests stop at the first one that fails and it's a bit like ah but you know the actual

informative one is four down sometimes yeah what do you think about that kind of scenario is there

a is there a magic key there to there are there are different attempts to solve this um yeah like

maybe if it's something like an http response you could imagine some kind of magic object that you

say you know say so code should be this header this should be there this should be in the body

you compare that object with the actual response and it figures out all the differences then it

displays an error, if there are any.

But more generally, there's a tool in unit test,

and therefore Django's test framework, called subtest.

And this is for a context manager

where you can mark a block of code as like,

this is part of this test.

If it fails, still continue.

And then capture and show all of the errors at the end.

And then in pytest, there's pytest support for subtest.

Or there's a plugin called, I think, pytest check,

have all these check functions that do the same. I don't think there's a perfect solution

out of all of these options right now. Subtest, if you want to use it for assert, that's like

doubling up your typing and making your test much less verbose. I guess it would be really nice if

you could have pytest automatically keep running through all the assert statements.

It's not always possible though. Sometimes you have an assert statement that

does another action implicitly and that's part of using the AAA pattern. If you have your act

like right there in the middle before you do any asserts then it's going to be easier to

make all those asserts. Yeah it's in Django's test where you subtest a lot for

parameterization you know you've got four or five different versions of the same test you know

pytest does it differently with parameterized fixtures but you know we use it a lot there for

for that but i like the idea of using it for keeping your your tests running i hadn't really

thought well as ever i mean so we're around 40 minute mark but there's two questions i definitely

want to get in um and i don't know you have carlton i want to ask you about mocking and then

i'd like to ask you about the django technical board and who is on that and what that entails

because it's important but i'm not sure how aware people are so um either one of those two i mean in

particular you have a thing called the danger with mocking which sounds sort of exciting

okay um was that the question this isn't like some part that will be cut out of the podcast

speaking out loud so i wanted to ask about the danger of mocking and then also um both of you

are on the technical board or the technical well let's talk about that okay yeah um so the technical

board is five of us who have been elected to manage Django's technical direction. This

doesn't mean we're involved in every single decision, but the idea is that for bigger

decisions, we are the voting body that can say yay or nay to any particular suggestion.

With the recent implementation of DEP10, we're meant to be taking a more active role as well.

So we'll start seeking projects and evaluating them together as a board.

And I know Carlton even made a GitHub team for us today

so that we can help review many of PRs whilst Carlton is on holiday.

So the five of us are myself, Andrew Godwin,

Aymeric Augustin, Marcus Holteman, and Simon Charette.

Yeah. Does that answer the question?

Yeah, I think so. Just, I, I think it's, you know, it's a very important board team that I know that not many people, uh, are necessarily aware of. Um, so we've had Andrew on the show. We're actually interviewing Amrick tomorrow. Um, we've had Marcus on and we hope to have Simon on. Um, but you know, when I talked to Carlton about who's really with their sleeves up doing things in Django, uh, all five of you are, I mean,

the mount you know not what is it mount not zeus anyways mount olympus mount olympus thank you

on the mount on the mount olympus of of active django things i have no idea which uh god or

demigod i'd be then probably not a very important one yeah um yeah like everyone on the technical

board yeah i'm impressed with all my fellow technical board members like contributions

I think there's two things that I'd say.

One, to be on the technical board, you have to be active.

You have to have a shown history.

To even be a candidate to be on the technical board,

you have to have a shown recent history of active contributions.

So that's one thing.

The second thing is that the technical board,

whilst they're there as this decision-making process,

normally most decisions are reached by a sort of consensus

in the discussion on django developers you know not often i mean recently there's a um a discussion

about um adding type hints to django and there was there's been lots of discussion over several years

and you know lots of proposals and lots of pros and lots of cons and at that point there wasn't

a consensus that came out and so you know we asked the technical board to you know make that decision

but that's quite rare normally you know there's a bit of debate a bit of conversation but

people are like yeah no actually that's the way forward um which is kind of nice i think yes i

think we work pretty well as a community and it's good that we aren't needed to make many big

decisions good thank you yeah i mean we'll link to there there's a team section on the docs um and

it is with depth 10 there'll be a little bit of changes but um so mocking what is mocking and what

makes it dangerous the danger with mocking yes um yeah i i guess there's just generally a trend

to overuse mocking in the python world because it's so easy python as a language lets you do it

well and then unit test dot mock is like um ready and there to go and it can just mock pretty much

anything and so i called my mocking chapter targeted mocking because i think this is really

the key point with mocking is that you should make them as targeted as possible and know what

role they're supposed to be fulfilling when you make them. Whether it's there just to check that

your system calls another system or it's there to stop you from calling the other system.

And then like making them really targeted is the key. If you use a unit test dot mock object,

It's like very untargeted by default. It's just this weird magical Python object that

you can call any function on it and it's there and it returns another mock and then that you can compare with anything and

Like I think from beginners, it's very hard to get your head around Python even allowing you to do this

Like there's an object that compares equal to every single number every single string every single dictionary

no type errors ever it can really like break your program but in tests it tends to just like

pass every test because it just kind of allows everything to happen

without actually testing anything at the end of the day also so i see people or projects where

there's been mocking and it's mocked out something but what you end up the unit tests are testing the

mocking not the not the system and it's like but this why did it break in production and he's got

all these unit tests it's like uh yeah but they in the end they're not asserting anything yeah

right you draw a boundary you put mocks at that boundary for the other system and then actually

what you tested against this mock is completely different so again it's like the mock should be

targeted it should be known to be like a good replacement for the other system and so like one

of the libraries I mentioned is RequestsMock. This is a good mock for the Python requests library.

It actually injects itself inside part of requests with a known public API. So,

quite a lot of the actual requests code gets tested when you run this. It's just

not the third-party HTTP requests. Yeah. Rather than mocking out with unit test.mock the entire

interface and you didn't even check for typos with like request.get yeah did you pass the right

did you call the right method name yeah thank you carlton i don't know if you had any other

things you wanted to ask adam no i mean no i mean i was i was reading the book last night

i was just you know blown away i was just like what really good you know really good thing it's

not for novices but it's you know yeah it's a must have if you're a django developer i think

you you you use django you write unit tests this like just get it even if you don't write

unit tests there's loads of other stuff in it it's really good but like well i hope it convinces you

to write unit tests yeah do you personally adam i mean you uh have any projects or um things you

want to call out that you've been working on well there's the book there's and there's time machine

library i mentioned earlier that i released this week that's that's actually my first python library

using c under the hood um and i'd like to see it tested a bit more it's like somewhat of a drop-in

replacement for popular freeze gun so if you're using that and you're listening then maybe try it

out um and other than that just like follow my blog but i guess it's not hard to come across it

if you're reading things like Django News.

Yeah, I mean, pretty much every week,

it's just like, well, there's a couple from Adam.

And, you know, it's, yeah, along with this podcast,

it's one of the weekly things.

So good.

Well, thank you so much for taking the time.

Thank you for writing the book.

I hope more people will read it.

I know many people have bought it,

but again, I think it's a core part

of the educational canon of Django stuff.

So, well, thank you.

Thanks for coming on.

Thank you very much for having me again.

bring it to chat okay everyone we are at chat django on twitter

and we'll see you next time bye-bye joyous next time bye-bye