Transcript: Optimizing Django Queries - Jamie Matthews

Hi, this is Will. A quick note before today's episode. I'm offering a Black Friday sale,

50% off my three books, Django for Beginners, APIs, and Professionals. Good until end of day

Friday, November 26th. Link in the notes. So if you're thinking about learning Django on your

own or for someone else, now's a good time. On to the show. Hi, welcome to another episode of

Django Chat, a fortnightly podcast on the Django web framework. I'm Carlton Gibson,

joins us by will vincent hello will hi carlton hello will and uh this week we've got jamie

matthews with us who's um dab apps creator of various uh packages in the django um ecosystem

long time django user and whatnot so we're going to chat about all those things hey jamie thanks

for coming on hello folks thanks for having me on oh thank you for joining us quite excited you've

been um producing some really interesting things that i'm keen to dig into but before we do thank

you before we do all of that um let's do our traditional um what's your backstory and how

do you get in and tell us about your your how do you find django okay so i i'll introduce myself

first so i'm a technical director of dab apps i suppose in america that might be cto um we're a

kind of web and mobile agency so we build bespoke web apps mobile apps uh usually with api backends

of some sort um we use django for all of our server-side code we're based in brighton which

is on the south coast of england then we've got an office in wales as well about 30 people

17 18 devs a mixture of back end and front end full stack um and we've probably built over the

years i was trying to add it up it's definitely over 50 and it might be kind of closer to 100

django projects um anything from like small code bases you know a few thousand lines of code up to

our biggest projects are sort of 70 80 100 000 lines of python um so we have some some really

big projects or well relatively speaking i think yeah i mean um you said 30 people right did you

say yeah so yeah 30 people total yeah so that's quite a big agency i mean you've got you know

there's a lot of turnover of work there yeah indeed yeah yeah i suppose we're we'd be considered

fairly big do clients come to you asking for django or because of django or do they come to

you and then you say, this is our toolkit we like to use? It's a mixture. Most of our clients

don't care or don't know or care about the stack other than the fact that they're happy that we're

good at making decisions about what technology to use. We do sometimes have clients who come

to us specifically for Django. That tends to be clients who have existing code bases that maybe

they've had someone else build and and they need that sort of taking over from people who who know

how it works okay so before we dig in further i was gonna ask you say you did mobile and native

apps and um these kind of things as well as web apps so can i just ask about the front end

integration like how do how do you um map between django on the back end and what's going on on the

front and next that's quite interesting so most most of our well all of our front ends are react

mostly written in typescript and for mobile apps we use react native um so yeah essentially we

build um apis with django rest framework um and we have uh you know yeah react and react native

apps that talk to them so yeah so django rest framework we have to we have to mention it here

because um the thing you didn't mention about dabs is that's where tom came from was tom christie c

yeah so i i realized i didn't i didn't really answer your question which was how i how i came

to Django. So I'll loop back around to that one. So I think I found, I was trying to remember where

I first used Python. I definitely came out, I did a computer science degree in Manchester and I

definitely came out of that with an awareness of Python. I can't remember whether I used it as part

of the course or whether I just sort of found it in my, you know, in my free time. I then moved

down to brighton to do a master's and i use python quite a lot there doing kind of scientific

computing type of stuff when i finished my master's i went and got a job as a web developer

but doing php um but i used to get so brighton has this really amazing sort of tech scene with

lots of meetups and there was one which i used to go to a lot which believe it or not was called

flash brighton right do you remember flash well yeah yeah a few years ago wasted half my teenage

reduce with that well indeed so i didn't really do very much flash but i kind of went there because

when they didn't have flash topics to talk about they got all sorts of other people and all sorts

of other technologies and one of the speakers there was simon willison who of course was one

of the original co-creators of django who at the time lived in brighton um so he was giving a talk

about django this would have been i think i think it was early 2009 so it would have been just after

1.0 came out and kind of just before 1.1 came out um so i went and saw this talk obviously i was

doing a lot of web dev at the time i really like python and this was like wow now i can build

websites in python this is the best thing i've ever seen so i went away and kind of dabbled with

it funnily enough the very first bit of django that i used was inspect db which i imagine basically

nobody uses um because what i did was we had this kind of internal tool it was some sort of

inventory management system which was like a php app that talked to a mysql database

and i kind of went oh watch this you know i went and pointed inspect db at the database

generated all the models wired them up to the admin and within about five minutes i just had

this like working admin system for this for this app and of course everyone was very impressed and

they all went wow well done let's go back to writing php um so um so yeah i did that for a

while played with django in my spare time you know got a book about it learned as much as i could

and then eventually myself and a couple of colleagues from from my first job decided that

we were kind of going to go off and do our own thing set up dab apps from day one it was it was

all about django um our first project was uh it was for a company who were a supplier of tesco

which i'm sure everyone's heard of um and so our very first project was kind of quite high profile

very briefly featured on like the front page of the bbc news and it didn't fall over let's just

have a footnote for american listeners tesco's is a large supermarket chain yeah that's it walmart

i've i've been to your side of the pond so i know i know it is but if i hadn't i wouldn't yeah so

other supermarkets are available of course um yeah so it was briefly quite high profile um and

you know it it worked really well it was really successful project so we kind of we got that we

made a few more projects and then at some point i'm trying to remember when it must have been

probably 2011 maybe um i met tom at a python meetup in a pub and at the time i remember him saying

so i'm i'm working on this this thing it's kind of it's like a framework for building

rest apis in django it's like a django rest framework but i'm really struggling to think

of what to call it so um so i said well you know what about django rest framework and he said oh

that's a good idea i bet i bet that would do well with like seo you know that that'd be nice and

easily googleable um so you know uh step forward a few months eventually we end up hiring tom he

was our first employee after the three founders um and not a bad pickup yeah yeah um yeah no that

it was really good he was really kind of crucial in in shaping a lot of the early tech decisions

at dab apps for sure um and he kind of um he was working on client projects for us but at the same

time he was also working on rest framework and he if you remember one um quite a while ago he did a

kickstarter to fund some rest framework development and he did that work while he was working for us

and so we used to kind of sit opposite each other and bounce ideas around and

i think i'm kind of i don't i haven't contributed a huge amount to rest framework but

i think i'm responsible for like serializer method field the original version of that was me and like

um you know things like that um so so so that was fun um so that was kind of back in the day

and then yeah we we sort of transitioned over to because originally we did i guess you might call

it a classic django app where you do the templates on the back end you know it renders html on the

back end that was you know for the first few years that's what we did and then we moved over to this

sort of api driven um react app type approach obviously using using rest framework um and that's

that's what we've been doing ever since really although i have kind of recently got quite

interested in the whole unpoly htmx thing which i know you're all fans of well i've been using it

on projects and it's just amazing to be honest for me it really is just like oh and to go back to

writing templates and just oh i need to you know so i've got a i don't know a form view and i just

want to you know jazz it up a little bit and a couple of things and it just works and you're

like oh yeah this is yeah this is good this is good it is so i was going to ask you what do you

think so i i really like it so we've recently been i've i kind of i guess i htmx i really like

the philosophy so I think the guy who you know he's the main developer on htmx is is amazing and

I think he has some brilliant ideas and I think he's basically right about everything um but what

we've actually been doing from kind of a practical point of view we've been using something called

unpoly which is um I hesitate to call it a competitor because I think they're very much

in the same vein and they're very much they support each other um but unpoly is I would say

a bit more full featured from the point of view of actually building web apps so it's come out of

an agency uh in in germany i believe who do rails backends um but other than that they're very

similar to us um so it allows you to do stuff like it very easily lets you um navigate around

without doing full page loads kind of makes makes it into a bit of a single page app and that kind

of stuff so that's been interesting and it's nice and easy to integrate with with react where you

need to drop some react components in well this was kind of what one guy so if you're you're

essentially a react shop and you're you're fully invested in there and then a new technology comes

along how do you kind of smuggle that in or how do you have to do a sort of separate project to

experiment or do you know how do you get going with the new technology how do you give it a go

because that's that's always a question i'd like to use htmx but how can i you know the work you

already use some others so it is quite a high risk thing to do i would say change the tech stack

particularly in an agency where everything that you build kind of exists in parallel if you like

you know if you're a startup and you switch technology once you've deleted all your rails

code and rewritten it all in django then you can like forget rails ever existed and you never have

to think about it again whereas in an agency if you switch from rails to django you've still got

a whole load of rails projects that sit around that you're never going to be able to rebuild so

it is a high risk thing to do it's very important that projects have a similar structure so that

you can switch exactly exactly and you know people you know are context switching a lot and so um

yeah it's very very important that things look as similar as possible so like i say it's a high

risk thing to do i think that the key is to try it on a small scale first so try it on internal

projects which we did and we got really confident and comfortable with it being um something that

we could we could experiment with a bit further and then you start introducing it into sort of

smaller scale client projects and um it's been it's been really good so i'm like you know we're

not we're definitely not ditching react because we do build a lot of projects which are very much

what react is good for you know that very dynamic user interface type of um type of thing but yeah

we're experimenting i imagine that's when you switched over to react that was similar calculation

around you know is this something we can commit to is how sustainable react be right this is like

you've been using React for a while. I mean, React has changed quite a lot over the years.

That's true. I think React was maybe a bit of an easier decision because we knew that we had

to build the sort of things that React let you build. But before React came along, we were

building them in all sorts of slightly janky ways, you know, starting off with jQuery, and then

maybe we did a bit of Backbone, and we tried out a few different things, and none of them really

felt very solid. And then React came along, and it was like, oh, yeah, yeah, that makes sense.

someone solve that problem now carlton you're still doing objective c right no no no i'm just

kidding no i mean i i'd say that with a sort of tear in my eye because i do miss it um but this

sort of last 12 18 months it's all gone swift now it's like objective c doesn't quite you know

you can still do it but actually going back to it it's you don't remember it's not as nice as

you remember it being if that makes sense like you have these fond memories of objective c but then

And it's not as good as Swift.

It just isn't.

So I've kind of, you know, chopped up.

Anyway, that's it.

It worked at Django.

So project structure, you kind of mentioned that.

I'm curious.

So presumably you have within the agency a default that you like to use.

I'm curious what that looks like around, especially how you organize your apps.

Because we had Ned Batchelder on recently, and we were talking about, you know,

is there a code base that's mature

that people can look at as a best practice?

And he was rightly saying every large thing

is kind of has things baked in that if you could change you would um but since you're in an agency

you you have maybe more of a chance to have some uh greenfield projects where you can structure

things so i'm just curious how you like to do it because everyone kind of does it a little

differently at scale i find yeah i mean we try as much as possible to not deviate from

uh django's sort of best practices i guess so um we it looks a little different to what you

would get if you just ran start project um but but essentially our our project template for the

past probably 10 years has been essentially what django gives you with some react stuff in there

as well all of the apps go into a separate top level package so instead of having just a big

kind of list of apps in your root directory they go into a separate top level package

that package is always has the same name so it's always called project

rather than being called the name of the project, right?

So the apps are called project?

Well, no, just sort of the top-level package that all the apps are in, if you like.

So, yeah, I mean, it's not a million miles away.

Having said that, again, sort of recently over the past year or so,

we've started to experiment a little with slightly different structures.

I'm increasingly of the opinion that dividing things up into apps

is not always a good idea um a couple of reasons for that one is it's when you start a project

you often it's very difficult to figure out where to draw those lines between things

and if you're not sort of building for reusability in terms of those apps which usually are not when

you're writing bespoke code for someone else um drawing those lines in exactly the right places

is quite hard and it's then quite difficult to change it um if you get it wrong so if you

you know django's migration framework doesn't really handle well actually i just want to move

this model between apps very well or certainly it didn't the last time i tried to to do it

so okay then but you so the opposite of that is you've got a giant models.py file or

essentially so you have a models.py package which has in it a file for each model okay and then you

import them all at the root so you essentially you have you have a models app if you like which

has got all of your models in it and then separate to that you have an app which has all of your

views in it right because really you know views and models are almost at right angles to each of

the kind of cross-cutting uh domains right because often you want different views to all talk to the

same set of models and so packaging them up together um in in an app always feels a little

bit odd to me particularly in bigger projects yeah and you you're always got the view that

imports the model from the other app and exactly yeah okay interesting so i wanted to ask you about

your open source work then jamie because um there's a couple of big projects that i you know

just um so i'll ask you about a couple and then you can um riff on those and then if there's others

you want to tell us about and that would be awesome but the two that come to mind are zen

queries and then the newer one which is django readers so do you want to tell us the backstory

on those and you know what the motivation is and we can chat chat about those packages because

they're amazing folks you should check them out sure okay so zen queries let me first explain

what it is at its core it's a context manager which lets you say which lets you mark a block

of code as not being able to run queries not being allowed to run queries so if you try and

run a query i.e you know evaluate a query set inside that block it will raise an exception

you're just not allowed to do it now why would you want to do that um what we find particularly in

in big django projects and i guess this is kind of the classic object relational impedance mismatch

thing right so django encourages you to put your business logic in uh in in the model layer

essentially so you have model methods or you have query set methods which you put your code in and

then you call that code from a either a template usually if you're if you're doing your html on

the server thing or you call it from a serializer and often you know the templates can do things in

loops or they can do things in doubly or triply nested loops or serializers can be very deeply

nested if you're pulling back quite complex object graphs and what ends up happening is that you

end up with the classic n queries problem right so you accidentally without realizing it perhaps

access an attribute or call a function call a method on on a model which ends up incurring

one or more database queries and if you do that in a loop or particularly in a nested loop

you end up with tens or hundreds or i've seen thousands and thousands of queries from from

single views but there's there's not really any sort of visibility of that if you like you know

you can't see that happening um and that's because the uh the code that calls the methods or the

calls the code that accesses the attributes is sort of over here in your templates or in your

serializers but then the code that you have to write in order to make that efficient is over

here in your views usually right because you would usually define a query set attached to a view

and so what you end up with is this kind of weird like binding between the two things

and they're very dependent on each other but the dependencies aren't visible in the code

and so what I kind of you know thinking about this for a very long time you know lots of years and I

spent a lot of time tuning query sets and prefetching and select related you know django

gives you the tools to solve these problems right but it's always for me you create the project the

problems and then you have to solve them you shoot yourself in the foot and then you wait for it to

hit you know you splint it kind of thing why is this feeling slow exactly so so what i wanted was

a way of saying well actually how can we avoid this how can we make this not be a thing so the

The point of Zen queries is to allow you to say,

okay, as part of rendering this template

or as part of accessing this serializer.data,

when you're kind of serializing some stuff with a serializer,

you're not allowed to run any queries, right?

If a query happens as part of doing that thing,

it blows up, it will give you an exception.

And so what that forces you then to do

is to think about it when you're writing the code.

So you have to, before you access that attribute in your template,

you have to have select related or prefetch the query set.

And as soon as you do that, then it's fine and it all works.

So the idea is, so it gives you this context manager,

but then it also gives you a few little tools sort of built on top of the context manager.

So it gives you a replacement for django.shortcuts.render,

which will allow you to render a template wrapped in the context manager.

So you just switch your import at the top of your views file,

and it kind of, you know, all your templates blow up

when you're running queries in them.

Or it also can give you a serializer mix-in

and a REST framework API view mix-in,

which will kind of magically mean that when you try and access

your .data attribute on your serializer,

again, if that runs any queries, then it blows up.

Yeah, and that's a good way of enforcing a certain amount of discipline.

Exactly, yeah.

And it sounds arduous, but really it's not.

All it's doing is pushing a little bit of work

to the start of the project

rather than six months in when the client's upset

because their project's running really slow, right?

And so just as a matter of best practice,

you use this on every project from the beginning?

Exactly. Yeah, that's the idea.

And it gives you an escape hatch as well.

Like if you need to do something,

well, actually, you know, I know what I'm doing

and I really, really do just need to run a query here.

It also comes with another context manager called Queries Dangerously Enabled,

which turns the queries back on again just for that block.

That's nicely named, nicely named.

Exactly, yeah.

And similarly, there's a template tag, which does the same thing.

Okay, cool.

So any more questions on Zen Queries?

Have I explained it?

Does it make sense?

No, no, that's all fine.

I think I would just say give it a try because I've used it myself.

So the other one then is that I'm really excited about at the moment.

and i i'm still on the um the fence i haven't you know i've played with that i'm a bit like do i dare

use do i dare smuggle this into a real project it's um django readers so yeah you tell us about

what's going on on that one because that's quite yeah i'll try so this this is one of those things

which is is quite difficult to explain purely over audio it makes a lot more sense if you can

see the code um so this is if you like the next step on from from django zen query so if zen query

says well you can't run a query here but it doesn't really give you any help of how to structure

your code how to how to um you know correctly prepare in order to render things by um pre-fetching

and select relating and doing all the other things that you need to do in order to make that query

efficient so the origin of django readers was in another project which also came out of dab apps

written by Paul who's one of our engineers and that was called serialization spec mixing

and as you can imagine that was very tightly bound to Django REST framework so the idea of that

was that you would have a what's called a spec which is I guess kind of the closest thing that

you can think of is it's a bit like a GraphQL query but rather than being in your front-end

code it's in your backend code if you like so it's it's a property of a rest framework view

spec equals and then it's a list and in the list you can have strings and the strings are just

field names on on the model yeah and you can also have dictionaries and the dictionaries the key in

the dictionary is the is the name of a relationship so let's say you have you know a book with an

author or whatever your your key would be author and then the value of in the dictionary is another

a list which specifies which fields of the author you're interested in so you kind of nested

serialization format exactly so so it's a very simple um spec for saying here's all the fields

that i'm interested in and then what it does it generates both a query set which is sort of

surgically pulls out exactly the data that you want from the database and not only does it

automatically prefetch and select related and and all of those things it also uses dot only so that

it only pulls back exactly the fields that you've asked for uh so you know normally a django query

set does the equivalent of select star essentially it doesn't it explicitly lists out all the fields

but usually that's fine but if you know let's say you've got some huge text field in there then all

of a sudden that can be slow or it can use lots of memory so or you're fetching your your custom

user model which has got 300 fields on because you've used it oh here we go exactly this is the

running gag it feels like you have some experience there yeah but anyway but the point is it just

gets the first name last name it doesn't exactly yes so so uh it pulls out only the data that you

actually need to uh you know create your api endpoint shape and then the other thing that it

does is it generates your serializer for you so it creates a serializer class with some nested

serializers which precisely serializes exactly the thing that you need so you just write one

quite concise thing and you get efficient queries and serializers kind of free so paul wrote this

thing amazing like beautiful bit of api design i don't think he would be upset with me for saying

that the code was kind of gnarly um so it was very much one of those projects where it's like

let's get it to work of course he understood it all but um it wasn't the easiest code to follow

it wasn't the easiest thing to maintain and when we sort of tried to add features and fix bugs in

it it was like well actually there aren't many people who really understand this well enough to

do it so that's sort of that's one side of it so if you just park that in your brain that's the

that's the high level thing we'll come back to that in a minute okay so that then got me thinking

okay so what would this look like if what we if we built it in such a way that it was simple

okay so rather than being this really complex gnarly thing rather than being bound to rest

framework very tightly and only working with rest framework what if we could turn this into a bit

more of a generic concept so this goes back to that thing that i was talking about earlier around

around model methods and model attributes being very tightly bound to query set calls so

the two places where you generally put bespoke business logic in a django project

for you know for reads if you like for for returning data from the database

are model methods query set methods now imagine if you you have a model method so you know

def method self, that does some stuff, you know, accesses some attributes and then munges them

somehow. Let's say it's a full name and it concatenates the first name and the last name.

That's a really terrible thing to do because lots of people believe falsehoods about names.

But let's say that that's what you're doing.

Imagine instead of that being attached as a method onto a model, you just move it to the left one level of indentation, right?

And it becomes just a function in a module, okay?

So it takes a model as an argument and it returns a value of some sort, okay?

So that's one thing.

And the other thing is, instead of a query set method, so a query set method being, you know, model.objects.something, some custom bespoke thing that you've written in your application.

Again, move it one level to the left, indentation, have a function which takes a query set and returns a query set.

And in that function, it does some stuff to the query set, right?

So it prefetches something or it only is a field or whatever it's doing.

okay so now i've got i've got my business logic and instead of being it bound instead of it being

part of a model or part of a query set it's just a function okay what can i do with those functions

now i can put them in a data structure so if i if i said imagine a two tuple and the first item in

the two tuple or is it tuple i don't know how do americans pronounce it tuple tuple tuple tuple

i was i call it a tuple i've either i said tuple but you know you say tuple i said tuple

let's call the whole thing but i've heard smart people say both so a structure with two items in

it anyway so the first the first item in the two tuple is the query set function so the thing that

takes a query set and returns a query set the second item in the two tuple is the model function

the thing that takes the model and returns a value so what you've then got is you've expressed

a dependency between those two things.

So what you're saying is in order to efficiently call the model function,

you have to have first called the query set function on the query set.

Okay, so does that make sense?

So once you've got that structure, two things,

you can then compose those into big trees of things, right?

So you can have lots of query set functions

and you pass the query set through each one

and each one does its thing each one sort of you know modifies the query set in some way

and then you evaluate your query set and you then iterate over the query set and call each of the

functions that returns the values and it kind of i'm skipping some of the details deliberately

because that's a little bit hard to explain go and read the readme um okay so now we've got a

way of expressing these kind of dependencies between between business logic that operates

on an instance and business logic that operates on the query set well now I've got that I can go

and build serializing serialization spec mix in again right all I need to do is to take my list

of strings and I need to let's say let's say we're doing books and in that list of strings is title

all i need to do is replace that title string with a pair that has a thing that says to the

query set hey i need the title field and another thing which says go and get the title field off

the instance right so all of a sudden you have this kind of elegant way of building the thing

that we had before but was a bit gnarly um and what it what it sort of suggests is a slightly

different way of structuring your business logic in Django projects so instead of putting things

like I say instead of putting things on models and instead of putting things on custom query sets

you just write functions you can put those functions wherever you want in your code base

we have a kind of emerging standard of where to do that but it allows you to then only import

the things that you actually need to actually you know call that view you import exactly the the um

bits of functionality that you need and then you assemble them together using one of these specs

and then yeah it efficiently queries the database and and serializes the objects and you can do that

before you pass off to the render function say so that you know that you've got you've got exactly

the fields that you wanted from the database you got them in a in a the minimum or close to the

minimum number of queries possible and you're not going to cook um you're not going to cause any

more queries when you render the template or the serializer or or well or to cast it because you

you get back a dictionary right you get back a something that's json serializable yes yeah so

that that's the idea it's sort of i guess solving two birds with one solving two birds with one

stone that doesn't make sense uh solving two problems at the same time um having efficient

and and high performance views but also having um you know better ways of structuring code in

a django project yeah i mean and it's i mean do go and check it i love that discussion because

i'm beginners who read my books and stuff often ask kind of what's what's advanced django look

like and i often say it's going to be all about query sets um and i think your description just

proved that i i couldn't agree more i think i think the orm is by far the most important bit

of django you know like you can imagine almost any other bit of django looking kind of different or

replacing it with something else but the orm is really the bit which is most of the complexity

and most of the power is in there yeah i mean there was some talk about wrapping django in a

service layer um a couple of years ago or a year or so ago there's some blog posts went back and

forth and one of the points is but if you're not but if you're not using the jet the ORM like then

why are you using Django you know like you might you might go somewhere else if you weren't going

to use the ORM. I and I completely agree with that you know I think that that part of I mean

Django readers I suppose is starting to slightly go in that direction of sort of being a little

bit like a service layer but very deliberately I wanted to keep it very familiar to Django

developers so that the whole idea is that you can mix and match it with what you already know

about django it's not it's not hiding django away it's still very much django it's just doing it in

a slightly different different way i suppose and what's really nice about is the composability

it's like you could define the individual um bits to pull pull the fields out or the individual

query set um methods and then they just slot together like little legos it's lovely yeah and

And so you kind of described this, you mentioned the G word, GraphQL, before.

So you're using this to build what nice nested response is that a mobile client can get all the data it needs in a single go and that kind of thing.

Yeah, exactly.

Yeah, so we tend to try to put as much business logic as we can in the back end rather than in the front end.

You know, we try and keep our front ends quite minimal to the extent that they can be.

So we do end up with quite complex and quite bespoke API endpoints.

So I suppose we're maybe veering a little bit away from that restful purity of having very basic sort of flat representations of things

and moving more into this API endpoint serves this particular purpose.

It allows this page in the app to render, I guess, rather than the front end having to make millions of API calls and stitching them all together again.

Yeah, I mean, one thing we talked about when we had Carson, who's the HTMX creator on, was that it's kind of easier for the back-end developer to craft the API endpoint than it is for the front-end person to, you know, that you've kind of got a mismatch where the front-end person is like, I need this field and that field and that field.

It's much easier for the back-end person to put that together and say, look, here's your endpoint for that page or that request.

Yeah, exactly.

Exactly that.

Jamie, do you have any other great open source packages that you want to plug while you're on?

Because those are the two I'm excited about, but what else have you got?

Well, I find it a bit of an alien concept to plug open source packages.

I don't care if anyone else uses them.

Okay.

Well, what's on your lawn that you give away for you?

Sure, sure.

So we very much, we have an approach of open sourcing general purpose utility things, which are useful to us.

And if other people find them useful, then that's great.

So a couple of other things that we have, we have a project called Django DBQ, which is if you like a kind of lightweight replacement for Celery, you know, Celery really does two, I suppose you would use Celery in two places, very, very high performance, high throughput, highly parallelized, dealing with millions of jobs as quickly as you possibly can.

well fine but a lot of people don't really need that really all i want to be able to do is to

send an email when you press a button but not have the person have to wait until the email's

been sent before it comes back right i just want to ditch it into a queue and and kind of i don't

care unless there's like 10 10 jobs a day sort of thing precisely yes yeah so django dbq is a really

lightweight um entirely orm backed queue uh which just kind of it presents its worker as a as a

management command so um you just you know manage.py worker worker will start up and then you

can just create um instances of a job model and you can pass them some arguments and a function

to call and it'll call the function and do the thing and then it and then it's done so we use

that a lot on all of our projects and and we don't currently have any projects at all which use

celery which there's nothing wrong with celery but it's a very big complex beast and it's quite

operationally complex sometimes it's like using a sledgehammer to crack a nut

exactly yeah yeah yeah and then our other probably our um our most popular open source package is

called django log request id which is popular because it's linked to from the heroku documentation

although they they specifically link to like version 1.0 which is from years and years ago

which is a bit annoying i don't really know who to contact i don't think anyone's work there in

four years i mean with all seriousness i mean their build pack is four years old yeah or the

jango heroku thing that they linked to in their docs that kenneth worked on has been closed for

three years yeah yeah we use we host all of our stuff on heroku and it's absolutely fantastic but

it doesn't change very much it seems to be pretty you know it works and they don't

do much in the way of maintaining it um so yeah what django log request id does is very simply

allows you to every time you write a log message it will attach a uuid to the at the level of the

request so if you've got a concurrent environment with lots of requests happening all at the same

time and you're capturing the log output from that it can be very difficult to tell with lots

of interleaved log messages from different threads or different processes how they join up with each

other so what django log request id does is it sticks a uuid um in all of the log messages that

have come out from a single request so that you can then grep your logs or search your logs for

that uuid and that will give you a nice list of all of the log messages that that request created

which is which is really useful okay so that's like we have a few other sort of yeah we have a

few other small open source packages that do various simple things but they're the main ones

i think and the other thing you've got is a and there's a dab act blog as well which i you know

there's um article of tom's on there from years ago about um wrapping your django logic in a you

know manager methods and whatnot this is one of your canonical blog posts it's no it's just like

oh wow i will i will keep mentioning that blog post till the universe here we go it probably

directly contradicts the advice that i just gave in django readers no don't do it like that anymore

Well, it's about, I think the essence of that particular post is about having a single point of control for, say, creating a model instance so that if there is business logic that needs to be applied, it's applied inside that method and then all client code can call that one method and then you know it happened.

Whereas if you, in a view, if you just go, you know, model, new, dot save, model, dot save, who knows whether the business logic got applied because it's, you know, it's a bit wild west at that point.

I think that's the long and short of it, but, you know.

It is, exactly.

And we still follow that principle.

You know, it's not exactly the same to the letter, but, you know, general principles like that, you're absolutely right, are very much worth sharing and will improve everyone's Django code bases, I think.

There's lots of super posts on the Dab Apps blog,

so worth having a look at.

I should probably write some more, shouldn't I?

Yeah, no, well, this is kind of what I'm getting to.

We need to keep that going.

Yeah.

Speaking of improving your code base, testing.

Can I ask just what your kind of default testing kit

you throw at a project is?

Yeah, so we're maybe in the minority

in the sense that we just use what Django gives us.

So we use Django's testing base classes,

you know, test client, all that stuff.

We don't use PyTest.

We don't use anything like that.

Again, mostly because it's all just there in the docs

and it's how it tells you to do it.

And so it's really easy to just point people at that

and say, there you go, you know, do that.

We use a few things on top of that.

So we use model bakery to kind of create model instances.

We obviously use the sort of the built-in mocking stuff

off um but yeah basically just just django i mean a lot of the test cases come with helpers right

you know the transaction test case and these kind of things yeah yeah yeah i think that's great yeah

i mean i've been i i ask just because great to share with others and that's something that comes

up if we're you know having a beer i'd ask you and also i often in my books like i have a quite

a lot of testing um but i haven't yet gone down the pi test coverage um you know everything else

rabbit hole um because it is a lot more um it's it's used a lot it's helpful but it is you know

what you said with celery it's like well it's this whole other ecosystem we're rolling in and

um and it is true that you can get i mean i can test all the stuff i do for people just with the

built-in tools yeah i mean it's a bit weird because it's that whole you know it came out of j unit and

it doesn't it looks weird right it doesn't look like python code but somehow i was thinking about

this the other day somehow i kind of like it because you know that you're looking at test

code right rather than looking at business logic code because it's all it's all slightly weird

python with like you know uppercase camel case methods and things like that um so it doesn't

bother me weirdly i can never probably should i can never remember the name of the asserts though

i'm always like uh self assert what am i starting now like oh i you know i've been doing django for

12 years or something like that and i still keep the docs open all the time you know it's it's just

it's a big library isn't it so there's a lot of surface area to memorize so running a large agency

i'm curious how you train up your devs like what are do you do pair programming do you have internal

stuff do you have other resources you like to use because you know you you bring in more junior

people um what does that process look like yeah so we we have a few different things that we do

um the starting point for people who've never really done any django before is is always the

django docs i think that the the tutorial in there is is good enough um and that's a good place for

people to start we have an internal uh company handbook which we call the dab apps docs which

sort of builds on top of that and says you know given that you understand basically how django

works this is how we do django um so it just has dab apps specific stuff and it has a load of other

things in there around how we run the company and um uh you know front end stuff and testing stuff

and operational stuff and and all that kind of thing um and then yeah pair programming uh mentoring

um it's it's been quite difficult onboarding people uh in the past couple of years with

everyone working remotely because it's so it's so great to just have a new person and an experienced

person just sitting next to each other at a desk and you know shouting questions at each other but

we've we've adapted and we we try and have people on on zoom calls as much as possible and now we

are able to go back into the office to some extent we try and encourage new starters to go in at

least a bit of the time and um and sort of you know work with the person who's trying to to

onboard them i mean that's one of the things having someone who's more experienced who's

also interested in mentoring you is you you can't put a price tag on it i mean i think

on some level that's why all these CS grads can go into these large organizations and

it's not that the organization itself has great training I would say it's just that there are

experienced people and there's sort of there's enough slack that they can spend the time to

level you up because if you sit next to someone who knows what they're doing you know in three

months they'll get you what would take a year on your own. Oh yeah and we really try and encourage

a culture of asking questions that to me that's absolutely key it's far better to

risk appearing stupid and of course there's no such thing as appearing stupid like people

who've been programming for their entire lives are stupid questions it's just how programming works

but if you can get over that that that fear of of asking a silly question it's so much more

efficient than going away for a week and trying to figure out something for yourself and coming back

with probably the wrong answer but certainly an answer that someone solved before 100 times and

you could have fixed it in 10 minutes if you just asked and i think code review is really important

as well i think we use github and we use pull requests and and we make sure that everything

gets reviewed before it can be released to clients and and that's definitely not a sort of command

and control hierarchical thing it's not more senior people reviewing more junior people's code

it's also junior people reviewing senior people's code and that allows the juniors to see how

seniors think and to read code written by someone with more experience than them

and i think that's that's really helpful as well yeah i mean i would say as people move from

beginner to intermediate level programming specifically with django it's learning where

that line is in terms of how long do i spend on something and then when do i ask because i think

as a beginner, you know, a beginner needs to put in some degree of work, but, you know, don't spend

a week on something that, you know, maybe it's an N plus one issue, or maybe it's just, you know,

a simple config thing. Um, and I mean, I still do that. I just, just yesterday, I, I had a question

around forms and views and I did enough research that I was like, okay, I think I kind of know,

but I still want to know what the best practice is. So I asked Carlton, asked some other people

And, you know, obviously, like, I sort of know what's a good question to ask someone.

But to your point, I still ask all the time around certain things.

But it's usually more, you know, what's the best way rather than, you know, how do I get this to work?

Because I can get it to work.

But do you want to give a plug, Carlton, to rap reviews?

Just because I am throwing it in the 3.2 update to my books.

But it's something close to your heart.

So the question Will asked yesterday was about, say you've got a view that you want to respond to differently to, I don't know, something on the request.

Like, you know, it might be the content type.

Like a detail view of the form.

Yeah, okay.

So you want to handle the detail view and the form view in the same thing.

And then, you know, so you can have a nice two separate views, one that just returns the detail view and one that handles the form.

And then you can have a kind of view that decides which to dispatch to in front of them.

It's kind of a nice pattern because you can do that with a list view

that you also want to be an RSS view or JSON response for the same thing.

And it's a nice pattern.

I was like, no, Will, do include that in the example

because I think that's something that's really handy to learn is this.

Because if you put it all in one big fat view, it gets really complicated.

I'm a really big fan of simple views.

i have to confess i'm not a huge fan of django's generic views django's well django's kind of class

based views i guess um to me i there's there's a fantastic uh quite short book um written by

luke plant called django i think it's called django views the right way that's a really really

good read and it kind of argues that the way django views encourages you to compose complex

functionality is quite confusing quite difficult to follow quite difficult to read and you only

can really follow it if you know the api really really well or if you kind of almost have the

source code open next to you so you're kind of going okay so that method is called by this thing

that's on that base view mix in and then that one calls that one and it's just this huge nest of

things calling other things and often it's just easier to write the damn code out you know

even if you're repeating yourself it's just clear and you can just see this happens then that happens

then that happens and then that happens done right and i'm a big fan of that exactly um i think

the class-based views are kind of as they're in they're just too clever then they're they're

they're kind of the best example i know of what goes wrong with object-oriented programming like

you know this good code it's great code it's super tested it's it's super well documented

it's got classy class-based views for when you you know when you need to use it but

that's not maintainable easy to reason about you know software carlton i have to read your tweet

because we had this exact discussion where i even said to carlton like i look i have it working

function-based views like but i really i'm i'm somewhat committed to the generic class-based

views and getting them you know tweaking it properly and carlton said and i quote if post

loads all logic and all in one place else loads more logic and all in one place versus breaking

it up so i agree with that but that is that you know to your point yeah you are going to duplicate

yourself a little bit but you know it's weighing that versus this big inheritance chain and and

part of why you split up class-based views is that when you have a detail view and another mix in

another mix in you know that gets wonky fast with in subtle ways so but as well a lot of the mixing

methods they're like um it'll be just like um self um the object is equal to self.get object

and then that will be the sub method a single object that will be the super method that you'll

call so you'll get so in your your method you go super um you know this method and then you write

some other logic and then so when you're looking at that six months later you have to go and look

at what the super method is but the super method was one line well from that six months later you

is much better off having that one line in front of you,

even if it was duplicated,

than having to dig through two or three files

to see what's actually going on in this method.

I would argue, and Tom's argued,

and Jamie, you're nodding, so you clearly agree.

I completely agree, yeah.

Come on, let's do our weekly plug

for Django Vanilla Views as well then,

because Tom Christie wrote this alternate set

of class-based views called Django Vanilla Views,

which is a nice example of a different way

of implementing the same functionality

but in a more sort of inline declarative manner.

Yeah, and again, Django Vanilla View

is another thing that Tom was working on

when he was at DevApps

and I seem to remember he wrote a blog post about that

which is probably still on our blog as well somewhere.

As we wrap up, I always want to ask about deployment.

You mentioned you use Heroku.

Are you using GitHub Actions

or what's the standard flow for deployments for you all?

Okay, so we very much try to keep our sort of

ops and deployment approach simple

and I don't like running servers.

I've done it for a long time

and I think that it's a big overhead

and you have to almost hire a team of people

to do it properly, you know,

with the proper security patches

and keeping things up to date.

So we use GitHub Actions for CI.

What we do is kind of,

we work in a pull request workflow.

So we'll have, let's say that we have a project

that's been live for a while

and we're working on a new package of work

that might consist of, I don't know, three features and two bug fixes.

Each one of those will go into its own pull request

against the main branch.

Each one will be reviewed independently.

We'll then build what we call a release branch,

which is essentially another pull request

which merges all of those together.

And that's another pull request into the main branch.

That will then get deployed to a staging environment.

And in order to do our deployments,

we just use the heroku web ui you know we we go in and we have a rule where we we pair on

deployment so you always have two people making sure that you're clicking the right button on

the right project kind of thing but you um yeah you know you you go in you press the button it

deploys it to the staging environment you test it internally the client signs it off and then

same process for deploying to production click a button and i'm i'm very keen on that approach

definitely sounds rock solid to me just always like to ask anything else you want to bring up

well i mean all i would say is um we're always on the lookout for good django people in the uk

so if there's anyone who fancies working in a an environment with a lot of knowledgeable django

people doing an interesting mixture of different types of client projects you know we really don't

specialize in any particular type of client we have all sorts of different projects in completely

different sectors all going on in parallel at the same time so it can be really really interesting

so if anyone is interested drop me an email jamie at dabapps.com um even if you don't see a suitable

job ad on our website just drop me an email anyway because we're always on the lookout yeah fun okay

super well i'm gonna say thank you for coming on like you know really really exciting um to

talk about your projects and really interesting on you know i'm so hyped about jango readers and

you know i bumped into it a little while ago and there was an issue that was is this usable

in production which you recently closed as yes 1.0 yes i tagged it 1.0 we're using in production

so that means that everyone else can okay super well that's that's honestly that's that's really

good to hear because i think you know i'm always slightly concerned that ideas that i have are a

bit crazy and so to hear someone else who's very very knowledgeable about django thinking that it

might be a good idea makes me very happy so yeah no i mean it looks it looks right in the in the

ballpark you know there's there's caters there's attas and caters it's pedantic there's you know

and in my sort of project folder django readers now is in there as like you know these are these

are the contemporary approaches to serialization so super i'm excited to be that's a good tagline

a contemporary approach to serialization that's trademark me right anyway let's call it quits

jamie thank you for coming on thank you for joining us everybody um we're uh django chat

dot com and chat django on twitter join us next time bye bye thanks jamie thanks bye