Transcript: Optimizing Django Queries - Jamie Matthews

Hi, this is Will.

I'm offering a Black Friday sale, 50% off my three books, Django for Beginners, APIs, and Professionals.

Good until end of day Friday, November 26th.

Link in the notes.

On to the show.

I'm Carlton Gibson, joined to several Will Vinson.

Hello, Will.

Hi, Carlton.

Hello, Will.

So we're gonna chat about all those things.

Hey Jamie, thanks for coming on.

Thanks for having me on.

Quite excited.

You've been um producing some really interesting things that I'm um keen to dig into.

But before we do before we do all of that, um let's do our traditional um what's your backstory and how'd you get in?

And tell us about your your how do you find Django?

So I'm uh technical director of Dab Apps.

I suppose in America that might be CTO.

Um we're a kind of web and mobile agency, so we build bespoke web apps, mobile apps, uh usually with API backends of some sort.

Um we use Django for all of our server-side code.

We're based in Brighton, which is on the south coast of England, and we've got an office in Wales as well.

Uh about 30 people, 17, 18 devs, uh mixture of back end and front end, full stack.

Um, and we've probably built over the years, I was trying to add it up, it's definitely over 50, and it might be kind of closer to 100 Django projects.

Um, anything from like small code bases, you know, a few thousand lines of code, up to our biggest projects are sort of 70,000, 80, 100,000 lines of Python.

Wow.

Um so we have some some really big projects, or well, relatively speaking, I think.

Did you say?

Yeah.

I mean, you've got, you know, there's a lot of turnover of work there.

Yeah, indeed.

We're I I suppose we're we'd be considered fairly big.

Most of our clients uh don't care or don't know or care about the the the stack, other than the fact that you know they're they're happy that we're we're good at making decisions about what technology to use.

Um we have we do sometimes have clients who come to us specifically for Django.

Um that tends to be clients who have existing code bases that maybe they've had someone else build and and they need that sort of taking over from people who who know how it works.

So uh could I just ask about the front end integration?

Like how do how do you um map between Django and the back end and what's going on on the front?

And because that's quite an interesting.

Um so yeah, essentially we we build um APIs with Django Rest framework, um, and we have uh you know yeah, React and React Native apps that that talk to them.

So um I think I found I I was trying to remember where I first used Python.

I definitely came out, I did um a computer science degree in Manchester, and I definitely came out of that with an awareness of Python.

I can't remember whether I used it as part of the course or whether I just sort of found it in my, you know, in my free time.

Um I then moved down to Brighton to do a master's, and I used Python quite a lot there, doing kind of scientific computing type of stuff.

When I finished my master's, I went and got a job as a web developer but doing PHP.

Um but I used to go so Brighton has this really amazing sort of tech scene with lots of meetups.

And there was one which I used to go to a lot, which believe it or not was called Flash Brighton, right?

Do you remember Flash?

Wasted half my teenage years with that.

So so I didn't really do very much Flash, but I kind of went there because when they didn't have Flash topics to talk about, they got all sorts of other people and all sorts of other technologies.

And one of the speakers there was Simon Willison, who of course was one of the original co-creators of Django, who at the time lived in Brighton.

Um, so he was giving a talk about Django.

This would have been, I think, I think it was early 2009, so it would have been just after 1.0 came out and kind of just before 1.1 came out.

Um so I went and saw this talk.

Obviously, I was doing a lot of web dev at the time.

I really liked Python, and this was like, wow, now I can build websites in Python.

This is the best thing I've ever seen.

So I went away and kind of dabbled with it.

Funnily enough, the very first bit of Django that I used was Inspect DB, which I imagine basically nobody uses.

Um, because the what I did was we had this kind of internal tool, it was some sort of inventory management system, which was like a PHP app that talked to a MySQL database.

And I kind of went, oh, watch this, you know, and went and pointed InspectDB at the database, generated all the models, wired them up to the admin, and within about five minutes I just had this like working admin system for this for this app.

And of course, everyone was very impressed, and they all went, Wow, well done, let's get back to writing PHP.

Um so um, so yeah, I did that for a while, played with Django in my spare time, you know, got a book about it, learned as much as I could.

And then eventually myself and a couple of colleagues from my first job decided that we were kind of going to go off and do our own thing, set up dab apps.

From day one, it was it was all about Django.

Um our first project was uh it was for a company who are a supplier of Tesco, which I'm sure everyone's heard of.

Um, and so our very first project was kind of quite high profile, very briefly featured on like the front page of the BBC News, and it didn't fall over.

Walmart or something.

Yeah, yeah.

Yeah.

Um, yeah, so it was briefly quite high profile.

Um, and you know, it it it it worked really well and was a really successful project.

So we kind of we got that, we made a few more projects, and then at some point, I'm trying to remember when it must have been probably 2011, maybe.

Um I met Tom at a Python meetup in a pub.

And at the time I remember him saying, So I'm I'm working on this this thing, it's kind of it's like a a framework for building REST APIs in Django.

It's like a Django REST framework, but I'm really struggling to think of what to call it.

So I said, Well, you know, what about Django REST framework?

And he said, Oh, that's a good idea.

I bet I bet that'd do well with like SEO, you know, that that'd be nice and easily Googleable.

Um so you know, uh stepped forward a few months, eventually we ended up hiring Tom.

He was our first employee after the three founders.

Um and

Yeah, yeah.

He he was really kind of crucial in in shaping a lot of the early tech decisions at Dab Apps for sure.

Um and he kind of um he was working on client projects for us, but at the same time he was also working on rest framework.

And he, if you remember one of um quite a while ago, he did a Kickstarter to fund some REST framework development, and he did that work while he was working for us.

Um and so we used to kind of sit opposite each other and bounce ideas around.

And I think I'm kind of I don't I I haven't contributed a huge amount to REST framework, but I think I'm responsible for like serialize a method field.

The original version of that was me, and like um, you know, things like that.

Um so so so that was fun.

Um so that was kind of back in the day, and then yeah, we we sort of transitioned over to because originally we did I guess you might call it a classic Django app where you do the templates on the back end, you know, it renders HTML on the back end.

That was you know, for the first few years that's what we did, and then we moved over to this sort of API-driven um React app type approach, obviously using using REST framework.

Um, and that's that's what we've been doing ever since, really.

Although I have kind of recently got quite interested in the whole unpolyhtmx thing.

Well, uh I've been using it on projects and it's just amazing to be honest.

For me, it really start, oh and to go back to writing templates and just oh, I need to, you know.

So I've got a I don't know, a form view, and I just want to, you know, jazz it up a little bit and a couple of things, and it just works, and you're like, oh yeah, this is this is good.

This is good.

So we've recently been I I've I kind of I guess I uh HTMX, I really like the philosophy.

So I think the guy who you know is the main developer on HTMX is is amazing, and I think he has some brilliant ideas, and I think he's basically right about everything.

Um, but what we've actually been doing from kind of a practical point of view, we've been using something called Unpoly, which is um I hesitate to call it a competitor because I think they're very much in the same vein and they're very much they support each other.

Um but Umpoly is I would say a bit more full featured from the point of view of actually building web apps.

So it's come out of a an agency uh in Germany, I believe, who do Rails backends.

Um but other than that, they're very similar to us.

Um so it allows you to do stuff like it very easily lets you um navigate around without doing full page loads, kind of makes makes it into a bit of a single page app and that kind of stuff.

So that's been interesting and it's nice and easy to integrate with with React where you need to drop some React components in.

So if you're you're essentially a React shop and you're you're fully invested in there, and then a new technology comes along, how do you kind of s smuggle that in?

Or how do you have to do a sort of separate project to experiment?

Or do you you know how how do you get going with the new technology?

How do you give it a go?

Because that's it's always a question.

I I'd like to use HTMX, but how can I, you know, like the work you already use some other snow?

You know, if you're a startup and you switch technology, once you've deleted all your Rails code and rewritten it all in Django, then you can like forget Rails ever existed and you never have to think about it again.

Whereas in an agency, if you switch from Rails to Django, you've still got a whole load of Rails projects that sit around that you're never going to be able to rebuild.

So it is a high risk thing to do.

It's very

Exactly.

And you know, people you know are context switching a lot, and so um, yeah, it's very, very important that things look as similar as possible.

So, like I say, it's a high risk thing to do.

I think that the key is to try it on a small scale first.

So try it on internal projects, which we did and we got really confident and comfortable with it being um something that we could we could experiment with a bit further, and then you start introducing it into sort of smaller scale client projects, and um it's been it's been really good.

So I'm like, you know, we're not we're definitely not ditching React because we do build a lot of projects which are very much what React is good for, you know, that very dynamic user interface type of um type of thing.

But yeah, we're experimenting.

Is how sustainable will React be?

Right?

This is like pre-ho you've been using React for a while.

I mean, React has changed quite a lot over the years.

I think React was maybe a bit of an easier decision because we knew that we had to build the sort of things that React let you build, but before React came along, we were building them in in all sorts of slightly janky ways, you know, starting off with jQuery, and then maybe we did a bit of backbone and we tried out a few different things and none of them really felt very solid.

Um and then React came along and it was like, oh yeah, yeah, that that makes sense.

Someone solved that problem now.

No, no, no.

Um, but this sort of last 12-18 months, it's all gone Swift now.

It's like Objective-C doesn't quite, you know, you can still do it, but actually going back to it, it's you don't remember it's not as nice as you remember it being, if that makes sense.

Like you have these fond memories of Objective-C, but then it's not as good as Swift.

It just isn't.

So I've kind of chopped up.

I'm curious.

So presumably you have uh within the agency a default that you like to use.

I'm curious what that looks like around, especially how you organize your apps, because we had um Ned Batchelder on recently, and we were talking about you know, is there an a code base that's mature that people can look at as a best practice?

And he was rightly saying every large thing is kind of has things baked in that if you could change you would.

Um but since you're in an agency, you you have maybe more of a chance to have some uh greenfield projects where you can structure things.

So I'm just curious how you like to do it, because everyone kind of does it a little differently at scale, I find.

So um we it looks a little different to what you would get if you just ran start project.

Um, but but essentially uh our our project template for the past probably 10 years has been essentially what Django gives you with some React stuff in there as well.

All of the apps go into a separate top level package.

So instead of having just a big kind of list of apps in your root directory, they go into a separate top level package.

That package is all always has the same name, so it's always called project, um, rather than being called the name of the project, right?

So the

Um so yeah, I mean it's not a million miles away.

Having said that, again, sort of recently over the past year or so, we've started to experiment a little with slightly different structures.

I'm very I'm increasingly of the opinion that dividing things up into apps is not always a good idea.

Um a couple of reasons for that.

One is it's when you start a project, you often it's very difficult to figure out where to draw those lines between things.

Um and if you're not sort of building for reusability in terms of those apps, which usually are not when you're writing bespoke code for someone else, um, drawing those lines in exactly the right places is quite hard, and it's then quite difficult to change it if you get it wrong.

So if you, you know, Django's migration framework doesn't really handle, well, actually, I just want to move this model between apps very well.

Or certainly it didn't the last time I tried to do it.

Okay, and then you import them all at the root.

So you essentially you have you have a models app, if you like, which has got all of your models in it, and then separate to that you have an app which has all of your views in it, right?

Because really, you know, views and models are uh are almost at right angles to each other, kind of cross-cutting uh domains, right?

Because often you want different views to all talk to the same set of models, and so packaging them up together um in in an app always feels a little bit odd to me, particularly in bigger projects.

So I wanted to ask you about your open source work then, Jamie, because um there's a couple of big projects that I you know just um I'll ask you about a couple and then you can um riff on those, and then if there's others you want to tell us about, then that would be awesome.

But the two that come to mind are ZenQueries and then the the newer one, which is Django Readers.

So, do you want to tell us the backstory on those and you know what the motivation is, and we can chat chat about those packages because they're amazing.

Folks, you should check them out.

So ZenQueries, let me first explain what it is.

At its core, it's a context manager which lets you say, which lets you mark a block of code as not being able to run queries, not being allowed to run queries.

So if you try and run a query, i.e., you know, evaluate a query set inside that block, it will raise an exception.

You're just not allowed to do it.

Now, why would you want to do that?

Um what we find particularly in in big Django projects, and I guess this is kind of the classic object relational impedance mismatch thing, right?

So Django encourages you to put your business logic in uh in model in the model layer, essentially.

So you have model methods or you have query set methods, which you put your code in and then you call that code from a either a template, usually, if you're if you're doing your HTML on the server thing, or you call it from a serializer.

And often, you know, the templates can do things in loops, or they can do things in doubly or triply nested loops, or serializers can be very deeply nested if you're pulling back quite complex object graphs.

Um and what ends up happening is that you end up with the classic nqueries problem, right?

So you um accidentally, without realizing it perhaps, uh access an attribute or call a function, call a method on a model, which ends up incurring one or more database queries.

And if you do that in a loop, or particularly in a nested loop, you end up with tens or hundreds, or I've seen thousands and thousands of queries from single views.

But there's there's not really any sort of visibility of that, if you like.

You know, you can't see that happening.

Um, and that's because the uh the code that calls the methods or the calls that the code that accesses the attributes is sort of over here in your templates or in your serializers, but then the code that you have to write in order to make that efficient is over here in your views, usually, right?

Because you would usually define a query set attached to a view.

And so what you end up with is this kind of weird like binding between the two things, and they're very dependent on each other, but the dependencies aren't visible in the code.

And so what I kind of you know, I think about this for a very long time, you know, lots of years, and I spent a lot of time tuning query sets and pre-fetching and select related, you know, Django gives you the tools to solve these problems, right?

But it's always for me, you create the pro the problems and then you have to solve them.

You shoot yourself in the foot and then you wait for it to hit to hit, you know, you splint it, kind of thing.

So so what I wanted was a way of saying, well, actually, how can we avoid this?

How can we make this not be a thing?

So the the the point of Zen queries is to allow you to say, okay, as part of rendering this template or as part of accessing this serializer.data, you know, when you're kind of serializing some some stuff with a serializer, you're not allowed to run any queries, right?

If if you if a query happens as part of doing that thing, it blows up, it'll give you an exception.

And so what that forces you then to do is to think about it when you're writing the code.

So you have to, before you access that attribute in your template, you have to have select related or pre-fetched the query set.

And as soon as you do that, then it's fine and it all works.

Um, so the idea is so it gives you this context manager, but then it also gives you a few little tools sort of built on top of the context manager.

So it gives you a replacement for Django.shortcuts.render, which will um allow you to render a template wrapped in the context manager.

So you just switch your import at the top of your views file and it kind of, you know, all your all your templates blow up when you're running queries in them.

Um or it also can give you a um a serializer mix-in and uh uh a REST framework API view mix in, which will kind of magically mean that it when you try and access your dot data attribute on your serializer, again, if that runs any queries, then it blows up.

And it's it it sounds arduous, but really it's not.

All it's doing is pushing a little bit of work to the start of the project rather than six months in when the client's upset because their their project's running really slow, right?

Yeah, that's the idea.

And it gives you an escape hatch as well, like if you need to do something, well, actually, you know, I know what I'm doing and I really, really do just need to run a query here.

It also comes with a uh another context manager called queries dangerously enabled, which turns the queries back on again just for that block.

Okay, cool.

So any any more questions on Zen queries?

Have I explained it?

Does it make sense?

I think I'd I'll just say give it a try because I've used it myself.

No, so the other one then is um that's I'm really excited about at the moment.

I don't know I I'm still on the um the fan.

I haven't you know I've played with that.

I'm a bit like do I dare use do I dare smuggle this into a real project?

It's um Django readers.

So yeah.

You tell us about what's going on on that one, because that's quite extreme.

So this this is one of those things which is is quite difficult to explain purely over audio.

It makes a lot more sense if you can see the code.

Um so this is if you like the next step on from Django Zen query.

So if Zen query says, well, you can't run a query here, but it doesn't really give you any help of how to structure your code, how to how to um you know correctly prepare in order to render things by um pre-fetching and select relating and doing all the other things that you need to do in order to make that query efficient.

So the origin of Django Readers was in another project which also came out of DabApps written by um Paul, who's one of our engineers, and that was called serialization spec mixing.

And as you can imagine, um that was very tightly bound to Django REST framework.

So the idea of that was that you would have a what's called a spec, which is I guess kind of the closest thing that you can think of, is it's a bit like a GraphQL query, but rather than being uh in your front-end code, it's in your back-end code if you like.

So it's it's a property of a REST framework view, spec equals, and then it's a list.

And in the list you can have strings, and the strings are just field names on the model.

And you can also have dictionaries, and the dictionaries, the key in the dictionary is the is the name of a relationship.

So let's say you have you know a book with an author or whatever, your your key would be author, and then the value of in the dictionary is another list which specifies which fields of the author you're interested in.

So you've got your

So so it's a very simple um spec for saying, here's all the fields that I'm interested in.

And then what it does, it generates both a query set which is sort of surgically pulls out exactly the data that you want from the database.

So not all not only does it automatically pre-fetch and select related and all of those things, it also uses dot only, so that it only pulls back exactly the fields that you've asked for.

Uh so you know, normally a Django query set does the equivalent of select star essentially.

It doesn't, it explicitly lists out all the fields.

But usually that's fine.

But if you know, let's say you've got some huge text field in there, then all of a sudden that can be slow or it can use lots of memory.

So or you're fetching your your custom user model, which has got 300 fields on because you've used it as a running gag.

So so uh it it it pulls out only the data that you actually need to uh you know uh create your API endpoint shape.

And then the other thing that it does is it generates your serializer for you.

So it creates a serializer class with some nested serializers, which precisely serializes exactly the thing that you need.

So you just write one quite concise thing and you get efficient queries and serializers kind of free.

So Paul wrote this thing, amazing, like beautiful bit of API design.

I don't think he would be upset with me for saying that the code was kind of gnarly.

Um so it was very much one of those projects where it's like, let's get it to work.

Of course, he understood it all, but um it it wasn't the easiest code to follow, it wasn't the easiest thing to maintain.

And when we sort of tried to add features and fix bugs in it, it was like, well, actually, there aren't many people who really understand this well enough to do it.

So that's sort of that's sort of one side of it.

So if you just park that in your brain, that's the top that's the high-level thing.

We'll come back to that in a minute.

Okay, so that then got me thinking, okay, so what would this look like if what we if we built it in such a way that it was simple?

Okay, so rather than being this really complex gnarly thing, rather than being bound to rest framework very tightly and only working with rest framework, what if we could turn this into a bit more of a generic concept?

So this goes back to that thing that I was talking about earlier around um query uh around model methods and model attributes being very tightly bound to query set calls.

So the two places where you generally put bespoke business logic in a Django project for you know for reads, if you like, for for returning data from the database, are model methods, query set methods.

Now imagine if you you have a model method, so you know, def method self that does some stuff, you know, accesses some attributes and then munges them somehow.

Let's say it's a full name and it concatenates the first name and the last name.

I know that's a really terrible thing to do because lots of people believe falsehoods about names, but let's say that that's what you're doing.

Um imagine instead of that being attached as a method onto a model, you just move it to the left one level of indentation, right?

And it becomes just a function in a module, okay?

So it takes a model as an argument and it returns a value of some sort, okay?

So that's that's one thing.

The other thing is what instead of a query set method, so a query set method being you know uh model.objects.something, some cut custom bespoke thing that you've written in your application, again, move it one level to the left, indentation, have have a function which takes a query set and returns a query set.

And in that function, it it does some stuff to the query set, right?

So it pre-fetches something or it only is a field or whatever it whatever it's doing.

Okay, so now I've got I've got my business logic, and instead of being it bound, instead of it being part of a model or part of a query set, it's just a function.

Okay, what can I do with those functions now?

I can put them in a data structure.

So if I if I said imagine a two tuple, and the first item in the two tuple, or is it tuple?

I don't know, how do Americans pronounce it?

Tuple, tuple, tuple, tuple?

Something I was I call it a tuple.

Let's call the whole thing.

Anyway, so the first the first item in the two tuple is the query set function.

So the thing that takes a query set and returns a query set.

The second item in the two tuple is the model function, the thing that takes the model and returns a value.

So what you've then got is you've expressed a dependency between those two things.

So what you're saying is in order to efficiently call uh the model function, you have to have first called the query set function on the query set.

Okay, so does that make sense?

So once you've once you've got that structure, two things, you can then compose those into big trees of things, right?

So you can have lots of query set functions and you pass the query set through each one, and each one does its thing.

Each one sort of, you know, modifies the query set in some way.

And then you evaluate your query set and you then iterate over the query set and call each of the functions that returns the values, and it kind of I'm I'm skipping some of the details deliberately because it's a little bit hard to explain.

Go and read the README.

Um Okay, so now we've got a way of expressing these kind of dependencies between um between business logic that operates on an instance and business logic that operates on the query set.

Well, now I've got that, I can go and build serializing serialization spec mix in again, right?

All I need to do is to take my list of strings and I need to let's say, let's say we're doing books, and in that list of strings is title.

All I need to do is replace that title string with a pair that has a thing that says to the query set, hey, I need the title field, and another thing which says, Go and get the title field off the instance, right?

So all of a sudden you have this kind of elegant way of building the thing that we had before but was a bit gnarly.

Um and what it what it sort of suggests is a slightly different way of structuring your business logic in Django projects.

So instead of putting things, like I say, instead of putting things on models and instead of putting things on custom query sets, you just write functions.

You can put those functions wherever you want in your code base.

We have a kind of emerging standard of of where to do that.

Um but it allows you to then uh only import the things that you actually need to actually you know call that view.

You import exactly the the um bits of functionality that you need, and then you assemble them together using one of these specs, and then yeah, it efficiently uh queries the database and and serializes the objects.

You got them in a in a the minimum or close to the minimum number of queries possible, and you're not gonna cook um you're not going to cause any more queries when you render the template or the serializer or or well, or to cast its because you you get back a dictionary, right?

You get back a something that's JSON serializable.

Yeah.

It's sort of, I guess solving two birds with one solving two birds with one stone, that doesn't make sense.

Uh solving two problems at the same time, um having efficient and and high performance views, but also having um, you know, uh better ways of structuring code in in a Django project.

And I often say it's gonna be all about query sets.

Um and I think your description just proved that.

I think I think the ORM is by far the most important bit of Django, you know.

Like you can imagine almost any other bit of Django looking kind of different or replacing it with something else, but the ORM is really the bit which is most of the complexity and most of the power is in there.

There's some blog posts went back and forth.

And one of the points is but if you're not but if you're not using the j the ORM, like then why use Django?

You know, like you might you might go somewhere else if you weren't going to use the ORM.

You know, I think that that part of I mean Django Readers, I suppose, is starting to slightly go in that direction of sort of being a little bit like a service layer, but very deliberately I wanted to keep it very familiar to Django developers.

So that the whole idea is that you can mix and match it with what you already know about Django.

It's not it's not hiding Django away, it's still very much Django, it's just doing it in a slightly different, different way, I suppose.

It's like you could define the individual um bits to pull pull the fields out or the individual query set um methods, and then they just slot together like little Legos, it's lovely.

Yeah.

And so you kind of describe this, you use the you said you mentioned the G word GraphQL before.

So you're building using this to build what nested nice nested response is that a mobile client can get all the data it needs in a single go and that kind of thing.

Yeah, exactly.

You know, we try and keep our front ends quite minimal to the extent that they can be.

Um, so we do end up with quite complex and quite bespoke API endpoints.

So we, you know, I suppose we're maybe veering a little bit away from that restful purity of having um, you know, very basic sort of flat representations of things and moving more into this API endpoint serves this particular purpose.

It it allows this this page in the app to render, I guess, um, rather than the front end having to make millions of API calls and stitching them all together again.

It's much easier for the back-end person to put that together and say, look, here's your endpoint for that page or that request.

Because those are the two I'm excited about, but what else have you got?

Like I don't care if anyone else uses them.

Okay.

So so we very much we have a an approach of open sourcing general purpose utility things which are useful to us.

Uh uh, and if other people find them useful, then that's great.

So a couple of other things that we have.

We have a project called Django DBQ, which is if you like a kind of lightweight replacement for celery, um, you know, celery really does two.

I suppose you would use celery in two places.

Very, very high performance, high throughput, highly parallelized, dealing with millions of jobs as quickly as you possibly can.

Well, fine, but a lot of people don't really need that.

Really, all I want to be able to do is to send an email when you press a button, but not have the person have to wait until the email's been sent before it comes back, right?

I just want to ditch it into a queue and and kind of I don't care.

So Django DBQ is a really lightweight, um, entirely ORM-backed queue, uh, which just kind of it presents its worker as a as a management command.

So um you just, you know, manage.py worker, the worker will start up, and then you can just create um instances of a job model and you can pass them some arguments and and a function to call, and it'll call the function and do the thing, and then it and then it's done.

So we use that a lot on all of our projects, and and we don't currently have any projects at all which use Celery, which there's nothing wrong with Celery, but it's a very big, complex beast and is quite operationally complex.

Yeah, yeah, yeah.

And then our other, probably our um our most popular open source package is called Django Log Request ID, which is popular because it's linked to from the Heroku documentation.

Although they they specifically link to like version 1.0, which is from years and years ago, which is a bit annoying.

I don't really know who to contact with the code.

No, I mean, with all seriousness, I mean their build pack is four years old.

Yeah, that or the Django Heroku thing that they linked to in their docs that Kenneth worked on is c been closed for three years.

We use we host all of our stuff on Heroku and it's absolutely fast fantastic, but it doesn't change very much.

It seems to be pretty, you know, it works and they they don't do much in the way of maintaining it.

Um so yeah, Jet what Django Log Request ID does is very simply um allows you to, every time you write a log message, it will attach a UUID to the requ at the level of the request.

So if you've got a concurrent environment with lots of requests happening all at the same time and you're capturing the log output from that, it can be very difficult to tell with lots of interleaved log messages from different threads or different processes how they join up with each other.

So what Django log request ID does is it sticks a UUID in all of the log messages that have come out from a single request so that you can then grep your logs, or search your logs for that UUID, and that will give you a nice list of all of the log messages that that request created, which is which is really useful.

That's like requesting the code.

No, don't do it like that anymore.

Whereas if you if in a view, if you just go, you know, model new.save model.save, who knows whether the business logic got applied because it's you know it's a bit wild west at that point.

Um I think that's the long and short of it, but

Um and we still follow that principle.

You know, it's not exactly the same to the letter, but you know, general principles like that, you're absolutely right, are are very much worth sharing and will improve everyone's Django code bases, I think.

We need to keep that going.

Speaking of improving your code base testing, can I ask just what your kind of default testing kit you throw at a a project is?

Like what

So we use Django's testing base classes, you know, test client, all that stuff.

We don't use PyTest, we don't use anything like that.

Um again, mostly because it's all just there in the docs and it's how you how it tells you to do it.

And so it's really easy to just point people at that and say, there you go.

Um, you know, do it that way.

We use a few uh uh a few things on top of that.

So we use um model bakery to kind of create model instances.

Um we obviously use um the sort of the built-in mocking stuff.

Um but yeah, basically just just Django.

You know, the transaction test case and these kind of things.

Yeah, I mean I've been I I ask just because it's great to share with others, and that's something that comes up if we're you know having a beer, I'd I'd ask you.

And also I often in my books, like I have a quite a lot of testing, um, but I haven't yet gone down the pie test coverage, um, you know, everything else rabbit hole.

Um because it is a lot more.

Um it's it's used a lot, it's helpful, but it is whole is you know, what you said with celery, it's like, well, it's this whole other ecosystem we're rolling in.

And um and it is true that you can get, I mean, I can test all the stuff I do for people just with the built-in tools.

It doesn't look like Python code.

But in it somehow, I was thinking about this the other day.

Somehow I kind of like it because you know that you're looking at test code, right?

Rather than looking at business logic code because it's all it's all slightly weird Python with like you know uppercase camel case methods and things like that.

Um so it doesn't bother me weirdly.

I could never remember the name of the asserts though.

I'm always like uh self assert, what am I asserting now?

You know, it's it's just it's a big library, isn't it?

So that there's a lot of surface area to memorize.

Like what are do you do pair programming?

Do you have internal stuff?

Do you have other resources you like to use?

Because you know, you you bring in more junior people.

Um, what does that process look like?

Um the starting point for people who've never really done any Django before is is always the Django docs.

I think that the the tutorial in there is is good enough, um, and that's a good place for people to start.

We have an internal uh company handbook, which we call the Dab Apps Docs, which sort of builds on top of that and says, you know, given that you understand basically how Django works, this is how we do Django.

Um so it just has Dab Apps specific stuff and it has a load of other things in there around how we run the company and um uh you know, front-end stuff and testing stuff and operational stuff and and all that kind of thing.

Um and then yeah, pair programming, uh mentoring.

Um it's it's been quite difficult onboarding people uh in the past couple of years with everyone working remotely because it's so it's so great to just have a new person and an experienced person just sitting next to each other at a desk and you know shouting questions at each other.

But we've we've adapted and we we try and have people on on Zoom calls as much as possible.

And now we are able to go back into the office to some extent.

We try and encourage new starters to go in at least a bit of the time and um and sort of you know work with the person who's trying to to onboard them.

I mean, I think on some level that's why all these CS grads can go into these large organizations and it's not that the organization itself has great training, I would say.

It's just that there are experienced people and there's sort of I don't know there's enough slack that they can spend the time to to level you up because if you sit next to someone who knows what they're doing, you know, in three months they'll get you what would take a year on your own.

That to me is that's absolutely key.

It's far better to risk appearing stupid.

And of course, there's no such thing as appearing stupid.

Like people who've been programming for their entire lives ask stupid questions.

It's just how programming works.

Um, but if you can get over that that that fear of of asking a silly question, it's so much more efficient than going away for a week and trying to figure out something for yourself and coming back with probably the wrong answer, but certainly an answer that someone solved before a hundred times, and you could have fixed it in 10 minutes if you just asked.

And I think code review is really important as well.

I think we use GitHub and and we use pull requests and and we make sure that everything gets reviewed before it can be released to clients, and and that's definitely not a sort of command and control hierarchical thing.

It's not more senior people reviewing more junior people's code, it's also junior people reviewing senior people's code.

And that allows the juniors to see how seniors think uh and to read code written by someone with more experience than them.

And I think that's that's really helpful as well.

Because I think as a beginner, you know, a beginner needs to put in some degree of work, but you know, don't spend a week on something that you know, maybe it's an N plus one issue or maybe it's just you know a simple config thing.

Um and I mean I still do that.

I just just yesterday I I had a question around forms and views, and I did enough research that I was like, okay, I think I kind of know, but I still want to know what the best practice is.

So I asked Carlton, asked some other people, and you know, um obviously I like I sort of know what what's a good question to ask someone.

Um but I st you to your point, I still ask all the time around certain things, but it's usually more, you know, the what's the best way rather than you know, how do I get this to work?

Because I can get it to work, but yeah.

Do you want to give a plug, Carlton, to rap reviews?

Just because I am throwing it in the 3.2 update to my books, but it's um well

So the question um Will asked yesterday was about like say you've got a view that you want to respond to differently to um I don't know something on the request, like you know, it might be the um content type.

So you want to handle the same the detail view and the and the form view and the same thing, and then you know, you so you can have a you know nice two separate views, one that just uh returns the detail view and one that returns handles the form, and then you can have a a a kind of view that decides which to dispatch to in front of them.

It's kind of a nice pattern because you can do that with you know a list view that you also want to be an RSS view or you, you know, or a JSON response for the for the same thing.

And it's it was a nice, it's a nice pattern.

I was like, no, we'll do include in my you know include that in in the example because I think that's something that's really handy to learn.

Is this this you know, because if you put it all in one big fat view, it gets really complicated.

I'm I have to confess I'm not a huge fan of Django's generic views.

Django's well Django's kind of class based views I guess um to me I there's there's a fantastic uh quite short book um written by Luke Plant called Django I think it's called Django views the right way that's a really really good read and it kind of argues that the way Django views encourages you to compose complex functionality is quite confusing, quite difficult to follow, quite difficult to read and you only can really follow it if

you know the API really well or if you kind of almost have the source code open next to you.

So you're kind of going, okay so that method is called by this thing that's on that base view mix in and then that one calls that one and it's just this huge nest of things calling other things.

And often it's just easier to write the damn code out you know if even if you're repeating yourself it it's just clear and you can just see this happens then that happens then that happens and then that happens done right and I'm I'm a big fan of that.

Like you know that it's good code it's great code it's super tested it's it's super well documented it's got classy class based views for when you you know when you need to use it but that's not maintainable easy to reason about you know software.

So I agree with that but that is that you know to your point yeah you are going to duplicate yourself a little bit but you know it's weighing that versus this big inheritance chain and and part of why you split up class based views is that when you have a detail view and another mix in another mix in you know that gets wonky fast with in subtle ways.

But the super method was one line well from that six months later you is much better off having that one line in front of you even if it was duplicated than having to dig through two or three files to to to see what's actually going on in this method I would argue and Tom's argued and you know Jamie you're nodding so you clearly agree.

Yeah.

I've done it for a long time and I think that it's it's a big overhead and you have to almost hire a a team of people to do it properly, you know, with the proper security patches and keeping things up to date.

So we use GitHub Actions for CI what we do is kind of we we we work in a a pull request workflow so we'll have let's say that we have a project that's been live for a while and we're working on a new package of work that might consist of I don't know three features and two bug fixes.

Each one of those will go into its own pull request against the main branch.

Each one will be reviewed independently we'll then build what we call a release branch which is essentially another pull request which merges all of those together and that's another pull request into the main branch.

That will then get deployed to a staging environment and in order to do our deployments we just use the Heroku web UI you know we we go in and we have a rule where we we pair on deployments so you always have two people making sure that you're clicking the right button on the right project kind of thing.

But you um you know you you you go in you press the button it deploys it to the staging environment you test it internally the client signs it off and then same process for deploying to production click a button and I'm I'm very keen on that approach definitely.

you've you

You know we really don't specialise in any particular type of client we have all sorts of different projects in completely different sectors all going on in parallel at the same time.

So it can be really really interesting.

So if anyone is interested drop me an email jamie at dab apps dot com um even if you don't see a suitable job ad on our website just drop me an email anyway because we're always on the lookout

1.0 yes

okay super well

're uh DjangoChat com and chat django on Twitter join us next time

bye bye thanks Jamie thanks bye