Transcript: Django Tasks - Jake Howard

Hello, welcome to another episode of Django Chat, a podcast on the Django web framework.

Carlton Gibson joined us ever by Will Vincent. Hello, Will.

Yeah, not too bad. Thank you very much.

Well, that's good. Thank you for coming on with us. I guess we should dive right into it. You're super famous now because you're the author of the whole Django Tasks framework, right?

as the kind of months tick on. So for people that have never met me before, professionally,

I'm a senior systems engineer at Torchbox. Torchbox are the people who make the Wagtail

content management system, which by extension is built on Django. Wagtail is used in very,

very many places um nhs mozilla google nasa loads and loads of people using it my house

yeah everyone's using it um on the django side i'm a django software foundation member um but

as of about this time last year i also joined the django security team dealing with all of the

security things so if you've seen a oh god here's a security thing i need to go and do something

it's probably my fault sorry um after that i'm a new parent um i occasionally speak at conferences

and i've been sort of dabbling in the django ecosystem for the best part of a decade but

sort of more prominently over the last kind of two years or so so i've been leaning into it more

heavily exactly yeah it's kind of the last ever since the um the debt 14 stuff which we'll get

into is everything's gone from i'm a person who works with django to like there are other django

developers who know my name which feels really weird yeah because i mean it's not exactly the

so go on which do you want to pick first because you mentioned tasks you mentioned the security

team i want to talk to you about both of those topics so but i'll let you um i mean the docs

these docs we use to guide the discussion so i'll i'll say let's talk about tasks so one of the

public facing things is you gave a talk at jingo con europe in 2024 um i suspect that wasn't the

start of you thinking about tasks and what we can do about things like where did where did it start

this idea of something should change and i should lead it how'd you get sucked into being the leader

The story of where Django Tasks came from, it sort of jumped on the scene, I would say,

sort of early 2024 was when Carlson popped up and said, hey, this thing you've been doing in

Wagtail, let's put it in Django. But I went, okay, having no idea how to contribute to Django

at all. I don't think even at that point I had an account on track, let alone knew the track

existed um so long long ago um a lot of the debt 14 stuff started in wagtail um wagtail being a

content management system it does a lot of things when you like click publish for example one of

the things it might do is go and build a search index it might yell at your cdn to purchase caches

currently or at the time of developing things now 2022 era when you click publish all of that stuff

happened inside the request response life cycle and that works and that's fine but as a site scales

it starts to get really really slow and so we were looking okay we want to shift some things into the

background as any developer knows when you're trying to like add in task things you can google

Django background tasks and you'll come up with basically a thousand different options

and you've got to weigh them. Wagtail had a slightly more unique problem in that Wagtail

is in itself a package, a framework you install. It can't really go, we're going to depend on

Celery and everyone is going to live with that. Because if maybe your dev team are more familiar

with RQ or Q2 or something like that, then you are sort of forcing a developer's hand as to

which framework they want to use. So what we wanted to do was, the original start was we want

to kind of build something inside Wagtail based on something that we like, but being very Wagtail

specific. Early on, we realized that's not really going to work. And we are going to need something

that is more sort of an API layer around existing ones where you can sort of plug and play with them

and go, okay, well, Wagtail just has this shim and it knows how to talk with this API layer.

When you install it in your project, if you are comfortable using Celery, you just point it at

Celery and it just works. If you want to point it at RQ, you change like one or two lines, make sure

you've got a Redis instance and you're good to go. Wagtail itself in the development side of things

doesn't need to care and that's where this sort of sparked on we had some proposals some ideas

and then we just kind of went here's a good idea and then it kind of sat gathering dust a little

bit there was an rfc open for 18 months or so but no one ever had the kind of the time the energy

to sit down and go okay i'm actually going to write this um for about 18 months and then someone

chimed in on mastodon spotted this thing going this is really cool this should not be in just

wagtail um and then this carlton gibson person who i'd heard of in sort of abstract appeared and went

yeah you should do that and i went okay and i messaged some people in work going this carlton

gibson bloke who's really important thinks what i did was really good i'm gonna sink half a day

into this and they went okay and out of that came the sort of the draft draft draft that

eventually became depth 14 jango background tasks well that that's a sign of a good workplace that

work on stuff right i mean because that's the thing is that the best things come out of a real

world need so carlton's neapolitan tasks right like it's not academic projects it's solving

like Django itself, a real world problem. And I remember that from the other side,

oh, wow, this is really cool. Because it's not just that if you go to Google and type in Django

background task, you can get, you know, a thousand options. It's that we would discuss this sort of

every six months on, you know, whatever the social media of the day was. And there'd be a different

option for every single Django developer. Now that's kind of fine. But if you're writing a

third-party package like wagtail or any other one it doesn't really give you much of a target for

do you know what we want to enable background tasks in a um an agnostic way and what i really

liked was that it was like hey and what i really like that it's it's maintained is it was going to

be a shim above the different concrete implementations yeah that's that's where that

is this is going to quote unquote change the Django world, as it were, that you've got this

single unified contract where particularly for package maintainers, they can go, I can now put

stuff in the background and it just works. You look at other ecosystems in like your JavaScript

Go, things like that. A lot of that stuff is built into the language. Python, you can kind of do that

if you are like fully native async, you can trivially kick stuff up into the background,

but you don't get the same kind of guarantees that you get from something with a proper queue store

and multiple workers and things like that putting this kind of common thing sat there just handles

everything nicely and as a developer you don't need to relearn an api if you're pivoting between

projects if you move company and go from using rq to celery or vice versa you don't need to

relearn all the different apis you go at task and it kind of does the rest you change it yeah and

as at the time um i think they just announced the solid queue um implementation for rails and it was

another one of these oh why haven't we got you know what rails has got moments we're doomed and

you know your proposal came along nicely at the same time and so actually we you know there's a

there is a community idea in play already that if we just advanced would um work but there was a

couple there was a big gap between that first conversation and then the final thing it well

you you go this is great django sort of quote unquote moves really slowly it's solid it's stable

but what that means is that when you're trying to develop it it means you have to put in quite

a lot of work and quite a lot of pushing to make anything happen and like i say with it's all well

and good like going i found a bug raise a track ticket progress things through this is i want to

give i want to add i think the final pr ended up being like two and a half thousand lines right

okay i can't really raise a track ticking go i want to give you two and a half thousand lines

that you're going to maintain well you can't but it's not going to get natalia sarah and jacob will

shoot it down very quickly rightly so to clarify like things like this need thought they need a

kind of an actual review dog i know carlson in particular you've sort of been trying to think

quite a lot on where the line sits between a track ticket the new new features repo a dep

maybe we need like some kind of full thing and it there is no like definitive line well if i'm

we really need but but i don't think tasks need an easier depth it's a very full um you know

full featured major scale proposal it's got its own um you know it's jango.task it's got its own

top part of the namespace right it's it's it's a feature of that kind of size and importance

you know the formal debt process was no problem for that it's where the formal debt process is

needed for something which is a bit more than just a track ticket but the impedance of that full debt

the original i suspect if i looked back at like that initial commit for debt 14 what we have now

is massively different to it yeah and there's a good reason for that what i had then was terrible

because i was the only person who really looked at it having some insights and people who have

either worked with background tasks or have been maintainers of other libraries and have used it

people have been burnt by other things and go okay this library does it in this way it's not great

let's try like let's bring it back to django and do it right so just from that review process and

perhaps you know not saying that other people weren't but you know perhaps a few people stick

out in your mind as oh that was really helpful this they you know said this and that yeah well

people who are involved in the all manner of parts of things whether it's sort of you carlton being

the shepherd on the dep pushing things forwards i've got um the actual kind of one of the people

who sits very much in the background

but was actually really, really pivotal

in this progressing through

was my colleague, Tom Usher.

He was the one that looked at the Wagtail proposal

and went, we can't just implement

a new background task thing.

We need a generic API.

And quite frankly, without that one piece of insight,

none of this would happen.

What Wagtail would have done is gone,

we're going to make our own thing.

We're going to make you use this.

And it would have been fine for Wagtail's use case,

But it doesn't progress the ecosystem quite as much.

And I think the important thing is not just to add another one that was the same, you know, another option, RQ, Celery, and now Jake's one.

It was that you were offering the interface above to abstract against them that's the key change, right?

like i don't want to be like the bdfl for celery mark 2 kind of thing like the last thing you need

it's it's the last thing i need like it's so in researching for kind of the ways we can implement

dev 14 background tasks working out what the different api layers should be what the interfaces

are you come across a thousand different people that have gone i've written a background queuing

thing. And what they've got works perfectly for their use case. And that's great. One of the nice

things is if you've written it, it's easy to maintain. It's easy to bug fix. You can go and

work with it. That starts falling apart when you need to build something on the scale of Django

itself. The number of edge cases that you have to account for just sort of escalates massively when

you're working at a app scale but also at django scale there's extra things like okay one of the

big ones that really really stumbled me up was there are lots of these handy things that work

really really well on linux i need django to work 100 on windows as well which means all these nice

little handy things are like oh yeah it's great you just add this one line and my answer is yes

but what about windows and it just falls apart then there are so many things that just don't

work nicely on windows they might someone who knows the win 32 apis can probably come in point

me in the right direction and we're good but i'm not a windows developer there are i think there

is one windows machine in this house and that's it no i mean i remember when i started as a fellow

so the DSF providers would want so that we could at least assure

that the test suite was working on Windows and all the rest of it.

particularly on speaking of the differences in Windows,

is one of the features that isn't in background tasks yet,

but will be, is adding the ability to time out a task.

if a task takes more than 30 seconds kill it implementing that if you look up how to kill

python code lots of people have gone oh just do this just do that but i need solutions that work

on windows on linux that work reliably that are maintainable that you can depend on and it's

really really hard it got so hard i actually ended up doing a conference talk on this at

pycon uk back in september about how to kill python code and all of these different weird

apis and the this works for 90 of the cases apart from these 10 and you can kind of overlap the

venn diagrams and find two that work nicely but it's still a real pain yeah okay perfect is that

It will be a, so how would I solve this?

And I will dive for like a week into dealing with how does this work?

Exactly.

It's a dilettante's curse, right?

You know?

It's a really like, it's a really fun way of doing things.

It's just that I will just dive deep.

I'll learn a thing.

And after a week, I will probably never touch that knowledge again.

But for that week, I'm having a lot of fun.

something comes up and you're like oh connection exactly yeah that's that's my hope is sort of

i should probably get rid of writing some of these notes down but that's well this is the

i'll i'll sometimes write it like yeah back back when people read tutorials i'd type some forget

something type it in and be like oh wow me three years ago was like really like really knew

something you know like what i've been doing since then so i want to ask you about um because

the lack of a back-ending core and the messaging around that and because we know we've just talked

about the why but you know you know from the horse's mouth so speak what's just how would

released django tasks upon the world as it was um the initial django dash tasks release was a

API package that contained three things, let's call it. The first one is the kind of the API

contract, the bits of scaffolding that are needed, the task class, the task result class,

those bits and the bits that deal with switching out backends. That's the stuff that was sort of

required the most amount of time in making sure we got it right. The second thing it contained

was the development backends as i called them the the dummy backend and the immediate backend

both of which are really really useful for local development unit testing things like that the

dummy backend doesn't run anything it just stores the tasks in the list so you can interrogate them

in your unit tests the immediate one still behaves like a backend but it runs your functions

immediately rather than spinning them off in the backend so it means that if you're using a package

that um it means if you're using a package that isn't kind of fully tasks aware or you don't want

to go through the effort of deploying things your code will just still work but be a bit slower

rather than these really important functions never getting run and those are the design decisions

around those those two things the the basic api contract and the development things those were

from day one intended to go into django in the kind of initial push that sort of two and a half

thousands of lines of code.

That's what we wanted to get in.

The third thing that Django dash tasks contained was a database backend.

This was, this required a huge amount of work because as

i say building task systems is really complicated and that's when i had to build one sort of from

scratch on my own that's where the kind of confusion came because someone looked at django

tasks and went but it's an api contract and it's some dev backends and it's this production back

end but i already have celery that does all those things why do i care and so i ended up shooting

myself in the foot there in the initial ones just trying to sort of get things out it was the right

thing at the time but as time went on particularly once the code landed in django corn was released

in 6.0 people got really confused they looked at django dash tasks and went what's this why is some

of it not in django why is some of it there how does this all work and the other part was why

useful and everyone's like but it's missing this thing i don't care and so what i ended up doing

was sort of in what i should have done probably about a year or so prior was i wrote down my

thoughts and went how can this actually work how should this work and then i shot an email to the

steering council and went here's what i think we should do am i insane and conveniently and

probably for the first time in my life everyone went no jake you're not insane we've been thinking

that too we've been thinking that too exactly and so that kind of that shone this light on okay

let's sink some time let's make this much much clearer and so particularly now as of recording

but also probably for the last two months or so things are a lot more sensible so to give sort of

the outline if you are going what on earth is this django tasks there are realistically three

things you need to care about. Django.tasks exists in Django 6.0 forward. It contains two things.

It contains the API contracts built into Django itself, and it contains the development backends.

Those are there. Those are ready to use. If you're building packages that target only 6.0 forwards,

use Django.tasks and pretend nothing else exists. That is the dream. That's what we wanted,

and that's what we've achieved. The second thing we have is Django dash tasks. Django dash tasks

is for people that want all those nice things that are in Django 6.0, but can't upgrade yet

for whatever reasons. Maybe their company policy is sticking to LTSs. Maybe they're just on a

really, really big code base and it takes a huge amount of time to move. It doesn't matter the

reasons. I want to support those people and get them using some shiny new things. If you're one

of those people, you can pip install Django-tasks, change a couple of your imports to make sure that

you're pulling from Django-tasks, not Django.tasks, and you're good to go. The third thing is, as I

say, there is that database backend. That is now separate from Django-tasks. You pip install it

separately. It's Django-tasks-db. You can pip install that. And the nice thing is, is currently

it still has a dependency on Django dash tasks, just because I haven't written the bit of code

that swaps between Django and the shim layer. Eventually, that package will have no dependency

on Django dash tasks at all. If you're on Django 6.0, it'll pull the things it needs from Django

dot tasks, and you can keep using it in Django 6.0. If you're on, for example, 5.2, you pip install

Django-tasks, Django-tasksdb, configure your settings, write your tasks, and you're good to go.

100% of the code written by me, because I can change everything I want to work exactly how I

want. But the whole point of this, without going power mad, is I need to be able to support third

party backends, things that are not written majority by me. And so because RQ is the tool

I'm most familiar with outside of the stuff I've written, I thought, okay, I'm going to write an

API contract against RQ. And so what also exists is Django dash tasks dash RQ. It fits in exactly

the same place in the model I just talked about as the database backend. But it means if you want

to use Redis and RQ.

You just change the one line in your backend config

that says, rather than database, use RQ.

None of your, you change that one line

and nothing else in your code needs to change.

You'll go from using the database to using RQ

and you're done.

At some point, and sort of this is the kind of

the next step is the, I've been working on this API contract

which shims between Django tasks and RQ.

One of those things already exists.

It's called Django RQ.

It's the shim between Django and RQ.

So I've got an issue open against the Django RQ repo

to try and get Django tasks RQ upstreamed into Django RQ.

And eventually that means Django tasks RQ

can completely go away.

And that's the dream.

It means if you want to use RQ,

you install django-rq and you do the things you don't need to care about i need this for the

caching bits of um sorry i need this for the rq bit but because i want to use django.task i need

this extra package everything is just there yeah and that's the dream right it's because the django

can have their their back end and for each option then they're just implementing against the back

end rather than you having to swap your development code for each for the particular queue that you've

i don't want to run an entire redis cluster in rq i want a database so i use the database one

it just works then when you scale you change an environment variable that says actually this is

like you now give it the various credentials for connecting to redis and it just works your

your business logic doesn't change okay super this episode is brought to you by button down

email marketing software services out there and they all pretty much offer the same thing

collect and clean addresses send out broadcasts or drip campaigns get analytics so you can see

what's resonating and what's not. Button Down is designed to hook into the tools that you already

care about. Everything from static site generators like Jekyll or Hugo to payment platforms like

Stripe and Memberful. You can hook your site up to Button Down with just a form element or a simple

REST core. Write emails in Markdown and then get on with the actual work you're supposed to do.

New customers can save 50% off their first year with Button Down using the coupon code Jango

and if you email support they'll white glove migrate your existing subscribers and archives

for free again i've just got one more then question what are the bits of the django tasks

interface that you think oh it would be nice if we could grow this or it would be nice if we could

grow that or actually i want to keep this exactly at the scope it is because an abstraction layer

you know people always say of the orm well it's the lowest common denominator because it doesn't

do all the advanced features that you know you might need your oracle can do that or your

postgres can do that well it's going to be the same here there'll be features from individual

for like the simple things you need to do if you need to run a task in the background that's fine

it'll do that the things it doesn't do which as you grow as you start needing to depend on your

tasks become really important aren't there yet and i do want to stress yet this is not a i've

written Django tasks I'm going to disappear off into the sunset like this is an ongoing effort

some of the things that don't exist that would be really nice to exist at some point is going to be

things like retries so you can say okay if this task failed for whatever reason wait a bit try

again things like that timeouts as I've already mentioned making sure that okay if this takes too

long kill it start again maybe we want to do some other really complicated things maybe I want to

i've got a lot of work to do i want to run five different tasks and once all five are finished i

need to run one final one to do a bit of cleanup that is possible to do in fancy things like

celery it's really hard to do in other languages and frameworks that don't have this as a native

concept you can shoehorn around it by sort of spinning up other tasks and it it doesn't feel

This is, I'm sort of channeling my inner,

like basically going full feature creep

of what could this thing do?

Because it's fun to work out

how could these API looks?

What could they do?

What are the edge cases?

That's the engineering mindset

is what can we make it do?

At some point, someone needs to kind of,

basically someone needs to be the bad guy

and go, no, you can't have that.

I think, unfortunately, that bad guy hat meets the steering council.

They have to be the bad guy in some of this.

They at least need to be going, or at least saying, we can't do it yet.

That's the main thing, is it needs more thought.

It's probably not big enough quite yet.

And I think the big one, which is one that the steering council rightly lean on,

is the, we don't know how important this is.

Build it out of core.

Once we know it's really important, then we can include it.

And the flips, the sort of the other point is, well, it's all right building it, but

what about maintaining it?

You know, a fully featured workflow engine to take the whole shebang with dependencies

and retries and branching and loops and those kind of control flows within a task's flow.

Wow, there are whole products that do that.

and are we going to import one of those into django even at the just the api um for it or

you know are we going to have the capacity to to to do that properly um i think it's much better

to have half a product than half a product if that makes sense exactly yeah and so that's that's

there's much more obvious that these are necessary it's very obvious that you need

some way of getting the return value of passing arguments that's fairly obvious do you need

retries does everyone need retries probably not for certain cases if you're building sort of a

small internal back-end tool or a pet project you might be all right without it it would be good if

you had it and if it was like one argument away fantastic but let's wait until we know the demand

operate at scale that they see failures on a regular basis the failures only come up when

you're running things at scale and then the uncommon things become common just because

you're doing it a lot but if you're sending email back as emails out say as a you know classic

example you could be sending you know thousands an hour which is much more than you know almost

all projects are ever going to send and you'll never see a failure so what's the need of retries

to say oh we couldn't we can't have any background tasks because we wouldn't have we couldn't have

retries for the the two percent of projects that are doing things that hit the numbers yeah and

certain things if that two percent is ten percent it's really useful if that two percent are a vocal

2% it might also be useful. At the moment that 2% is a hypothetical 2% and at that point 2%

it. Like I do need to work out what the contribution process looks like for how do we design retries

It's how do we design these more complicated bits that I haven't sat down and sort of discussed with the steering council and worked out where that line sits.

But everything else, like this is where open source wins.

If you have a weird bug, a weird edge case, open an issue, submit a PR.

Let's make these things better.

Yeah.

I firmly believe that three or four goes in the ecosystem shows the right way forward.

Because, you know, people have go one, go two, go three, go four.

And then it becomes clear what the right route is, whereas you could never have seen that without the four attempts.

And it might take a while for those four attempts to play out.

But the end result is something that's lasting and correct.

Yeah.

How does that fit your kind of philosophy?

And there is kind of, we do need that kind of falling on the face of it going, okay, that doesn't work.

Let's try again.

That's why I kind of put the initial Depth14 proposal out there.

It got sort of, I think at the time it had sort of 150 comments on one PR.

Like GitHub got really, really slow trying to load that PR.

And that's what I wanted.

I want that.

It's all well and good building something based on either my experiences or what I think

other people's experiences are.

I can't know what other people's experiences actually are unless they tell me.

So I make a proposal.

People shoot it down.

cover about tasks and jack because before we move on to a different topic not not massively i do

and it takes i end up saying the same thing again and they go okay that makes sense

the with django tasks you create a task you create a function and when you call it you pass

arguments into it with django tasks dot tasks dash tasks whatever those arguments must be json

serializable there is absolutely no pickling in any part of django tasks and it's very very much

by design i'm sure we'll get into in a second sort of my security team background on things but

it was very much chosen on a let's do things right if i can shoehorn people down the don't

touch pickle jason is fine then it solves a huge number of problems anyone who's done weird things

and sort of like gone oh i'll just pass this request session object into my task it will work

fine on say the immediate backend because it's the same process but as soon as it kicks off into

like it could be on a separate machine it could be halfway around the world that like tcp session

that is being represented by the request.session doesn't exist anymore because it's on a machine

a thousand miles away as the stuff just falls over whereas again as you say carlton you want

of implement things based on the lowest common denominator you can do basically everything you

need to do with json the big one that comes up is um passing models between tasks yeah it's a

really really obvious use case but you can't json serialize a model what you can json serialize is

a primary key yeah how to look up so if you exactly just to look up so all you do is you

pass that through and then in your task you just retrieve it again that's got a bunch of nice

benefits it means you're not passing around stale data where things might change between in queuing

and running it means your task um the data that's stored in the task queue is also really really

small it's a number or a uuid versus an entire fat model with all of its columns it means you

haven't got weirdness of maybe when you loaded that model you deferred some of the fields

maybe you've only got one of these keys but the task assumes another one exists so you've got a

bunch of weird performance issues those kind of things you still you don't have to think about

when you're just passing ids around or you're just passing these like primitive data types

it just works yeah and so for the kind of it seems to be a very vocal minority that are like

why is this not pickle there are lots of reasons it's not pickle i've just talked about all of them

and so by forcing you to think about things in json you avoid a bunch of weirdness for

a slight little bit of oh i need to like grab its primary key and do a database query to pick it up

doing that little bit of work for avoiding a bunch of extra issues felt worth it to me it felt worth

it to everyone else who reviewed the adep and i'm pretty happy with that stance yeah no it's

or async code

like they're passing

trying to pass a model

back from a coroutine

it's hang on a second

but that could be

crossing a thread boundary

because it's crossing

a sync to async

boundary

it's like you can't

pass models across threads

this is not

you know

just don't do that

if you need a model

in a particular place

look it up there and then

so I think that's

and maybe the interesting

point here is

how Wagtail

and Django

overlap there

because I think we've

talked a little bit

about the security team

but

maybe you could shed some more light on that right like do things sometimes come in

to django via wagtail vice versa how does how does that play out so obviously the the poster

wagtail is quite a large project that has quite a large number of installations

and so it gets sort of it exercises a lot of edge cases that a standard django project might not

you're doing a lot of things in wagtail the content management system pokes a lot of internal things

in the django internals because it's trying to do some of those things some of that works well

sometimes it doesn't work well and so there are sort of i suspect that a lot of the um the core

wagtail team have raised a lot of issues upstream towards django and even to other projects going

this weird thing happens

is it a bug

is it a feature

and so that kind of

that relationship works really well um we have also had sort of the relationship work the other

way whereby wagtail goes we need this sort of thing to work a little bit nicer maybe we put it

up less as a like we've hit a bug more a let's change this so it works a little bit better

and then it can cascade downstream it's unfortunate that both projects have quite

long release cycles it's not like we open a ticket and a week later it's available on

ipi ready to go like it's going to take kind of three six months sometimes to actually be able to

use the thing but by contributing upstream they can come back down it's useful to us and hopefully

few years and i've uh took me a while to get a crowd cost accustomed to the the wagtail release

cadence it's all good it's brilliant but uh it wasn't it was because um wagtail will use

dependencies that are then pinned because the incompatibilities and whatnot and i'm busy trying

to update my dependencies and um finding conflicts with what wagtail had pinned versus what i wanted

discussions going can you please unpin the upper bound on this dependency please i think we're at

kind of come to terms with it i think it's only um what's the model tree library that you use that

uh um uh tree beard yeah tree beard that you've got that pin so that's um but that's fine i can

cope with it yeah so the security team jake you joined you joined there a year or so ago you said

sort of very very early in my professional career two things have stuck out as being particularly

interesting and that's security and performance and so naturally django has a security team that

kind of do quite a lot of really cool things around maintaining the security of quite a

important um software package there's no kind of it's not obvious how to get involved in that team

how to join that team and quite frankly it still isn't um that is an ongoing problem people are

thinking about how we can deal with that there are some things happening internally to try and

make that a bit clearer um i just happened to basically almost be in the right place at the

right time um at the time of this um tubo colas the then president of the jenga software foundation

had mentioned that the security team was looking for kind of some more people and i went i can do

some of these things at the time i was on the security team for wagtail all of our kind of

anything internal, Torchbox, security related, generally passes my desk. And so I was like,

I have some experience, like, am I helpful? And at some point, I assumed he went away,

talked to some people. And I got a message in Discord sort of three or four weeks later going,

would you like to join the security team? And I kind of, I had to kind of double take

that message a bit because I was like, you want me? Because it is this kind of,

And it's really weird. Like Django is used by so many different organizations, so many different individuals. And there's a pool of, what, eight, 10 people in charge of making sure it doesn't have massive security holes. And I am now one of those people is a really, really weird and humbling feeling is that, like, maybe I am kind of good at what I do.

like there is a list on the django website that has security team and there is my name next to it

and i can point to that and go i am making this thing better and that feels pretty damn great

team does like how does it how does it sort of function

it's a mystery right it's what it's the secret it's this secret cabal that no one knows anything

I mean, conveniently, none of us are kind of the hoodie up, sitting with bright light

screen, just tapping away at stuff.

We are human beings.

I have met the majority of the other people on the team.

We are human beings.

We're not like supercomputers sitting in the background.

The start to finish on how things work is if you are sitting for so and so reason, listening

to my voice at the moment going i have a security hole in wagtail but i don't know what to do about

it the thing to do is at the time of recording anyway is email security at jango project.com

that goes to all the members of the security team we'll triage it we'll talk to you we'll work out

is this a bug how severe is it can we reproduce it is it like are you holding it wrong things like

that all these things that are really important the security team gets a lot of emails most of

them are spam unfortunately right um i would say on sort of a volume maybe a third of them are

like people legitimately trying to help what that means is that two-thirds of the time when i open

an email it's garbage then you have to read every one right and i have to this is the thing i think

people forget is if you're looking at an email from a nigerian prince for example it's pretty

obvious i have not actually won a million pounds by having done nothing if someone goes i have

found a bug in django here's a bunch of code here's some bits of stack trace here's why it's

a thing it can take a lot of brain power and a lot of time to go no this is nothing we get a lot

of things varying from you're calling apis that don't exist or you're calling weird bits of

internal stuff that is not intended to be used in this way or is not publicly accessible or even a

like we've had people report that for example when you call cursor.execute and you pass it

arbitrary user input you get sql injection and the response is yes if you take a user's input

and pass it to the database that is sql injection that is kind of you have to sanitize your user

input and so i'd say sort of nine months or so ago um a bunch of the security and got together

and we sort of added some extra things into our security policies to try and make some of those

kind of you're holding it wrong more obvious so that someone can kind of self-police certain

things so for example you need to make sure that if you're passing random stuff into the database

you should sanitize it in most cases the rm will do that for you if you're passing it into raw sql

stuff that's on you you need to be dealing with that things like making sure that some of the

stuff you're passing through the django template language if you're passing the entire works of

shakespeare and complain that it's a little bit slow we can't do much about that we're limited by

certain things we can make things faster and we're always up for this is a bit slow i want to make it

faster absolutely let's do that but does it constitute a security issue probably not yeah

and finding that line again unfortunately there is no oh if it takes more than like this amount

per character or something it's a security problem it it doesn't work that way it's a feeling it's a

is this too slow could this be a problem if it's a yes we'll triage things through ideally the person

who's reported it has gotten if you change things like this it gets faster or better great we love

people who actually submit um full kind of proof of concepts or patches things like that it makes

our lives so much better and so much easier but some people don't and we also don't want to put

that as a requirement something we'd want to know about any vulnerability even if you've gone

i don't really understand the internal bits we get reports about some of the weird internals of

the rm i have never dug into the weird internals of the rm so i go i can reproduce this but i have

no idea why it's a problem and there's someone else on the security team that does know the

internals of the rm they can look at it they know that side of it i can help by going this is

legitimate and sort of shield them from the garbage as it were yeah and then they can dig

the the effect that that's had on the process and the team this is can i say this is a new

kind of don't really realize about the reports like this is that ai isn't magically creating

more reports it's not like there are bots out there just submitting stuff one of the big things

is that people will sort of go i think i have found something dump it into an llm most llms are

intently designed to make you feel good about yourself so they'll go so if you ask it is this

a vulnerability they'll go yes you're absolutely right it's definitely a thing well done you

that bit of like the confidence boost and we're all for that but then what happens is the llm

will naturally just sort of vomit out a large amount of text and so what goes from a like if

you call this function with this input it's really slow comes out at about a thousand words and

they'll sort of talk about things like oh well it affects these versions of django and if it's

really slow it could have these knock-on effects and really at a certain stage we sort of stop

caring about all of those details. The things we need to know is if you do this, this happens.

That's the important bit. If you want to go into some more bits of details and go, okay,

it's really slow because it's in this part of Django, this has this knock-on effect that makes

this bit particularly slow. All really, really helpful bits of context. But that can be a

sentence or two. That doesn't need to be a paragraph of stuff. And that's the bit that

kind of really, really slows things down.

It goes from, I can

fit this in like half a screen

and know this is all the

context I need, versus I need to kind of

sit and read and

think and digest, because it's the same

amount of information

in about 10 times the amount of text.

And unfortunately,

that's not something that is just

plaguing the security team.

I know in the last

week or two, Natalia

in particular has been sort of

getting really really frustrated at the low quality pull requests that have been coming

through to django repo where they are kind of a lot of noise for something that is either

not actually an issue or it's someone going this could be better without any justification

and that gets to be a real problem um security kind of it's not unique to just us it's not unique

just security but it just pads a huge amount of time and effort into reviewing every single report

of days at reading because the django security team it's shrouded in mystery but it's also

one of the oldest teams that exists in the kind of the django foundation i guess is it runs really

really old almost everything is done by email um and so i've not looked at my emails for probably

almost a week or so i've got 16 unread emails in that inbox right most of them are probably

the internal team going this is garbage but that means that some of them are the initial report

giving us garbage and so i need to then sit down and go through 16 emails worth of following a

discussion working out is this garbage is this not for some cases i can just go they've concluded

if it's garbage, it's garbage. Occasionally, and this has happened a couple of times, is we'll

look at something and go, actually, I don't think this is garbage. This is, there could be something

here. Nine out of 10, the person who reported it hasn't seen it that way. They have submitted

garbage. But if you look at it with a slightly different angle, you go, okay, well, if they've

done this, it's actually this other thing. Unfortunately, because of some of the specifics

of things i can't talk about the like oh there's this api that if you call it in this way it's a

massive problem um and also my memory sucks and so i can't remember most of that detail

but things like this happen there's a lot of nuance around security of massive frameworks

like this because it's django the code base itself is gigantic and the security team kind

of has remit over literally all of the code but just some of the kind of the symptoms of things

if it's slow that's a problem if it's too slow it's a security problem yeah yeah and just one

some of the recent work in the security team because there was you know a number of issues

that have been patched over the last month last few months and some of those were sql injections

and they were you know perhaps um question like they were you have to go very very wrong to do

this but the by the they were legitimate sql injections and by the the standards there um

an sql injection is um category high it's a you know it's a serious severity high issue

and it's it's kind of a it's probably it's like well we've got a release coming up

it is a high issue and then people put on a team to to do it and then oh but we would never just

pass this you know these keyword args to filter in this way and it's it's kind of difficult how

do how do we balance that when there's so many reports coming in and we're trying to do the best

gang in thailand there was the release around the kind of hit a large number of web frameworks

around the like uh email issues with like the dotless i in certain things yes really really

famous hijack potential the account hijack one exactly it was a problem around password reset

where emails that as a human aren't the same the database would look at and go these are the same

and you can take over accounts it's been fixed for like eight years now it's not a problem but

what happened is that affects basically everyone it had a because it was very well known the time

between there is going to be a security release and it being released upon the world was expedited

it was one day rather than the standard seven and so naturally i getting that email going there's a

release tomorrow lost my mind because i was like what the hell is going on about a month a month

and a half later there was another email that came around it was again it was a security release it

was marked high but it had the standard seven day embargo i was like oh god is this going to be

another kind of i need to spend a day patching all of my projects somewhat conveniently i happened

to be out of the office on the day that um that it was due to be released so i was there i was

sort of sitting um in the passenger seat of the car panicking checking my emails going okay is

this a big thing am i gonna need to like triage some things in slack and it was it was a small

sql injection specifically on oracle right which if you're using oracle is a big deal

we weren't using oracle we were using postgres and so to us it was not even a low it was a complete

but if you see an email saying there is a security vulnerability it's rated high

you just see that you have to react based on that and i don't actually think there's an easy way of

fixing that you can't give more you can't give too much information because someone will try and

reverse engineer it or work out what it is or try and kind of beat that embargo we also can't do um

we can't sort of sneak little bits of information to certain people for example django has got i

forget the formal name of it but basically if you are a big enough important organization

you can ask us very very nicely and we will tell you the specific details about security release

when we send that early warning rather than when we sort of do the final release and what that

means is that for for certain companies where it's really really important they get patched

as soon as that information is out they need a little bit of warning and we want to accommodate

that but we can't give that to everyone at the time i was working at a small django agency

our projects are not big enough and important enough to need that kind of information but where

does that line sit yeah who is big enough and important enough is it based on the size of the

team the number of developers is it how important the project is that's generally the way we've gone

distributions redistributing django right so if you've they need time to be able to get it into

to patch their version of Django

because they're not necessarily shipping Django 6.0.

They're shipping 5.2 point with all the patches

and they need time to be able to update their repository.

So when you apt-get or apt-update or unattended updates,

it automatically pulls in the Django update for you.

So we have security contacts at Debian, at Red Hat.

They get the emails.

there are some organizations that get a similar thing.

It's really, really ad hoc.

We kind of, we don't have rules for it.

There are some listed in the docs, but they're not perfect.

It's very much a, if you ask, we'll tell you.

last three weeks or so we had two companies come to us who i will not include their names

one we said yes one we said no and it just comes up that way sometimes yeah but to come to sort of

then even a high a high severity issue is a non-issue right exactly yeah and there is there

release um configuration basically that encodes as part of a security release like i'm doing a

release the issue is in this function if you are calling it in this way and it's programmatically

defined so it means you can run like basically static analysis on your project and know not only

am i using like the standard one am i using dependencies that have vulnerabilities in

i can also know am i vulnerable to it and that would be massively helpful in an ecosystem like

python but the reason you get some of that in go is it's got static types python does not have

quite so static types let's call it or indeed yes any we have other things in our notes around

one hour time limit and so that leaves times for books and projects the important things

do you i believe you have one queued up jake is there a book you want to recommend yeah i've i

a massive reader sort of from, I would say in the last 15 years, I have probably read two fiction

books at all. And I don't think any nonfiction books from start to end. I've tried to trick

myself into reading like if I buy myself a Kindle, I'm kind of playing with some tech,

but also reading. It worked for a couple of months and then I got bored. I'm trying to pick

that up again. And one of the things that's helped is conveniently at the same time, there's sort of

been a tech reading group that's formed at work of people that are kind of interested in reading

and learning stuff and so the the first book and the book we're reading at the moment is um designing

data intensive applications and so conveniently that's also very well aligned to kind of

the systems programming performance background tasks and stuff that are kind of interesting to

me at the moment and so i've quite enjoyed i'm a couple of chapters behind the the rest of the

group but kind of slowly catching up being involved just talking about some of this stuff

had a group with some people from the janga community reading um rust in action back a few

years ago and it was really good we'd meet up i think it might have been been during the pandemic

time so it was nice to have some sort of social interaction as well as you know how we're going

to get through this book on rust carlton you have one uh oh i do i do well i've been um i picked it

back up off the shelf um solar power finance without the jargon by jenny chase um jenny

chase is an analyst that works at bloomberg i'm back with oil over a hundred dollars a barrel i

thought i'd pick it up it's it says it's about solar power finance it's sort of more really

about solar power there's a little bit about finance and prices and supply and demand and

But it's a really interesting book about the economics of solar power and then how renewable energy has grown and will continue to grow, especially when alternatives are inflated due to geopolitical events.

So I'm reading that.

Solar power for finance without the jargon.

There was just some study showing that solar panels can, they thought they had a lifespan that was quite short, but in fact, almost indefinitely for decades, they can continue to pump out.

And then the second five years, it turns out that those estimates were a lot shorter, exactly as you say.

I mean, I'll just say one interesting thing as an American with what's happening.

You know, the U.S., we're the largest producer of oil and gas.

And because of that, we can't make our mind up about what we want.

You know, so China, China can't produce enough oil.

So it's forced to invest more in alternatives, even though it seems fine to deal with polluting things like it just knows it has to.

The U.S. doesn't really have to, and so we're much more prey to, do we want high prices? Do we want low prices? There's a whole discussion around that, but it's a little more complicated in the U.S., I guess, which is one of the reasons why we're so reticent, at least this year, to invest in alternatives. I'll just say that.

no it's interesting i think yeah okay i'll check that out um for me quickly just fiction

the passage by justin cronin this is actually a reread for me i think it's 10 years old

um and this is catnip for me because he's like a literary author who won a bunch of awards and

then decided to write something that people wanted to read so it's essentially a vampire book

um and it's actually a trilogy it's a literary page turner yeah right it's you know um

ah it's just really good i just really like you know a plot and to me this you know little quick

sketches and portraits and it's uh yeah i need that so highly recommend it um um projects i'll

just keep going quickly so um this is well as we record which is march 10th this will come out a

little bit later but um andrej karpathy just released a gist uh micro gpt.py which is an

entire way to run a GPT in under 200 lines of code. So train, inference, tokenizer, autograd,

initialize parameters, optimizers and buffers, repeat and sequence, and then do inference.

And it's not optimized at all for scale, but it's meant as a teaching tool. And I've been sort of

circling around, how do you actually build these things? I recommended recently, there's a book,

build your own lm from scratch this is really great to just see in one file oh this is you

know academically how it's all happening so i would highly recommend check it out it's commented

few and far between in certain cases i found myself sort of originally sort of when i was

much younger and had a lot more energy i would kind of i'd be having thousands and thousands

of different things and slowly my github is slowly filling with these kind of the things i've built

and i'll go yeah i'll i'll maintain that and then i realize oh it's been like two years since i've

had a commit that's fine that's been not so great um one of the things that's kind of particularly

in the last week or so that's been on my mind is actually one of the things um that carlton

think you've been working on is which i'm going to hand over to you quite nicely is around type

been writing typed python code quite a lot is that by its nature a lot of python and django

is hard to type and so writing a kind of type safe layer around an untyped safe api

is a really really nice thing i do want to kind of

randomly operate a little bit it's a really really nice idea

but i've also seen it in a couple of other places that's okay um not not in python actually

funnily enough it's also a really really common model in rust um so i mean at least you can say

you're copying this thing from rust and pulling it into python which is something okay so i mean

mantle drf which is uh drf um add-on for it so that you can use it with your existing drf project

now but what it is is um so it builds on attas so you talk about a type safe layer so jango doesn't

you've got your Django model you're never going to make that type safe like and there's some new

pep about um I can't remember adaptive typing I can't remember exactly what the the word is

even if that comes in we're not going to be able to retrofit that into the art we're just not like

Django is dynamic to the core it's built on you know Python's dynamic capabilities and

we should just embrace that but for my work code from I want to have like modern Python modern

simple plain typed python classes so i use attas for that i can talk about why some other time

so i use matt attas for that and then the challenge isn't writing the attas class right

anyone can write a data class you know it's got a name and that's a string and it's got an age and

that's an int and you know those classes aren't hard to write the problem is mapping that from

the orm and so what mantle provides and it uses django readers that i've talked about on the show

billion times i love it it uses django readers to create efficient orm queries from your your

outer class and then the those queries will have just the fields that were in it and it works with

nested data doing prefetches related automatically and and then it serializes back so you just do a

query you give it a query set and the shape you want and it gives you back typed objects that

already did all the fetches and then you can pass them to your api or you can pass them to your

templates, or you can add the domain logic that you want, and it's all independent from the ORM.

And so it's a way of decoupling your business logic or your domain logic from your ORM models.

Your ORM models are busy enough already. They're busy dealing with the relationship to the database.

And one of the big problems we have is when we slap on loads and loads and loads of business

logic into our models, and they become this big, difficult deal with monstrosity, essentially.

where it's a way of cutting cleanly

between Django's dynamic core

and then the type-safe layer on top.

Now, the bit that Mantle brings is that flow.

Atta's not unique to me, not original to me.

Kata's, the serialization tool I'm using,

not original to me.

Django Reader's not original to me.

It's a synthesis of how I've been using those tools

over the last few years, basically.

And Mantle DRF is, as I say,

It gives versions of DRF's generic views, its mixins, its view sets,

so that you can just change your imports and use a mantle shape class in your DRF API.

So anyway, go check that out.

It's on my website.

We'll put a link into it.

I'm glad you've been looking at it, Nick.

I haven't really talked about it yet.

which happens to be the one place that I happened to see it,

which was the PyTV conference last week.

Yeah, yeah, I'll put a link.

Exactly, yeah.

And conveniently, the two talks I happened to see

between meetings and other bits was yours and Sarah's,

and they were both fantastic.

And Manto in particular was just that kind of,

it's nice that for a while I found this kind of type safety

on top of un-type safe stuff

as a really good way of thinking about stuff.

you can kind of hide a bunch of weirdness particularly if you need to kind of do a bunch of

weird things to make it type safe you can hide that stuff in a function and you can just use this

public api to make things nice yeah and we're doing we do that for like operating system

abstractions rust does it for safe and unsafe code why can't we do it for safe and unsafe types

i just felt like manto in particular obviously is the django way of doing it but that that thought

process of kind of facading stuff i think is a tool that works for developers in general even

are of use and you don't have to abandon all that power and leave that power behind

and i think that's really important as well so and this was like you remember that thread on

forum about django needs a rest story and i said oh well we need a serialization story well that's

attas and characters but for me but the point i wanted i said was well it's got to be orm aware

and that's the that's what mantle gives you as well is these is the type safety and the modern

serialization story but it's orm aware and that's so anyway i've just i'm putting it out now so i'm

court europe so i'm going to do a long version director's cut talking about you know typing and

mantling a bit more depth than i was able to do in um on the pi tv event and then i'm going to

give a shorter version again at um python italia so if you're anywhere in europe this spring

anything we didn't ask you

or that you want to highlight?

Magic wand.

Oh, yeah.

All right.

I'll hand it to you.

Digitally hand it to you.

Here's your magic wand.

You can't change tasks,

but something else in Django.

Well, you don't need a magic wand.

It would be very weird for me to go,

actually, I want to rewrite all of this.

It's like, it's terrible.

Rewrite it and rest.

Yeah, yeah.

The, yeah, I mean,

you're going to have to explain

the magic wand to me a little bit.

Generally, we don't have guests who've actually changed something in Django.

So it's just this idea of code community.

Just, oh, wouldn't it be nice if we could just change one thing?

That's possibly a spicy one.

It's a fact.

It's allergic to dependencies.

my hot take is i don't think that's good i think i don't know if i go as far as to say it's bad

but i think if we were less allergic to things some things could be made a lot better so for

example if we i'm not even thinking like on the mass of things like oh we should just depend on

jango restaurant what can be done with and we could solve the rest story once and for all

I don't think that actually solves anything, but there are lots of things inside Django where we've

had to not so much reinvent the wheel, but we've had to kind of write some stuff ourselves because

we can't use a third-party library, or we've had to vendor little bits of it to work for what we

need. If we could instead pull in a dependency, that works out great. It was really restrictive

in django tasks because that's meant to be a backport django tasks has no dependencies it

would be really nice if for certain things i could pull in various dependencies for other things but

i know that if as part of shipping django tasks i also shipped five additional dependencies for

django it would be a real problem and i don't i don't really know what the solution is to this

but i think sort of possibly the easier way is making a good argument for this dependency in

particular brings us a bunch of benefits um particularly around like some of the like

utils and the core http bits of django they're not unique to django they are used by basically

every web framework out there why can't we have we need one unifying implementation to bring them

all together um i don't want to start this meme again that basically everything i do is making a

unified implementation but at least if we if there are implementations already out there maybe we can

use them maybe we can remove a bunch of stuff inside django maybe we can shrink the code base

a bit shrink the amount of work that the fellows need to do and use trusted not just pulling in a

random thing that a person has written third-party tools ideally as hard dependencies rather than

sort of the optional ones we've already got that bring in those extra benefits without making our

without giving ourselves a ton of extra work no that's a really good one that's a really good

good code in there and it's trustworthy enough that you use the word trusted that the the issue

with you know the javascript project that brings in 400 dependencies and then they all break and

who can't update right that's the sort of the that's the historic counter push back is why

very few projects have the stability requirements that django would need to you know pull them in

as dependencies um but yeah those kind of core cases i think there's a real good case i like

whispering to me you know like oh there's this thing coming this thing coming so you know um

and it's still coming i think i'm glad i'm glad you had a chance to address some of the questions

and tasks is not going away and in fact will be easier and more powerful to use going forward

if you're on 6.0 use Django tasks get on with nice easy background tasks as time goes on it'll just

get better and better and better it does also it's I really hope it's not just me working on

this for the next 10 years like if you are interested in background tasks API design

the kind of the systems programming that you need to do for building background workers please sort

of appear contribute to things get involved in discussions like i don't want to have to

maintain this entire thing on my own please okay call to action folks yeah exactly all right so

email newsletter.