Transcript: Django Tasks - Jake Howard

Hello, welcome to another episode of Django Chat, podcast on the Django web framework.

Carlton Gibson joined us ever by Will Vincent.

Hello, Will.

Hello, Will.

How are you doing, Jake?

Yeah, not too bad.

Well, that's good, good.

I guess we should dive right into it.

You're super famous now because you're the author of the whole Django Tasks framework, right?

Um, so for people that have never met me before, um, professionally I'm a senior systems engineer at Torchbox.

Um, I Torchbox are the people who make the Wagtail content management system, which by extension is built on Django.

Wagtail's used in very, very many places.

Um, NHS, Mozilla, Google, NASA, loads and loads of people are using it.

Um, use

it here.

Um on the Django side, I'm a Django Software Foundation member.

Um, but as of about this time last year, I also joined the Django security team dealing with all of the security things.

So if you've seen a, oh god, here's a security thing, I need to go and do something, it's probably my fault, sorry.

Um, I'm a new parent.

Um, I occasionally speak at conferences and I've been sort of dabbling in the Django ecosystem for the best part of a decade, but sort of more prominently over the last kind of two years or so.

It's kind of the last ever since the um the debt 14 stuff, which we'll get into, is everything's gone from I'm a person who works with Django to like there are other Django developers who know my name, which feels really weird.

Normally.

Uh which do you want to pick first?

Because you mentioned tasks, you mentioned the security team, and I want to talk to you about both of those topics.

So, but I'll let you um

Should we start with that?

Go on then.

So I'll I'll say let's talk about tasks.

So one of the public-facing things is you gave a talk at DjangoCon Europe in 2024.

Um I suspect that wasn't the start of you thinking about tasks and what we can do about things.

Like where did where did it start, this idea of something should change and I should lead it?

How'd you get sucked into being the leader?

That's kind of lost a time a little bit.

The the story of where Django tasks came from, sort of, it sort of jumped on the scene, I would say, sort of early 2024, was when Carlson popped up and said, Hey, this thing you've been doing in Wagtail, let's make put it in Django.

But I went, uh, okay, having no idea how to contribute to Django at all.

I don't think even at that point I had an account on track, let alone knew the track existed.

Um so long, long ago, um, a lot of the DEP 14 stuff started in Wagtail.

Um, Wagtail being a content management system, it does a lot of things when you like click publish.

For example, one of the things it might do is go and build a search index.

It might yell at your CDN to purchase caches.

Currently, or at the time of um developing things now 2022 era, when you click publish, all of that stuff happened inside the request response lifecycle.

And that works and that's fine.

But as a site scales, it starts to get really, really slow.

And so we were looking, okay, we want to shift some things into the background.

As any developer knows, when you're trying to like add in task things, you can Google Django background tasks and you'll come up with basically a thousand different options and you've got to weigh them.

Wagtail had a slightly more unique problem in that Wagtail is in itself a package, a framework you install.

It can't really go, we're going to depend on celery, and everyone is going to live with that.

Because if your maybe your dev team are more familiar with RQ or Q2 or something like that, then you are sort of forcing a developer's hand as to which framework they want to use.

So what we wanted to do was the original start was we want to kind of build something inside Wagtail based on something that we like, but being very Wagtail specific.

Um early on we realized that's not really going to work and we are going to need something that is more sort of an API layer around existing ones where you can sort of um plug and play with them and go, okay, well, Wagtail just has this shim and it knows how to talk with this API layer.

When you install it in your project, if you are comfortable using Celery, you just point it at Celery and it just works.

If you want to point it at RQ, you change like one or two lines, make sure you've got a Redis instance, and you're good to go.

Wagtail itself on the development side of things doesn't need to care.

And that's where this sort of sparked on.

We had some proposals, some ideas, and then we just kind of went, here's a good idea, and then it kind of sat gathering dust a little bit.

There was an RFC open for 18 months or so, but no one ever had the kind of the time, the energy to sit down and go, okay, I'm actually gonna write this.

Um for about 18 months, and then someone chimed in on Mastodon, spotted this thing, going, This is really cool.

This should not be in just a Wagtail.

Um, and then this Carlton Gibson person who I'd heard of in sort of abstract appeared and went, Yeah, you should do that.

And I went, Okay.

I'm gonna sink half a day into this.

And they went, okay.

And out of that came the sort of the draft, draft, draft that eventually became depth 14 Django background tasks.

But also that, you know, you can you can work on stuff, right?

I mean, because that's the thing, is that the best things come out of a real world need.

So Carlton's Neapolitan tasks, right?

Like it's not academic projects, it's solving, like Django itself, a real world problem.

I remember that someone mentioning it or Mastodon and seeing it and being like, oh wow, this is really cool.

Because like it's not just that you if you go to Google and type in Django background tasks, you can get, you know, a thousand options.

It's that we would discuss this sort of every six months on on you know, whatever the social media of the day was, and there'd be a different option for every single Django developer.

Now that's kind of fine, but if you're writing a third-party package like Wagtail or any other one, it doesn't really give you much of a target for do you know what we want to enable background tasks in a um an agnostic way.

And what I really liked was that it was like, hey, and what I really like that it's it's maintained is it was gonna be a shim above the different concrete in implementations.

That was that's what kind of lit that light bulb of a, this is a big deal.

Like this is this is gonna quote unquote change the Django world, as it were, that you've got this single unified contract where particularly for package maintainers, they can go, I can now put stuff in the background, then it just works.

You look at other ecosystems in like your JavaScript, Go, things like that.

A lot of that stuff is built into the language.

Python, you can kind of do that if you are like fully native async, you can trivially kick stuff up into the background, but you don't get the same kind of guarantees that you get from something with a proper cue store and multiple workers and things like that.

Putting this kind of common thing sat there just handles everything nicely.

And as a developer, you don't need to relearn an API if you're pivoting between projects.

If you move company and go from using RQ to Celery or vice versa, you don't need to relearn all the different APIs, you go at task and it kind of does the rest.

As well, there was one more thing as at the time.

Um, I think they just announced the Solid Q um implementation for Rails.

And it was another one of these, oh, why haven't we got you know what Rails has got moments?

We're doomed.

And it you know, it your proposal came along nicely at the same time.

And so actually, we you know, there's a there is a a community idea in play already that if we just advanced would um work.

But there was a couple of there was a big gap between that first conversation and then the final thing.

And I think the problem is is that for it's something that as a developer you you go, this is great.

Django sort of quote unquote moves really slowly, it's solid, it's stable.

But what that means is that when you're trying to develop it, it means you have to put in quite a lot of work and quite a lot of pushing to make anything happen.

And like I say, with it's all well and good, like going, I found a bug, raise a track ticket, progress things through.

This is I want to give I want to add I think the final PR ended up being like two and a half thousand lines.

Right, okay.

I can't really raise a track ticket and go, I want to give you two and a half thousand lines that you're going to maintain.

Well you

Rightly so to clarify like things like this need thought.

They need a kind of an actual review dog I know Carlton in particular you've sort of been trying to think quite a lot on where the line sits between a track ticket, the new new features repo, a depth maybe we need like some kind of fourth thing and it there is no like definitive line.

Well if I'm

But but I don't think tasks need an easier depth it's a very full you know full featured major scale proposal.

It's got its own um you know it's Django dot tasks it's got its own top part of the namespace right it's it's a it's a feature of that kind of size and importance you know the formal debt process was no problem for that.

It's where the formal debt process is needed for something which is a bit more than just a track ticket but the impedance of that full depth seemed it does seem to be an impedance.

Yeah and there's a good reason for that what I had then was terrible because I was the only person who really looked at it.

Having some insights and people who have either worked with background tasks or have been maintainers of other libraries and have used it people that have been burnt by other things and go, okay, this library does it in this way it's not great.

Let's try like let's bring it back to Django and do it right.

He was the one that looked at the Wagtail proposal and went we can't just implement a new background task thing.

We need a generic API and quite frankly without that one piece of insight none of this would happen.

What Wagtail would have done is gone we're gonna make our own thing we're gonna make you use this and it would have been fine for Wagtail's use case but it doesn't progress the ecosystem quite as much.

ing.

One of the nice things is if you've written it's easy to maintain it's easy to bug fix.

You can go and work with it.

That starts falling apart when you need to build something on the scale of Django itself.

The number of edge cases that you have to account for just sort of escalates massively when you're working at a at scale but also at Django scale there's extra things like okay one of the big ones that really really stumble me up was there are lots of these handy things that work really really well on Linux.

I need Django to work 100% on Windows as well which means all these nice little handy things are like oh yeah it's great you just add this one line and my answer is yes but what about Windows?

And it just falls apart then there are so many things that just don't work nicely on Windows.

They might someone who knows the Win32 APIs can probably come in point me in the right direction and we're good but I'm not a Windows developer.

There are I think there is one Windows machine in this house and that's it.

So if a task takes more than 30 seconds kill it.

Implementing that if you look up how to kill Python code lots of people have gone oh just do this just do that but I need solutions that work on Windows, on Linux, that work reliably, that are maintainable, that you can depend on.

And it's really really hard.

It also hard I actually ended up doing a conference talk on this at PyCon UK back in September about how to kill Python code and all of these different weird APIs and the this works for 90% of the cases apart from these 10% and you can kind of overlap the Venn diagrams and find two that work nicely but it's still a real pain.

Yeah okay perfect.

It will be a so how would I solve this and I will dive for like a week into dealing with curiosity random I can relate exactly yeah it can be

But for that week I'm having a lot of fun.

I should probably get run over writing some of these notes down but that's a promises.

I mean I'll I'll sometimes write it like yeah back back when people read tutorials I'd type some forget something type it in and be like oh wow me three years ago was like really like really new something you know like what have I been doing since then

know from the horse's mouth so to speak what's just what how would you describe that um situation?

s that's the stuff that was sort of required the most amount of time in making sure we got it right.

The second thing it contained was the development backends as I called them the the dummy backend and the immediate backend both of which are really really useful for local development, unit testing, things like that.

The dummy backend doesn't run anything it just stores the tasks in a list so you can interrogate them in your unit tests.

The immediate one still behaves like a backend but it runs your functions immediately rather than spinning them off in the back end.

So it means that if you're using a package that um it means if you're using a package that isn't kind of fully tasks aware or you don't want to go through the effort of deploying things, your code will just still work but be a bit slower rather than these really important functions never getting run.

And that those are the design decisions around those.

Those two things, the the basic API contract and the development things, those were from day one intended to go into Django in the kind of initial push.

That sort of two and a half thousand lines of code, that's what we wanted to get in.

The third thing that Django dash tasks contained was a database backend.

This was this required a huge amount of work because, as I say, building task systems is really complicated.

And that's when I had to build one sort of from scratch on my own.

That's where the kind of confusion came because someone looked at Django tasks and went, but it's an API contract and it's some dev backends and it's this production backend, but I already have Celery that does all those things.

Why do I care?

And so I ended up shooting myself in the foot there.

In the initial ones, just trying to sort of get things out, it was the right thing at the time.

But as time went on, particularly once the code landed in Django Core and was released in 6.0, people got really confused.

They looked at Django Dask Dash tasks and went, What's this?

Why is some of it not in Django?

Why is some of it there?

How does this all work?

It ended up being some kind of like, why have you only done half the job kind of thing?

Yeah.

Like, and I'm like, no, what we have here is great.

It's really useful.

And everyone's like, but it's missing this thing, I don't care.

And so what I ended up doing was sort of in what I should have done, probably about a year or so prior, was I wrote down my thoughts and went, How can this actually work?

How should this work?

And then I shot an email to the steering council and went, Here's what I think we should do.

Am I insane?

And conveniently, and probably for the first time in my life, everyone went, No, Jake, you're not insane.

And so that kind of that shone this light on, okay, let's sync some time, let's make this much, much clearer.

And so, particularly now as of recording, but also probably for the last two months or so, things are a lot more sensible.

So, to give sort of the outline, if you are going, what on earth is this Django tasks?

There are realistically three things you need to care out, care about.

Django.tasks exists in Django 6.0 forward.

It contains two things.

It contains the API contracts built into Django itself, and it contains the development backends.

Those are there, those are ready to use.

If you're building packages that target only 6.0 forwards, use Django.tasks and pretend nothing else exists.

That is the dream.

That's what we wanted, and that's what we've achieved.

The second thing we have is Django Dash tasks.

Django Dash tasks is for people that want all those nice things that are in Django 6.0 but can't upgrade yet for whatever reasons.

Maybe their company policy is sticking to LTSs.

Maybe they're just on a really, really big code base and it takes a huge amount of time to move.

Doesn't matter the reasons, I want to support those people and get them using some shiny new things.

If you're one of those people, you can pip install Django-Tasks, change a couple of your imports to make sure that you're pulling from Django dash tasks, not Django.tasks, and you're good to go.

The third thing is, as I say, there is that database backend.

That is now separate from Django dash tasks.

You pip install it separately, it's Django-Tasks-dB.

You can pip install that.

And the nice thing is, is currently it still has a dependency on Django-Tasks, just because I haven't written the bit of code that swaps between Django and the Shimlayer.

Eventually, that package will have no dependency on Django-Tasks at all.

If you're on Django 6.0, it'll pull the things it needs from Django.tasks, and you can keep using it in Django 6.0.

If you're on, for example, 5.2, you pip install Django-Tasks, Django dash tasks db, configure your settings, write your tasks, and you're good to go.

Yeah, yeah.

Perfect.

And that you've got a Redis backend as well somewhere, haven't you?

But the whole point of this without going power mad is I need to be able to support third-party backends, things that are not written the majority by me.

And so because RQ is the tool I'm most familiar with outside of the stuff I've written, I thought, okay, I'm gonna write an API contract against RQ.

And so what also exists is Django-Tasks-RQ.

It fits in exactly the same place in the model I just talked about as the database backend, but it means if you want to use Redis and RQ, you just change the one line in your backend config that says rather than database, use RQ.

None of your you change that one line and nothing else in your code needs to change.

You'll go from using the database to using RQ and you're done.

At some point, and sort of this is the kind of the next step, is the I've been working on this API contract which shims between Django Tasks and RQ.

One of those things already exists.

It's called Django RQ.

It's the shim between Django and RQ.

So I've got an issue open against the Django RQ repo to try and get Django Tasks RQ upstreamed into Django RQ.

And eventually that means Django TasksRQ can completely go away.

And that's the dream.

It means if you want to use RQ, you install Django RQ and you do the things.

You don't need to care about I need this for the caching bits of R of um, sorry, I need this for the RQ bit, but because I want to use Django.tas, I need this extra package.

Everything is just there.

Is because the Django Q2 could have the their um integrate their back end for Django tasks, you know, the the salary people, they can have their their backend.

And for each option, then they're just implementing against the back end rather than you know you having to swap your development code for each for the particular queue that you've picked.

I want a database.

So I use the database one, it just works.

Then when you scale, you change an environment variable that says, actually, this is RQ now.

Give it the various credentials for connecting to Redis, and it just works.

Your your business logic doesn't change.

This episode is brought to you by Buttonown.

That's buttonown.com.

Email software for developers like you.

There are hundreds of email marketing software services out there, and they all pretty much offer the same thing.

Collect and clean addresses, send out broadcasts or drip campaigns, get analytics so you can see what's resonating and what's not.

Button down is designed to hook into the tools that you already care about.

Everything from static site generators like Jekyll or Hugo to payment platforms like Stripe and Memberful.

You can hook your site up to button down with just a form element or a simple rest call.

Write emails in Markdown and then get on with the actual work you're supposed to do.

New customers can save 50% off their first year with Button Down using the coupon code Django, and if you email support, they'll white glove migrate your existing subscribers and archives for free.

What are the bits of the Django Tasks interface that you think?

Oh, it would be nice if we could grow this, or it would be nice if we could um grow that.

Or actually, I want to keep this exactly at the scope it is.

Because an abstraction layer, you know, people always say of the ORM, well, it's the lowest common denominator because it doesn't do all the advanced features that you know you might need.

Your Oracle can do that or your Postgres can do that.

Well, it's going to be the same here.

There'll be features from individual cues that you can't use in Django tasks.

If you need to run a task in the background, that's fine.

It'll do that.

The things it doesn't do, which as you grow, as you start needing to depend on your tasks, become really important, aren't there yet?

And I do want to stress yet.

This is not a I've written Django tasks, I'm gonna disappear off into the sunset.

Like this is an ongoing effort.

Some of the things that don't exist, that would be really nice to exist at some point, is gonna be things like retries.

So you can say, okay, if this task failed for whatever reason, wait a bit, try again, things like that.

Timeouts, as I've already mentioned, making sure that okay, if this takes too long, kill it, start again.

Maybe we want to do some other really complicated things.

Maybe I want to, I've got a lot of work to do.

I want to run five different tasks, and once all five are finished, I need to run one final one to do a bit of cleanup.

That is possible to do in fancy things like Celery.

It's really hard to do in other languages and frameworks that don't have this as a native concept.

You can shoehorn around it by sort of spinning up other tasks, and it it doesn't feel very nice.

This is, I'm sort of channeling my inner, like basically going full feature creep of what could this thing do?

Because it's it's fun to work out how could these API look, what could they do?

What are the edge cases?

That's the the engineering mindset is what can we make it do?

At some point, someone needs to kind of basically someone needs to be the bad guy and go, no, you can't have that.

I think, unfortunately, that bad guy hat meets the steering council.

They have to be the bad guy in some of this.

They at least need to be going, or at least for at least saying they we can't do it yet.

That's the main thing is a, it's we need more thought.

It's probably not big enough quite yet.

And I think the big one, which is one that the steering council rightly lean on, is the we don't know how important this is.

Build it out of core.

Once we know it's really important, then we can include it.

Then the flips, the sort of the other point is, well, it's all right building it, but what about maintaining it?

Um you know, you know, a fully featured workflow engine to take the you know the whole shebang with, you know, dependencies and retries and branching and loops and those kind of control flows within a task's flow.

Well, there are whole products that do that.

And are we gonna import one of those into Django, even at the just the API um for it?

Or, you know, are we gonna have the capacity to do that properly?

Um, I think it's much better to have half a product than half-assed product, if that makes sense.

And so that's that's why I went with I'm gonna build the basic bits, let's get those in, because they're sort of there's much more obvious that these are necessary.

It's very obvious that you need some way of getting the return value of passing arguments.

That's fairly obvious.

Do you need retries?

Does everyone need retries?

Probably not for certain cases.

If you're building sort of a small internal back-end tool or a pet project, you might be alright without it.

It would be good if you had it and if it was like one argument away, fantastic.

But let's wait until we know the demand is there.

That like the vast majority of projects don't operate at scale, that they see failures on a regular basis.

The failures only come up when you're running things at scale, and then the the the uncommon things become common just because you're doing it a lot.

But if you're sending email back, as emails out, say, as a you know, classic example, you could be sending, you know, thousands an hour, which is much more than you know, almost all projects are ever going to send, and you'll never see a failure.

So, what's the need of retries to say, oh, we couldn't we can't have any background tasks because we wouldn't have we couldn't have retries for the the the two percent of projects that are doing things that hit the numbers?

And for certain things, if that 2% is 10%, it's really useful.

If that 2% are a vocal 2%, it might also be useful.

At the moment, that 2% is a hypothetical 2%, and at that point, 2% is basically zero.

If you want it, come and build it.

That I haven't sat down and sort of discussed with the steering council and worked out where that line sits.

But everything else, like, this is where open source wins.

If you have a weird bug, a weird jet, a weird edge case, open issue, submit a PR.

Let's make these things better.

I firmly believe that three or four goes in the ecosystem shows the right way forward because you know people have go one, go two, go three, go four, and then it it becomes clear what the right route is, whereas you could never have seen that without the the four attempts.

And it might take a while for those four attempts to play out, but the end result is something that's lasting and correct.

How does that fit your kind of philosophy?

And there is kind of we do need that kind of falling on the face of it, going, okay, that doesn't work, let's try again.

That's why I kind of put the initial DEP 14 proposal out there.

It got sort of, I think at the time it had sort of 150 comments on one PR.

Like GitHub got really, really slow trying to load that PR.

And that's that's what I wanted.

I want that it's all well and good building something based on either my experiences or what I think other people's experiences are.

I can't know what other people's experiences actually are unless they tell me.

So I make a proposal, people shoot it down, I make a different proposal, we try again.

Yeah, yeah, yeah.

Yeah.

Okay, good.

My dream podcast.

I always tell Carlton is I say hello and then I say goodbye.

And I just let Carlton and the guests carry on.

Because before we move on to a different topic?

I do want to address one one thing which people occasionally pop up and go, huh, that's weird.

And it takes I end up saying the same thing again, and they go, okay, that makes sense.

The with Django tasks, you create a task, you create a function, and when you call it, you pass arguments into it.

With Django tasks, dot tasks, dash tasks, whatever, those arguments must be JSON serializable.

There is absolutely no pickling in any part of Django tasks, and it's very, very much by design.

I'm sure we'll get into in a second, sort of my security team background on things, but it was very much chosen on a let's do things right.

And if I can shoehorn people down the don't touch pickle, JSON is fine, then it solves a huge number of problems.

Anyone who's done weird things and sort of like gone, oh, I'll just pass this request session object into my task, it'll work fine on say the immediate back end because it's the same process.

But as soon as it kicks off into like it could be on a separate machine, it could be halfway around the world, that like TCP session that is being represented by the request.session doesn't exist anymore because it's on a machine a thousand miles away.

And so stuff just falls over.

Whereas, again, as you say, Carlton, you want to implement things based on the lowest common denominator.

You can do basically everything you need to do with JSON.

The big one that comes up is um passing models between tasks.

Yeah.

What you can JSON serialize is a primary key.

So if

So all you do is you pass that through, and then in your task, you just retrieve it again.

That's got a bunch of nice benefits.

It means you're not passing around stale data where things might change between in queuing and running.

It means your task, um, the data that's stored in the task queue is also really, really small.

It's a number or a UUID versus an entire fat model with all of its columns.

It means you haven't got weirdness of maybe when you loaded that model, you deferred some of the fields.

Maybe you've only got one of these keys, but the task assumes another one exists.

So you've got a bunch of weird performance issues.

Those kind of things you still you don't have to think about when you're just passing IDs around or you're just passing these like primitive data types.

It just works.

Yeah.

And so for the kind of it seems to be a very vocal minority that are like, why is this not pickle?

There are lots of reasons it's not pickle.

I've just talked about all of them.

And so by forcing you to think about things in JSON, you avoid a bunch of weirdness for a slight little bit of, oh, I need to like grab its primary key and then do a database query to pick it up.

Doing that little bit of work for avoiding a bunch of extra issues felt worth it to me.

It felt worth it to everyone else who reviewed the depth.

And I'm pretty happy with that stance.

We see similar things with people working with channels or async code who they're like they're passing it, trying to pass a model back from a co-routine is hang on a second, but that could be crossing a thread boundary because it's crossing a sync to async um boundary, and it's like you can't pass models across threads.

This is not you know, just don't do that.

If you need a model in a particular place, look it up there and then.

So I think that's a great decision.

Because I think we've talked a little bit about the security team, but maybe you could shed some more light on that, right?

Like do things sometimes come in through Django via Wagtail, vice versa.

How does how does that play out?

I'm Wagtail is quite a large project that has quite a large number of installations.

And so it gets sort of it exercises a lot of edge cases that a standard Django project might not.

You're doing a lot of things in Wagtail.

The content management system pokes a lot of internal things in the Django internals because it's trying to do some of those things.

Some of that works well, sometimes it doesn't work well.

And so there are sort of, I suspect that a lot of the um the core Wagtail team have raised a lot of issues upstream towards Django and even to other projects, going, This weird thing happens, is it a bug?

Is it a feature?

And so that kind of that relationship works really well.

Um we have also had sort of the relationship work the other way, whereby Wagtail goes, we need this sort of thing to work a little bit nicer.

Maybe we put it up less as a like we've hit a bug, more a let's change this so it works a little bit better, and then it can cascade downstream.

It's unfortunate that both projects have quite long release cycles.

It's not a like we open a ticket and a week later it's available on PyPI ready to go.

Like it's gonna take kind of three, six months sometimes to actually be able to use the thing.

But by contributing upstream, they can come back down.

It's useful to us, and hopefully it's useful to someone else as well.

I mean I've I've been a um Wagtail user for the last few years, and I've uh took me a while to get cost accustomed to the the Wagtail release cadence.

It's all good, it's brilliant, but uh it wasn't it was because um Wagtail will use dependencies that are then pinned because incompatibilities and whatnot, and I'm busy trying to update my dependencies and um finding conflicts with what Wagtail had pinned versus what I wanted to use, and like how am I gonna work for that?

I think we're at at least three or four by now.

Yeah, yeah, yeah.

But I've I've got kind of come to terms here.

I think it's only um what's the model tree library that you use that uh um

Yeah.

Like I've from sort of very, very early in my professional career, two things have stuck out as being particularly interesting, and that's security and performance.

And so naturally, Django has a security team that kind of do quite a lot of really cool things around maintaining the security of quite an important um software package.

There's no kind of it's not obvious how to get involved in that team, how to join that team.

And quite frankly, it still isn't.

Um, that is an ongoing problem.

People are thinking about how we can deal with that.

There are some things happening internally to try and make that a bit clearer.

Um, I just happened to basically almost be in the right place at the right time.

Um at the time of this, um, Tebokalas, the then president of the Jango Software Foundation, had mentioned that the security team was looking for kind of some more people.

And I went, I can do some of these things.

At the time I was on the security team for Wagtail, all of our kind of anything internal at Torchbox security related generally passes my desk.

And so I was like, I have some experience, like am I helpful?

And at some point I assumed he went away, talked to some people, and I got a message in Discord sort of three or four weeks later going, Would you like to join the security team?

And I kind of I had to kind of double take at that message a bit, because I was like, You want me?

Because it is this kind of it's really weird.

Like, Django is used by so many different organizations, so many different individuals, and there's a pool of what, eight, ten people in charge of making sure it doesn't have massive security holes, and I am now one of those people, is a really, really weird and humbling feeling, is that like maybe I am kind of good at what I do.

Like, there is a list on the Django website that has security team, and there is my name next to it, and I can point to that and go, I am making this thing better, and that feels pretty damn great.

Yeah, good.

I like that.

Like, how does it how does it sort of function?

It's a mystery, right?

It's what it's this secret, it's this secret cabal that no one knows anything about it.

Like we are human beings, like I have met the majority of the other people on the team.

We are human beings, we're not like supercomputers sitting in the background.

The the sort of the start to finish on how things work is if you're if you are sitting for some unair reason listening to my voice at the moment, going, I have a security hole in Wagtail, but I don't know what to do about it.

The thing to do is, at the time of recording anyway, is email security at DjangoProject.com.

That goes to all the members of the security team, we'll triage it, we'll talk to you, we'll work out, is this a bug?

How severe is it?

Can we reproduce it?

Is it like, are you holding it wrong?

Things like that.

The security team gets a lot of emails.

Most of them are spam, unfortunately.

Right.

Um I would say from sort of a volume, maybe a third of them are like people legitimately trying to help.

What that means is that two-thirds of the time, when I open an email, it's garbage.

If someone goes, I have found a bug in Django, here's a bunch of code, here's some bits of stack trace, here's why it's a thing, it can take a lot of brain power and a lot of time to go, no, this is nothing.

We get a lot of things varying from you're calling APIs that don't exist, or you're calling weird bits of internal stuff that is not intended to be used in this way, or is not publicly accessible, or even a like we've had people report that, for example, when you call cursor.execute and you pass it arbitrary user input, you get SQL injection.

And the response is yes.

If you take a user's input and pass it to the database, that is SQL injection.

That is kind of you have to sanitize your user input.

And so I'd say sort of nine months or so ago, um a bunch of the security and got together and we sort of added some extra things into our security policies to try and make some of those kind of you're holding it wrong more obvious so that someone can kind of self-police certain things.

So, for example, you need to make sure that if you're passing random stuff into the database, you should sanitize it.

In most cases, the RM will do that for you.

If you're passing it into raw SQL stuff, that's on you.

You need to be dealing with that.

Things like making sure that some of the stuff you're passing through the Django template language, if you're passing the entire works of Shakespeare and complain that it's a little bit slow, we can't do much about that.

We're limited by certain things.

We can make things faster, and we're always up for this is a bit slow, I want to make it faster.

Absolutely, let's do that.

But does it constitute a security issue?

Probably not.

Yeah.

It it doesn't work that way.

It's a feeling.

Could this be a problem?

If it's a yes, we'll triage things through.

Ideally, the person who's reported it has gotten, if you change things like this, it gets faster or better.

Great.

We love people who actually submit um full kind of proof of concepts or patches, things like that.

It makes our lives so much better and so much easier.

But some people don't.

And we also don't want to put that as a requirement.

Some we'd want to know about any vulnerability, even if you've gone, I don't really understand the internal bits.

We get reports about some of the weird internals of the RM.

I have never dug into the weird internals of the RM.

So I go, I can reproduce this, but I have no idea why it's a problem.

And there's someone else on the security team that does know the internals of the RM.

They can look at it, they know that side of it.

I can help by going, this is legitimate, and sort of shield them from the garbage, as it were.

Yeah.

And then there you can dig into the internals.

Yeah, that's good.

Um, I guess we need to talk about um AI and the the the the the effect that that's had on the process and the team.

Anyway, anyway, let's not.

Apologies, Adam.

Yeah.

It's not like there are bots out there just submitting stuff.

One of the big things is that people will sort of go, I think I have found something, dump it into an LLM.

Most LLMs are intentionally designed to make you feel good about yourself.

So they'll go, so if you ask it, is this a vulnerability?

They'll go, yes, you're absolutely right.

This is definitely a thing.

Exactly.

But then what happens is the LLM will naturally just sort of vomit out a large amount of text.

And so what goes from a like, if you call this function with this input, it's really slow, comes out at about a thousand words, and they'll sort of talk about things like, oh, well, it affects these versions of Django, and if it's really slow, it could have these knock-on effects.

And at a really at a certain stage, we sort of stop caring about all of those details.

The things we need to know is if you do this, this happens.

That's the important bit.

If you want to go into the some more bits of details and go, okay, it's really slow, because it's in this part of Django, this has this knock-on effect that makes this bit particularly slow.

All really, really helpful bits of context.

But that can be a sentence or two.

That doesn't need to be a paragraph of stuff.

And that's the bit that kind of really, really slows things down.

It goes from I can fit this in like half a screen and know this is all the context I need, versus I need to kind of sit and read and think and digest because it's the same amount of information in about 10 times the amount of text.

And unfortunately, that's not something that is just plaguing the security team.

I know in the last week or two, um, Natalia in particular has been sort of getting really, really frustrated at the low-quality pull requests that have been coming through to Django repo, where they are kind of a lot of noise for something that is either not actually an issue or it's someone going, This could be better, without any justification.

And that gets to be a real problem.

Um, security kind of it's not unique to just us, it's not unique to just security, but it just pads a huge amount of time and effort into reviewing every single report.

This is you know, okay, we're probably understuffed slightly, but it was fine.

It was a it was a a comfortable pace.

But the last year or so, I think it's just picked up and it was like every every day, multiple issues, you know, and it's it becomes too much.

It becomes, you know, you never need 20 people on the team instead of 10, and well, we don't have that capacity.

Almost everything is done by email.

Um, and so I've not looked at my emails for probably almost a week or so.

I've got 16 unread emails in that inbox.

Right.

But that means that some of them are the initial report giving us garbage.

And so I need to then sit down and go through 16 emails worth of following a discussion, working out, is this garbage, is this not?

For some cases, I can just go, they've concluded it's garbage, it's garbage.

Occasionally, and this has happened a couple of times, is we'll look at something and go, actually, I don't think this is garbage.

This is there could be something here.

Nine times out of ten, the person who reported it hasn't seen it that way.

They have submitted garbage, but if you look at it with a slightly different angle, you go, okay, well, if they've done this, it's actually this other thing.

Unfortunately, because of some of the specifics of things, I can't talk about the like, oh, there's this API that if you call it in this way, it's a massive problem.

Um, and also my memory sucks, and so I can't remember most of that detail.

But things like this happen.

There's a lot of nuance around security of massive frameworks like this, because it's Django, the code base itself is gigantic, and the security team kind of has remit over literally all of the code, but just some of the kind of the symptoms of things.

If it's slow, that's a problem.

If it's too slow, it's a security problem.

Yeah, yeah.

So um Jacob Wals, the fellow, put a post on the um um on the blog recently about some of the recent work in the security team, because there was, you know, a number of issues that have been patched over the last month, uh last few months, and some of those were SQL injections, and they were, you know, perhaps um question like they were you have to go very, very wrong to to do this.

But the by the they were legitimate SQL injections, and by the the standards there, um an SQL injection is um category high, it's uh you know, it's a serial severity high issue.

Uh and it's it's kind of a this problem where it's like, well, we've got a release coming up, it is a high issue, and then people put on a team to to do it, and then they're like, oh, but we would never just pass this, you know, these keyword args to filter in this way.

And it's uh it's kind of difficult.

How do how do we balance that when there's so many reports coming in and we're trying to do the best and we've got our procedures?

Yes, it was really, really famous hijack potentially.

It was a problem around password reset where emails that as a human aren't the same, the database would look at and go, these are the same, and you can take over accounts.

It's been fixed for like eight years now, it's not a problem.

But what happened is that affects basically everyone.

It had a because it was very well known, the time between there is going to be a security release and it being released upon the world was expedited.

It was one day rather than the standard seven.

And so naturally, I getting that email going there's a release tomorrow, lost my mind because I was like, what the hell is going on?

About a month, a month and a half later, there was another email that came around.

It was again, it was a security release, it was marked high, but it had the standard seven-day embargo.

I was like, oh god, is this gonna be another kind of I need to spend a day patching all of my projects?

Somewhat conveniently, I happened to be out of the office on the day that um that it was due to be released.

So I was there, I was sort of sitting um in the passenger seat of the car, panicking, checking my emails, going, okay, is this a big thing?

Am I gonna need to like triage some things in Slack?

And it was it was a small SQL injection specifically on Oracle.

Right.

We weren't using Oracle, we were using Postgres.

And so to us, it was not even a low, it was a complete non-exal.

But if you see an email saying there is a security vulnerability, it's rated high, you just see that you have to react based on that.

And I don't actually think there's an easy way of fixing that.

You can't give more, you can't give too much information because someone will try and reverse engineer it or work out what it is, or try and kind of beat that embargo.

We also can't do um, we can't sort of sneak little bits of information to certain people.

For example, Django has got, I forget the formal name of it, but basically, if you are a big enough important organization, you can ask us very, very nicely, and we will tell you the specific details of that security release when we send that early warning rather than when we sort of do the final release.

And what that means is that for for certain companies, where it's really, really important they get patched as soon as that information is out, they need a little bit of warning.

And we want to accommodate that, but we can't give that to everyone.

At the time I was working at a small Django agency.

Our projects are not big enough and important enough to need that kind of information.

But where does that line sit?

Yeah.

Who is big enough and important enough?

Is it based on the size of the team, the number of developers?

Is it how important the project is?

That's generally the way we've gone.

And do

Oh, we can't know that.

I know, I know.

So you've it it they need time to be able to get it into patch their version of Django, because they're not they're not necessarily shipping Django 6.0, they're shipping 5.2 point with all the patches, and you know, they they need time to be able to update their their um repository.

So when you apt get or apt update or unattended updates, it automatically pulls in the Django update for you.

You're so we have security contexts at Debian at Red Hat, they get the emails.

There are some organizations that get a similar thing.

Um it's really, really ad hoc.

We kind of we don't have rules for it.

There are some listed in the docs, but they're not perfect.

Um it's very much a if you ask, we'll tell you.

In the last three weeks or so, we had two companies come to us who I will not include their names.

One we said yes, one we said no.

And it just comes up that way sometimes.

Exactly, yeah.

I did learn recently in in the Go ecosystem, they have a kind of security release um configuration basically that encodes as part of a security release.

Like undoing a release, the issue is in this function if you are calling it in this way, and it's programmatically defined.

So it means you can run like basically static analysis on your project and know not only am I using like the standard one, am I using dependencies that have vulnerabilities in, I can also know am I vulnerable to it?

And that would be massively helpful in an ecosystem like Python.

But the reason you get some of that in Go is it's got static types.

Python does not have quite so static types,

let's call it.

Oh, indeed, yes.

And so that leaves times for books and projects, the important things.

Do you I believe you have one queued up, Jake?

Is there a book you want to recommend?

My brain works in very, very weird ways.

I've I'm not actually a massive reader, sort of from I would say in the last 15 years, I have probably read two fiction books at all, and I don't think any non-fiction books from start to end.

I've tried to trick myself into reading.

Like if I buy myself a Kindle, I'm kind of playing with some tech, but also reading.

It worked for a couple of months and then I got bored.

I'm trying to pick that up again.

And one of the things that's helped is conveniently at the same time, there's sort of been a tech reading group that's formed at work of people that are kind of interested in reading and learning stuff.

And so the the first book and the book we're reading at the moment is um designing data-driven data-intensive applications.

And so conveniently that's also very well aligned to kind of the systems programming, performance, background tasks you stuff that are kind of interesting to me at the moment, and so I've quite enjoyed.

I'm a couple of chapters behind the the rest of the group, but kind of slowly catching up, being involved, just talking about some of this stuff has been really interesting to me

at least.

I um had a group with some people from the Janga community reading um Rust in Action back in a few years ago, and it was really good.

We meet up.

I think it might have been been during the pandemic time, so it was nice to have some sort of social interaction as well as you know, how are we going to get through this book on Rust?

oh I do, I do.

Um Jenny Chase is an analyst that works at um Bloomberg and back with oil over a hundred dollars a barrel.

I thought I'd uh pick it up.

It's it says it's about solar power finance, it's sort of more really about solar power.

There's a little bit about finance and prices and supply and demand and guts guts and things, but it's a really interesting book about um how um well about the the the economics of solar fire uh of solar power and then how um renewable energy is grown and will continue to grow, especially you know when um alternatives are inflated due to geopolitical events.

So I'm reading that solar power finance without the jargon.

There was just some study showing that solar panels can they thought they had a lifespan that was quite short, but in fact, almost indefinitely for decades they can continue to

I mean, one and I'll just say one interesting thing as an American with what's happening.

Um the US, we're the largest producer of oil and gas, and because of that, um we can't make our mind up about what we want.

You know, so China, China can't produce enough oil, so it's forced to invest more in alternatives, even though it seems fine to deal with polluting things, like it just knows it has to.

The US doesn't really have to, and so we're much more prey to do we want high prices, do we want low prices?

There's a whole discussion around that, but it's a little more complicated in the US, I guess, which is one of the reasons why we're so reticent, at least this year, to uh invest in alternatives.

I'll just say that.

Um for me, quickly, just fiction, the passage by Justin Cronin.

This is actually a reread for me.

I think it's 10 years old.

Um, and this is catnip for me because he's like a literary author who won a bunch of awards and then decided to write something that people wanted to read.

So it's essentially a vampire book.

Um it's actually a trilogy.

It's you know, um, it's just really good.

I just really like, you know, a plot and to me this, you know, little quick sketches and portraits, and it's uh yeah, I need that.

So highly recommend it.

Um projects, I'll just keep going quickly.

So um this is well, as we record, which is March 10th, this will come out a little bit later, but um Andresh Karpathi just released a gist uh micro GPT dot PY, which is an entire um way to run a GPT in under 200 lines of code.

So uh train, inference, tokenizer, autograd, initialize parameters, optimizers and buffers, repeat in sequence, and then do inference.

And it's not uh optimized at all for scale, but it's meant as a teaching tool.

And I've been sort of circling around how do you actually build these things?

I recommended recently there's a book, Build Your Own LLM from Scratch.

This is really great to just see in one file.

Oh, this is you know, academically how it's all happening.

So I would highly recommend check it out.

It's commented.

Um very cool.

Cool.

Nice one.

Check.

I found myself sort of originally sort of when I was much younger and had a lot more energy, I would kind of I'd be having thousands and thousands of different things.

And slowly my GitHub is slowly filling with these kind of the things I've built.

And I'm going, yeah, I'll I'll maintain that.

And then I realise, oh, it's been like two years since I've had a commit.

That's fine.

That's been not so great.

Um one of the things that's kind of particularly in the last week or so that's been on my mind is actually one of the things that Carlton and you've been working on is which I'm gonna hand over to you quite nicely, is around type safety and your project mantle.

So go ahead, you can.

But this it's a way I've been writing typed Python code quite a lot, is that by its nature, a lot of Python and Django is hard to type.

And so writing a kind of type-safe layer around an untype-safe API is a really, really nice thing.

I do want to kind of rein the operate a little bit in that it's a really, really nice idea.

But I've also seen it in a couple of other places.

That's okay.

Um, not not in Python.

Actually, funnily enough, it's also a really, really common model in Rust.

Um, so I mean, at least you can say you're copying this thing from Rust and pulling it into Python, which is uh something.

Django Mantle is my project of the week and um also its partner um project Mantle DRF, which is uh DRF um add-on for it, so that you can use it with your existing DRF project now.

But what it is is um so it's it's builds on atters.

So you talk about a type safe layer.

So Django doesn't, you've got your Django model, you're never gonna make that type safe.

Like and there's some new pep about um I can't remember adaptive typing, or I can't remember exactly what the the word is.

Even if that comes in, we're not gonna be able to retrofit that into the art.

We're just not like Django is dynamic to the core, it's built on you know um Python's dynamic capabilities, and we should just embrace that.

But for my work code, I I want to have like modern Python, modern uh simple, plain, typed Python classes.

So I use Atters for that.

So I use map atters for that, and then the challenge isn't writing the Atters class, right?

Anyone can write a data class, you know, it's got a name and that's a string, and it's got an age, and that's an int, and you know, those classes aren't hard to write.

The problem is mapping that from the ORM.

And so what Mantle provides, and it uses Django readers that I've talked about on the show a billion times, I love it.

It uses Django readers to create efficient RM queries from your atter class, and then those queries will have just the fields that were in it, and it works with nested data doing pre-fetches related automatically, and and then it serializes back.

So you just do a query, you give it a query set and you the shape you want, and it gives you back typed objects that already did all the fetches, and then you can pass them to your um API, or you can pass them to your templates, or you can add the you know the domain logic that you want, and it's all independent from the ORM.

And so it's a way of decoupling your business logic or your domain logic from the your ORM models.

Your ORM models are busy enough already, they're busy dealing with the relationship to the database.

And one of the big problems we have is when we slap on loads and loads and loads of business logic into our models and they become this big, difficult to deal with monstrosity, essentially.

Where it's a way of cutting cleanly between Django's dynamic core and then the type safe layer on top.

Now, the the the bit that Mantle Mantle brings is that flow.

Atters, not unique to me, not original to me.

Catas, the the serialization tool I'm using, not original to me.

Django readers, not original to me.

It's a synthesis of how I've been using those tools over the last you know few years, basically.

And Mantle DRF is, as I say, it's a it gives um versions of um DRF's generic views, its mix-ins, its viewsets, so that you can just change your imports and you know the use a mantle shape class in your DRF API.

So anyway, go check that out.

It's on my website.

And uh, I'm glad you should

be looking at it, Nick.

It's like

Yeah, yeah.

I'll put a link.

Yeah.

Yeah.

Exactly, yeah.

And Mantu in particular was just that kind of it's nice that for a while I found this kind of type safety on top of untype-safed stuff as a really good way of thinking about stuff.

You can kind of hide a bunch of weirdness, particularly if you need to kind of do a bunch of weird things to make it type safe.

You can hide that stuff in a function and you can just use this public API that makes things nice.

Yeah.

Why can't we do it for safe and unsafe types?

And it just felt like Mantua in particular, obviously, is the Django way of doing it.

But that that thought process of kind of facading stuff, I think, is a tool that works for developers in general, even outside of Python.

Um and I think that's really important as well.

So and you and this was like you remember that thread on the forum about Django needs a rest story?

And I said, Oh, well, we need a serialization story.

Well, that's at us and katters, but for me, but the point I wanted I I said was, well, it's got to be ORM aware.

And that's the that's what Mantle gives you as well, is these is the type safety and the modern serialization story, but it's ORM aware.

And that's so anyway, I've just I'm putting it out now, so I'm glad you've seen it already.

Are you going to be able to do that?

So I'm gonna do a long version director's cut talking about you know sort of typing and mantling a bit more depth than I was able to do in um on the PyTV event, and then I'm gonna give a shorter version again at um PyCon Italia.

So if you're anywhere in Europe this spring, uh anything you can come and see me talk about.

All right.

Is there anything we didn't ask you or that you want to highlight?

Magic wand.

All right, I'll hand it to you.

Digitally hand it to you.

Here's your magic wand.

You can't change tasks, but something else in Django.

Well, you don't need a magic wand for that.

It's like it's it's kind of hard to read it.

So it's just this idea of code community, just oh, wouldn't it be nice if we could just change one thing?

I think one of the big ones is the Django core code base seems to be a little bit allergic to dependencies.

That's possibly a spicy.

My hot take is I don't think that's good.

I think I don't know if I go as far as to say it's bad, but I think if we were less allergic to things, some things could be made a lot better.

Cool.

So, for example, if we I'm not even thinking like on the massive things, like, oh, we should just depend on Django Restoring work and be done with, and we can solve the rest story once and for all.

I don't think that actually solves anything.

But there are lots of things inside Django where we've had to not so much reinvent the wheel, but we've had to kind of write some stuff ourselves because we can't use a third-party library, or we've had to vendor little bits of it to work for what we need.

If we could instead pull in a dependency, that works out great.

If we can um it's it was really restrictive in Django Tasks, because that's meant to be a backport.

Django Tasks has no dependencies.

It would be really nice if for certain things I could pull in various dependencies for other things.

But I know that if as part of shipping Django Tasks I also shipped five additional dependencies for Django, it would be a real problem.

I don't I don't really know what the solution is to this, but I think sort of possibly the easier way is making a good argument for this dependency in particular brings us a bunch of benefits.

Um, particularly around like some of the like utils and the core HTTP bits of Django.

They're not unique to Django, they are used by basically every web framework out there.

Why can't we have we need one unifying implementation to bring them all together?

Um, I don't want to start this meme again that basically everything I do is making a unified implementation, but at least if we if there are implementations already out there, maybe we can use them.

Maybe we can remove a bunch of stuff inside Django, maybe we can shrink the code base a bit, shrink the amount of work that the fellows need to do and use trusted, not just pulling in a random thing that a person's written, third-party tools, ideally as hard dependencies rather than sort of the optional ones we've already got, that bring in those extra benefits without making our without giving ourselves a ton of extra work.

That's a really good one.

It's the whole hyper project around HTTP and Python, which, you know, they've got some good stuff, good code in there, and it's trustworthy enough.

You use the word trusted, that the the the issue with you know the JavaScript project that brings in 400 dependencies and then they all break and so can't update, right?

That's the sort of the the that's the historic counter pushback, is why is it very few projects have the stability requirements that Django would need to you know pull them in as dependencies.

Um, but yeah, those kind of core cases I think is a real good case.

I like that.

That's a good, good, spicy hot tag.

Well, Jake, thank you so much for coming on.

Thank you for thank you for all the work on TAS.

I mean, Carlton was sort of whispering to me, you know, like, oh, there's this thing coming, this thing coming.

So, you know, um, and it's still coming.

I think I'm glad I'm glad you had a chance to address some of the questions and TAS is not going away, and in fact, it will be easier and more powerful to use going forward.

It's we've done a chunk of it.

It's usable today.

If you're on 6.0, use Django tasks, get on with nice, easy background tasks.

As time goes on, it'll just get better and better and better.

It does also, it's I really hope it's not just me working on this for the next 10 years.

Like, if you are interested in background tasks, API design, the kind of the systems programming that you need to do for building background workers, please sort of appear, contribute to things, get involved in discussions.

Like, I don't want to have to maintain this entire thing on my own, please.

Yeah.

All right, so we're DjangoChat.com, we're on YouTube, and we'll see everyone next time.

Bye bye.

Bye bye.

See you later.