Transcript: Boost Your Django DX - Adam Johnson

Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework.

I'm Carlton Gibson, joined as ever by Will Vinson. Hello, Will.

Hi, Carlton.

Hello, Will. Happy New Year as well.

And today we've got Adam Johnson back with us, who's, you know, you know who Adam Johnson is,

but he's a Django Technical Board member, Django author.

So you've all got speed up your Django tests.

And coming up soon is improve your Django developer experience, which we're going to talk about today, and various other things.

Adam, thank you for coming on the show again.

Thank you very much.

And Happy New Year to you both.

Happy New Year to you, sir.

Carlton, you missed one.

Adam just won the Malcolm Frederick Memorial Prize.

Yes, he did.

Which is well-deserved.

I voted for you.

And I would just say the two top vote-getters are the two of you this year.

so it was a very very easy choice um for the board no thank you that's good congratulations

very well deserved thank you yeah well done adam like um you do you're massive you're doing posts

all the time you're putting out packages after packages after packages you're always there on

the mailing list you know giving people the the welcome pointers so super thank you very much

i'm honored to receive the prize and it came in uh the worst year of my life so it's a nice end

helped me a lot.

We have lots to talk about.

I think there's going to be a lot of the two of you

talking, but I would love to tee up the Django

Technical Board. Maybe we could tell

folks what that is and what they do since

Adam, you're on it, and that's quite a

big deal. It's also a relatively new thing

and how that works

with the Django codebase for new releases since

4.0 came out just a couple weeks

ago.

Sure. So the Technical Board is

five members voted

in for each release cycle.

And where the buck stops with decisions on whether things go into Django or not.

Like quite often, like features are clearly like a good idea.

So it's okay for a few people to suggest it and the fellows to go and review.

But for larger scale changes or contentious changes, a technical board vote would be held and we'd need a consensus whether or not something would be added.

Yeah, so if it was an easy decision, it would not reach the board.

In the same way for a community, that's what the Django Software Foundation handles, non-technical things.

But the DSF has no sway over the technical board decisions by design.

Yeah, exactly.

Actually, I should know.

Does that go to the long-term releases, or is that the number releases?

It's every number release.

Like the 2-2 to 3-2, or is it?

So I've been on the technical board since 2-2, and voted in each time since.

There's like maybe eight or nine people who've stood over that time.

So most people who are standing are getting elected at least once.

It's just kind of cycling between a few people, I think.

It's kind of interesting that because, I mean, one comment was made on the Django Software Foundation DSF members list that the elections hasn't been particularly diverse and that it's all, you know, it's all white men still.

And part of that, I think, is because we still have a quite narrow experience contributor base.

Like the qualification for the technical boards, you have to be involved in the day-to-day running.

I mean, you don't have to have been there forever, but you do have to be active on the discussion list and things like this.

And we still don't have a very broad contributor base there.

So, you know, there's not necessarily a wider pool to draw from, which is something we, you know, we kind of think about.

It's definitely something that would be nice to correct.

It's very hard because you have to have been contributing for years

before you're really eligible to be on the technical board.

You have to have some deep knowledge of various parts.

And of course that's correct, right?

There's a tension.

It's chicken and egg.

Yeah.

Well, the DSF doesn't have that requirement.

So, you know, you don't even have to be an individual member

to be voted on, though it is voted on by the individual members. So you do need to be known

in the community. And all those things are true. I think one thing I found this past year,

I think it's basically for the first time since I've been on the board, this is my third year,

we had basically all the same people reelected. And diversity is nice. I mean, I'm not planning

to stay indefinitely, but it is easier logistically to have the same people because

certainly for two years terms because you kind of know how to do things whereas initially that first

couple months people are ramping up so when we my first year there was or my second year there was

like half the board was new and so a lot of time was spent kind of getting people up to speed so

anyways internal baseball but maybe longer term there is something i think to multi-year terms

just to kind of avoid the ramp up and ramp down but at the same time a recycling of voices is

healthy as well absolutely i'm not i'm not sure about this can is it it's not so there's janga

release every eight months it's it's every is it every major release cycle now so it's every

eight months or every two and a half years for the technical board i thought it was every eight

months right because it was two two i was voted in on and now i was in the four zero yeah yeah

it's kind of talking to maris about this do we need a new election or it's a ceremony that goes

pass very quickly doesn't it yeah well eight months is too quick i think you know um well i

think you know that the dsf board a year is if it's if i had total control i think is too quick

too i think there's something to you know perhaps a you know three out of three two or two two three

year term just to avoid the nuisance because it does take up quite a bit of time um anyways

something for all of us to discuss with our our bodies so we should give a couple of examples of

things the technical board done so it not quite often we discuss on the mailing list and we just

come to a consensus right and it's it's but there are a few issues so one that has been really

important for um was a discussion about typing because there was no consensus at all and um

you know there was a dep and there was massive discussion on the mailing list and there was a

you know maybe even been draft pr i don't know um but you know maris and i just aren't in the

position where we can go oh yeah we decided to merge this or we decided to close this so we

asked the technical board for that and you were involved in that discussion then yes that was a

big one and we came to the conclusion that uh jagger wouldn't add type hints at the time and

that was like a couple years ago at this point and obviously typing has moved forwards as many

of us have gained a bit more experience and so it would be nice to reapproach that decision

at some point yeah and yeah i guess the second point is who's going to do the work because it's

it would be a lot of work to add the type hints into django yeah i mean so you've been blogging

loads and loads about um you know types um so would it be fair to say that you know you'd be

you'd be keen now to think again about adding those yeah i think i i'd probably be in favor

but we'd probably have to come to a decision that we'd only officially support my pi or

um something like that because you will need a plug-in for a type checker to tell it

you know when you make a django model the field definitions map onto these types

okay um i mean my one thing that i've i kind of been flirting around is around the widget api

because you know there's lots of there's lots of areas there where it's like what do i get

what exactly do i get out of value for data dict here and you know i could really i could really

appreciate some typing there but my kind of thought is oh i wonder if we could do this just

in say the widgets file you know just in one kind of module where it's it's at the kind of leaf you

know that the widgets don't really impinge on anything else they're not impinged by anything

else could we type those and that might be a sort of progressive way of doing it i don't know yeah

progressive would be would be how i'd like to see it happen anyway um yeah all these leaf modules

like all the things in django utils um they'd be perfect candidates to be in with but we'd need to

discuss is there value in adding just a small amount of type hints initially or is that going

to make it harder for people because then they're going to have oh we're trying to use the tight

pins but everything else is showing up as any and causing us headaches yeah some research is

definitely required yeah okay interesting interesting something else technical i'd love to

mention um so as i went to bed last night i saw there was a there's a new security release and

this morning i see in the notes adam you've done up a whole blog post on it which is insane so i

wonder if we could briefly talk about that and that's a chance to mention that you're also

on the security team of Django which is a separate group for a few months now and

oh only a few months I think Colton invited me in September something like that oh wow well

I just assumed you were part of it the last couple years look let me give a bit of background let me

just give a bit of background there and um so there's a I don't know about 10 members some of

the older contributors that have been around.

And, you know, they're super knowledgeable,

but not necessarily involved in Django day-to-day anymore.

And, you know, there's a few people

who are still very active,

like Marcus Holtzman and Florian Apollina.

But we were finding that, you know,

it was me and Marius and maybe Marcus

and maybe Florian commenting.

And we just needed some more people

who were sort of directly involved,

kind of very regularly.

And so amongst the security team,

we talked about it with Adam and Nick Pope came up

and so we invited both of them and they came in

and they've been super helpful and active

because we get a lot, not a lot of reports.

How many reports would you say we get, Adam, in you?

I'd say the average is one every two weeks.

That's really actionable.

Would you say that's about it?

Yeah, I mean, it's not a lot,

but they all need dealing with, they all need looking at.

And it's not easy.

It's really not easy.

You know, I mean, I'm not a security expert.

And a thing comes in and it's like, I don't know, is this?

And without the security team, there's no way I could handle those issues.

Yeah, okay, some of them, it's easy.

No, that's not a problem.

Some of them, oh, yeah, that clearly is.

But then there's this vast area of that's a really difficult question.

I think I really like seeing how everyone, like, filled in the holes of knowledge and ideas in solving these issues.

And yeah, because we'd have a lot of back and forth before even our pull request is open and to be like, oh, we could do this, we could do that.

And yeah, so there were there were three fixes in today's release, right?

Yeah.

Shall we run through them quickly?

OK, so the first one was a denial of service possibility in user attribute similarity validator.

Now, that's a mouthful, isn't it?

That's a password validator that checks that the password given isn't too similar to a user's other attributes, like their email address.

So they're not just recycling their email address or adding a one and exclamation mark at the end for their password.

Yeah, who does that?

I think quite a lot of people.

That's exactly what I do.

Yeah. So it's active by default when you do start projects. But who was it that reported that one?

Chris Bailey reported that if you submit a very large password, then it has kind of n squared or

exponential runtime. I'm not sure which. You have to submit like a very large password, like 100,000

characters, but then that can lead to many seconds of processing on an average server.

you know submitting a hundred thousand

letter A's over gzip is not much of a request body for the client to send, but then it can result in

in many seconds of processing so it becomes like a denial of service vector.

Someone submitting loads of these requests could cause your server just to hang up.

So the fix that I came up with along with Florian, Carlton and Marius is to

avoid checking the similarity if password if the password is like so much longer than the

attribute value that it's not going to be regarded as similar anyway so how do you come to that

conclusion right i mean are you like who do is you didn't did you come up with that or is that

something that other security researchers have used as a best practice um we had to come up with

this one i don't think there's much groundwork for using this kind of algorithm right now um

yeah i mean i mean this was quite an interesting example because adam had an idea to just well you

know if we would take it like if it's if it's a plural you know a multiple sufficient multiple

cut it off and then there was various back and forth and different approaches it was really nice

this adam's point about like everyone throwing in their ideas was that there were two or three

different takes and it's like yeah actually this one's quite sweet and it's a nice ratio that

this is a realistic cutoff at different scales of input and yeah and whatnot cool and then there's

two more just okay we'll go through them a little quicker because i'm a bit less familiar yeah

um the second one was with the dict sort template filter this is one that i've not seen used but it

takes a list of dictionaries and then sorts them by a key or perhaps an attribute um and uh security

researcher at SonaSource, Dennis Brinkroff, submitted us a report that you can use it to

access arbitrary attributes and inside of a string, you can access the characters of the

string one by one. So if a user managed to submit which order to sort your list of dictionaries by,

they could sort them by character zero, then character one, then character two,

and see the sort order changing. And then they could infer the value of that attribute. And if

that value was like security sensitive like a password or api key they can kind of infer okay

this this user appeared at the top of the list so their password starts with a and then when i

sort by character two they're in the middle of the list so that's probably something around l

and then i can kind of combine that with other info to get a statistical guess at what their

what their secret value is yeah i like you phrased it in your blog post a good example of don't

Repeat yourself causing problems.

Yeah, because the problem came from the filter reused the Django variable class.

Probably over time, variable gained more abilities.

And then that has led to the ability to index characters in a string and do the sorting.

Whereas it was never advertised for doing that, and probably no one needs that.

Well, I think that the more common example of this, right, is with URLs,

where why you would use a UUID.

So instead of going user slash one or, right, because as you said, it could apply to bank

records or something important.

You don't want someone to infer the order or the number.

And then number three, I can read it.

So there's a potential directory traversal via storage.save class, which, so that's a

file storage API thing.

Yeah.

So when you're uploading a file.

Well, I think it's just, it's interesting to talk about this because, you know, I think

like many, I see the reports that come out and I'm just like, yeah, it sounds good to

me and unless it's something i directly have don't get a chance to fully dive into it or even know

the process behind it so yeah and these these file driver these file directory traversal ones

there's been several of those over the throughout the you know the upload handlers and then and you

know blah blah blah and there was a number and this is kind of like the last one and you'd have

to use the storage directly which kind of nobody does so it's already been bulletproofed around

the outside but the idea would be that you know you go from the media file the media directory

where you're meant to be saving it to somewhere else on the file system which you know you're

not supposed to have access now you know if you do everything right that can't you know that's

already ruled out by various other bits in django but you know this is the sort of last bolt down

on that area um there might be a bit of a rewrite coming in the future who knows saying it's the

last possible directory traversal bug is very brave carton well no in that little bit it's the

last possible one in that that cluster of you know that that cluster of upload handlers and files and

we because we've gone over them no doubt that we won next week i think there's been like sufficient

eyeballs on it now um it's pretty pretty good this was again by um dennis brinkroff and sonosource so

i guess they were running some kind of scanner or just deciding to audit open source projects so

yeah they found it that was cool well something else just to shed light on all the things you do

adam just prs to django itself so 4.0 came out i guess carlton this was in your your week notes

thing where i saw where um i don't think it was on the release notes where you listed all the

active contributors and adam you were one of the largest ones we'll link to that it's not in the

release notes right i don't think the formal no no not yet i know that's no so i've so so one so

what I've got for 4.0 is a breakdown of new contributors of people who like I've called

them on a roll people contributed in the last in 3.2 as well and then people who didn't contribute

in 3.2 but had previously and could put those in the category welcome back and what I want to do

is write that up and then put that somewhere either on my site or on jangaproject.com if we

can find a suitable place for it and then keep that rolling for 4.1 and as we go forward that

would have happened already if it weren't for you know 2020 or the usual excuses um

but that that's coming and the idea there is just to have a bit of fun calling out contributors and

you know particularly for new contributors you make a contribution if can we find a way of just

flagging up and you know i it was it was on my list to get done for the end of the year and that

just didn't happen so okay over the next month or so i'll pull it up and you know finish finish

that off and have that going forward but that that the idea there is to do something to call

out contributors a little more than we have you know um you know i don't know it should it's just

a bit of fun it's just i think it's important though i think it's important to highlight those

things you know i think it's important because of exactly this problem of diversity we've already

talked about just just now like it's all right if you're sort of economically privileged and

you can contribute loads of time to open source and you can spend the time to get a contributor

record and you know but if a lot of people they need some recognition for it you know so I've made

one contribution did that did that pay back to me well no it didn't because it just disappeared into

the void whereas if we can put it on a website somewhere and then they can point to it it adds

a little you know for for someone getting into tech it adds a a validation point that might help

and get a job and then oh you know i can contribute you know i can continue we need to to to do things

to smooth the pathway there or we're never going to change the situation we're in it's always going

to be a slow break we're not going to you know instantly have a super diverse contributor base

but if we don't do these things we're never going to have that contributor base well part of it adam

i mean because even just your commit your um commits to django there's a wide range of how

how big they are right like i think you tweeted there was one a template like it's like a one

line thing with you and marius you know and then you have you know signal receiver function stuff

you know so there's i think it's great that you highlight that because that's something where it

seems small but like that's exactly a onboarding way versus going into like a core piece of django

and thinking that's what it takes to do a pr right yeah this this one line change i just was reading

the docs i found that there's a class that exists but the setting that you'd put that class in

didn't document it it said here's the list of built-in and listed two of the three so it's

just adding the third one on that list and and i think there are like a number of these small

things out there you might spot when reading the docs and it's great to just go make that change

and so it helps everyone else in the future yeah it's also i wanted to ask you i mean i know you're

a consultant and so you poke around a lot of code bases like how do you find so many things i mean

because i guess i'm just not using django to the full bore the way you are but i mean the range of

PRs, you have SQLite, signals, templates. I mean, is it client work? Is it a docs mismatch? Or is

there any pattern there? Because you find so many. Yeah, I guess I just end up going through a bunch

of different stuff, trying to solve people's problems they bring to me. I also enjoy reading

the code. And I'll sometimes take that as first reference, even skipping the docs when I'm like,

oh what's that method on on the storage for example i have django open in a sublime text

window all the time um yeah i guess the third thing is i'm always just uh finding it fun and

expanding my knowledge and my curiosity by diving into things i'm not really not not really looked

at before so like sqlite i was just interested in sqlite a bit more because they came out with

That new release, SQLite has strict tables now.

I was like, oh, I wonder how good Django's SQLite support really is

and could you get that strict thing in there?

And when I was reading the database back end,

I came up with that optimization PR.

I was like, oh, this seems fun.

I could do it.

I love that.

And Nick Pope was very good at reviewing and helping me out there.

So while we're talking about the ORM and contributing,

So you're, you're one of my sort of go-to ORM knowledgeable people, you know, that I can flag up on a PR. Hey, Adam, can you look at this? You know, obviously you did all the work on Django MySQL and, you know, you've been contributing there for a long time. One of the questions we always have from prospective new contributors is how do I get going contributing to the ORM? So can I put that question to you? What might you, what would be your kind of initial thoughts if somebody's coming, you know, somebody wants to get involved and they want to get involved in the ORM specifically?

That's a hard one, isn't it?

I think the problem with the ORM is that it does 99.9% of all the queries people need.

But most developers probably only know how to write 99% of those.

So there's this kind of ton of extra features that are hard to discover.

And some people might be tempted to just try and help, but it's already there in some way.

Yeah, I don't know if there's an easy path into the ORM.

I know there's like a James Bennett three-hour talk at a DjangoCon that is like a deep dive into the ORM.

And it's a few years ago, but the fundamentals won't have changed.

If you can sit through that, you're definitely going to be a good ORM contributor.

I've tried three times.

I still really want to.

It's no slag on him.

It's just it gets deep fast.

Yeah.

Yeah.

Yeah, it's a bit of a trial by five.

Maybe that's a new year's resolution for me.

Okay, cut one slightly different angle then.

You said there are lots of features that are kind of hard to discover.

So, like, you know, do you think there's a bit of a doc shortage in terms of learning more advanced SQL features or SQL features

and how to use them with the orm like annotations and aggregations and then the expressions api and

you know all those you know output field types and blah blah all that all that stuff which is

it's documented but there's no real easy learning path for that i would think yeah definitely um

one one thing i i've liked and tried to promote a bit is check constraints i've done a number of

posts on that if you read the check constraint docs in django they are like purely a reference

they're not going to explain to you what a check constraint really is and how it's defined on the

database um and we don't really have like a topic guide there so if someone was looking for a topic

guide that'd be to contribute that'd be a good place to start and perhaps that's a good first

contribution um it's like in migrations we have the reference for all the different operations and um

the topic guide of how to do X in migrations

is actually where you can learn how to really use that stuff.

Like, that has a few topics now, but could definitely

be expanded to, like, how do you add an index concurrently?

Like, we have that now for Postgres

in Django Contra Postgres, but there's definitely not

a topic guide.

Like, here's what you should be doing to use it.

So there's all these features, but, yeah.

MARK MANDELMAN- Carlton, it

seems I have a lot of questions.

don't want to overrun you do you no no okay i wanted to ask so so your new book um was it boost

your django developer experience is that that's the title yep you've used your django dx or

developer experience so you've um i've seen you i don't know if it's on twitter your blog post

mentioned like book driven development because so could you talk a bit about that because you've

put out i mean we have a long list in the notes so many projects posts um but as you've been

writing the book so i'm really interested in that and if you could expand on that topic yeah

absolutely so the the book is um just trying to describe and like categorize all the tools

that aren't django itself but like live around it that can really help you like accelerate your

development and and i guess book-driven development is where i start writing a section i'm like okay

i think i know the best way that i've seen to do this and i get halfway i'm like actually you know

that i'm just describing some snippets that you could copy paste instead it would be better if

there was a package out there that would would combine all this for you um so like a good example

there is a django rich a package i just created to integrate will mcgougan's rich that does nice

terminal output with django and currently it only provides like a new management command

yes calvin is there a is there a test runner coming for that because a while ago there was a

a PR to get colored output into the Django test runner,

and it didn't make it because of maintainability concerns,

but perhaps a third-party package wrapping rich as a test runner,

that'd be a nice thing to see.

It would be a nice thing to see.

I did briefly look into it and was like,

maybe the package could do this,

but a unit test is really the blocker there

because you've got to override this class and that class,

Django's test results and its debug results

inherit from the text results.

There's like a chain of like four classes that refer to each other in subclass and it would be a pain.

Okay, interesting.

I think maybe an approach would be if there was a unit tests extension package that used rich, then Django could inherit off that.

Okay.

If it was there.

The rich package only provides a management command that gives you the way to wrap Django's output with rich and then turn it off if you're piping to another terminal command.

And what about this Django browser reload?

I mean, that was one that I know came out and I guess asking the two of you, is that

a future core Django thing or are there just clear limitations on that?

Because that's pretty helpful.

Sure.

Briefly, Django browser reload is a tool that you install and then in development, whenever

you hit save on a template or save on a Python file that causes the server to restart or

a static asset changes, then it will hit reload on the browser for you automatically.

It does this with a different approach to what I've seen before, in that it sets up a very small JavaScript-based listener in the browser that's listening for a stream of events from a Django view that captures them.

And then that JavaScript worker, it hits reload on the most recently opened tab that you have for your development server.

So if you've opened five of them, it's only going to reload one.

I was inspired to do this because I was writing about DX.

I knew that this is a big problem.

People have this in JavaScript.

They have even the hot replacement where the page doesn't reload.

It just swaps in one section.

But there's not really been something I've seen in Django that works amazingly.

I came across one package, I think, for FastAPI,

and this was using Selenium to control it.

I was like, there's got to be a simpler way.

I don't want to force everyone to install Selenium

and use a Selenium development browser to do this.

there's got to be a way in javascript and yeah it didn't take long to fumble through the javascript

apis and come up with something very basic that's great no that's really interesting that

that is a common problem and that's an elegant approach which i guess time will tell but

i don't know why it wouldn't work it would be cool to have it in cool that that's the

interesting question is what makes it in what doesn't and you know i mean uh yeah i i would

like to know in what situations it doesn't work before we really go ahead and consider it it's

the sort of thing that's suitably useful that there's a case to be made right well i suppose

i mean there's almost i don't know if it's a conflict of interest but if it's your own package

and you're on the technical board and the security team it's not really on you to propose that it be

integrated but i guess when it's a third-party package you know people can use it on their own

and if there are issues they'll come to the fore so so many times we have like um the the standard

argument the standard line the standard approach is can we get this feature into janga well can

you put it into a third-party package and can we see how that goes for you know and quite a long

time i mean you know janga rest framework is 15 years 10 years out there you know it's it can live

a third-party package forever um and so what's the case for bringing it in you know that's

you know jammer t-bug toolbar that would be a you know candidate but that's never going to get

merged into core um you know these kind of hyper useful pack third-party packages they will they

can just stay as third-party packages i don't know it's but we can't just go right new feature

bam can we can i ask some more about about the book i mean sort of you've i think you put well

I've had a chance to look at it, as has Carlton, but topics covered.

Because this is your second book, you also have your Speed Up Your Django Tests,

and the book will be released a couple of days after this comes out.

So we'll have a link.

Can they still pre-order it up until the release?

You can pre-order it right up until the hour it's released.

It will be released at 12 noon GMT.

That's the same as UTC right now, on the 10th of January.

Okay, well, this will come out.

We're recording on the 4th.

It will come out on the 5th.

So we'll have a big prominent link to that.

But anyways, yeah, to the book.

So what was, you know, because I find the question is always what not to include, right?

Because the first draft is like, let me just dump everything in.

And then it's like, what to strip out.

And then it's like, okay, and is this actually what I want to say?

As you were, you know, with these areas, you start to give it a beginner's look with some experience.

And then sometimes you come up with different approaches.

Yeah, so I tried running one chapter for each kind of like broad area of the development of a Django application.

So it starts off with like looking at docs and then how you make virtual environments and manage dependencies.

And it goes on to other things like later it's things that are actually in your code, like your settings and models and migrations.

So, yeah, it's about everything that's like alongside the actual writing of Django code.

There's not much like, here's how to build an X in Django,

but it's here's how to make it faster for you to build X in Django.

So faster and more bug-free.

You're very right about cutting things.

There's so many things that are actually involved in the process.

One thing that's helped me a lot is I've added a dropped file in the repo

with all the topics that I've dropped in bullet point.

That could have been like a whole chapter on Git,

a whole chapter on GitHub, a whole chapter on text editing.

Yeah, I have all these ghost chapters too.

Yeah, I have to ignore it.

Well, yeah, it's a combination of, yeah, it's like, oh, this is hard or like the deadline.

And you're like, yeah, if it's important, I'll, I mean, I had a whole huge thing on logging for my professionals book.

And I'd still like to put it in there, but I was also like, I just got to ship it.

Logging could be its own book.

Well, right.

Similarly, like I, in this book, I say, hey, you could go install MyPi.

and then it's like i can't cover type hints here there's no real way of even starting so

here's just some links to resources i mean there's a question because you did this whole series of

posts on on typing i did wonder if that was kind of you know drop blogging the first draft of there

was going to be a typing book come out of that but i've considered i i wrote those um for my

own education as well as to like help push typing a bit because it should be possible i think to like

search how do i do type hints for this python feature and find a reasonable explanation so i

just try to cover many of these spaces and i think a book would just be like a bundle of these posts

that you could go read like a typing cookbook maybe that's the title well i love that you're

doing these you know not just advanced but um related to django but not django itself

approach. I feel like that's maybe an approach I should have done, because I struggle more with

my more advanced books than the beginner one. Because the beginner one, I feel a little more

confident saying, I don't need to dive into everything. Just sort of trust me on this,

or here's how you can go deeper. But the more advanced ones, it's still kind of the same thing,

but it's harder to do. I feel a little worse about waving my hands, or maybe I'm just not as

clear on it and i'd have to say you know okay here's three ways to do it i like this one but

i don't feel the same level of confidence on that as i would on the beginner thing where it's like

we're just gonna backfill a little bit here um sorry so i'm incredibly impressed that you're i

mean of course you have the background to write these books but i feel in some ways they're harder

to write but but taking a more from the side you know not just doing like the most badass

Django project ever approach like that's that's a recipe for failure that's like building a Django

puzzle which is what I have in my professionals book so don't do that well I'm trying to fill in

the gaps of of what what materials out there um it's very easy to say to someone hey go set up

pre-commit and then like the pre-commit docs don't say here's how to do pre-commit on a python project

with flaky iso and black and whereas that that's that's what most people want that's what Django

does um and yeah there could be some opinion um i do respect your book's will and i think the fact

that you're a bit uh uncertain about what's going into the professional's book is just an indication

of it being senior right at senior you have to make trade-offs yeah yeah it all it all depends

that'll be my my fourth book right carlton we'll just we'll list all the unknowables i um

i think your point about pre-commit is a good one like because you go to the pre-commit docs

you're like okay it take it can take all day to get it set up properly whereas you can you give

the recipe look here bam this is how you get pre-commit going in your django problem your

django project quickly so one thing i wanted to ask you about because you seem to manage so many

open source projects and um part of that surely it must be tell me is that you've got this tooling

kind of lined up to automate the repetitive parts of managing a package is that right

absolutely um i use a tool called my repos um which i guess is a bit it's a bit weird because

it's written in pearl and it's not always so easy to use um but it it lets you like run many run

commands in multiple directories and it's a git aware so it has like a status command if you need

that. So I kind of treat all my open source packages as one large repo with like some

variability in each of them if necessary, but as little as possible. So if there's a new Python

version, what I'll do is actually write a script that will make the appropriate changes in the

files using SD, that's a Rust-based replacement for sed. And so I just run my repos and run that

script and do the

and all those files so as long as everything stays relatively uniform i can do i can upgrade

you know 33 packages i think it is now in the same time that many people could do one or two

so it's like boost your github dx yeah i guess

that's it that's a great question i was wondering the same thing i'm i'm trying to like nuke open

source repositories i have because i just can't deal with maintaining them and and so i i love

seeing that you're going the other way um i mean your output honestly it just makes me tired but

i think i'm just tired of little kids but i'm just like oh my god like every you know because

your weekly posts which people should sign up for it's like a couple packages couple blog posts

couple core commits you know plus everything else i do it's awesome it's it's honestly it

It gives me energy, though, to see that you can be so prolific with this stuff.

I'm glad to hear, yeah.

I just wanted to ask you about one more package, which is Django Upgrade.

Can you tell us a little bit about that?

Oh, yeah, yeah, yeah.

That's very exciting.

Yeah.

Okay, yeah.

That's worth mentioning.

So I wrote this between August and September.

It's a code rewriter that will upgrade your Django syntax,

and it knows about some changes as far back as 1.10.

um because they were there um bruno alla wrote a version of this before called django code mod

and this used um instagram's libcst which is their concrete syntax tree library

and there are a couple problems with that underlying library though first it's like

incredibly slow because it's a python parser written in python and it also like turns features

on and off depending upon python versions so it's like got all these branches um and and then the

second would be that they don't really update it for new python versions instagram is still running

their own fork of 3.8 as i understand so there's 3.9 and 3.10 syntax that's just not supported

um so the end result is that django code mod is like okay for most people but it's not something

you can add in the way that i like it which is you know it runs through on your pre-commit hook

so you never can commit old syntax it will just get rewritten when you try

so Django Upgraders is like a rewrite of Django CodeMod but it's using the same approach that

a tool called PyUpgrade uses and PyUpgrade just uses the built-in Python syntax tree and tokenizer

to figure out what changes to make Django Upgrade does the same thing so if you're running it on

python 310 it automatically supports all the features in 310 because the python standard

library will support that little play with that ast stuff it's it's quite sort of in depth it

takes it's quite slow going i would i'd say it was very entertaining to learn a bit more about it

and and especially the especially the going back from the ast to the tokens because when you're at

token level you're basically manipulating like this thing all we know about it is it looks like

a name we don't know what it relates to in this in the actual source tree is it a variable name or a

class name and assignment so yeah you're fiddling around with those tokens it's quite fun okay so i

have i have great hopes for django upgrade in that one of the big not problems one of the great

strengths of django is the stability um policy for the the api stability policy then we don't

break stuff unless you know there's a good reason to do so and i was asking on twitter today after

i was like you know because i was releasing 2.2 um as well as 3.2 and 4.0 and i was like who's

still on 2.2 like well you know because i don't use the lts myself um and i tweeted out and it's

a you know agency a couple of big agencies so dab apps um jamie replied and then um tobias from

cactus replied um saying you know well we use it because our clients it's very hard to justify the

eight monthly upgrade whereas a two yearly upgrade we can we can do that and um other other replies

were like look it's stability we use the lts for stability um and one of the big challenges is oh

we want to deprecate this um but people won't be able to upgrade and you know which you know when

we changed i don't know to using um you had to import the view rather than just use the string

view name i think you know actually we lost instagram over that change you know they weren't

prepared to go through and so they fork Django and you know they're forever on a fork Django

now rather than on mainstream Django and maybe they'd have gone anyway but could we avoid that

and well one one hope of avoiding that kind of breaking change is automatic script so when we

do introduce a deprecation there's also a matching script that will go through your code base and

correctly update it is that is that is that a pipe dream too far or is that is that realistic hope

that we might have well i definitely share this hope um instead is one thing to like deprecate

and add a warning and then that warning just shows up inside in your test suite it's like

here's a thousand things to go change that's no fun um so yeah there's definitely a place for this

um i'd be excited to see it going forwards i couldn't spot anything in 4.0 that can actually

be automatically rewritten um without huge amounts of effort so definitely a balancing act

there's plenty of changes in 4.0 that we can't automate with it so here's a tester for you for

4.1 like um we brought in them the uh zone info time zone changes for django 4.0 and there's this

django.utils.timezone.utc constant which is just uh you know it we should get rid of that ideally

but that's everywhere and so is it worth the check we could just leave it there forever

but if we are going to get rid of it we need to get rid of it for you know before 5.0 because it

needs to go at the same time as pi tz so you know could we could we could we use pi upgrade to

to make a deprecate that remove it but be make removing it you know just run a script that would

be a cool thing that that one's definitely easy to change from the ast's perspective because you

can see it being imported and you can be like okay what do we replace it with and i guess we

just replace it with the the one in python's datetime module right yeah so exactly that it's

just replace it with datetime datetime.ugc but like it would be it would be nice because my only

reticence about removing it is that it's everywhere you know everywhere in the ecosystem people have

been using that so if there was a scripts that came along with with django 5.0 that said hey

you've got to remove this but just run this and it's done that would be an amazing um you know

proof of concept of of that this is the future of upgrading django because

stability right that's that's the usb yep definitely and there's there's a good argument

to remove it right because it looks like it's a different thing and once it's just an alias for

python's one all you've got is like the potential for confusion yeah so why are we keeping it we're

keeping it to avoid the churn but that that churn can that that confusion can stay there forever

and tutorials and people's code-based snippets that get copy-pasted and oh i should be using

django's one oh it's exactly the same it's just an alias yeah yeah exactly okay one one thing i

found in django upgrade is some historical changes where there were aliases for python

building functions that have been rewritten um and they weren't ever documented but when they

were removed i found the old commits and people were debating should we really be doing this

yeah okay yeah we're definitely better off that we did now so okay interesting so we should push

that for 5.0 then it's been decided here on the podcast it's a quorum yeah so the book's coming

out at least for me there's always it's like it's not like a downer after something comes out but

there's sort of like a loss of purpose at the same time there's there's like a oh thank god like

i can worry about something else so long term what else do you do you know what you know to

work on projects books like pie in the sky um things you have planned around django um well

i've i've taken a break from client work since july last year so i'll be getting back into that

and no doubt diving through people's code bases i'll spot some new things that i'm interested in

in changing yeah um for the book uh my next writing project will be to go update the speed

up your django test one for 4.0 yeah i got you on 4.0 you got me on 3.2 but i got you on 4.0

for one of my books i'm still still waiting on the other one but yeah when i pushed that

well i double checked i was like i beat adam in something

congrats it's not a it's not a big change it's not a big change it's a pretty small change

or at least i i think i think for yours as well right it's not i don't think

try to remember i i'm lucky that my books aren't like go run exactly these things and it will

definitely work for you they're more like hey this is something that could work oh my god it's you

know it's it's like it's it's a test of something because i you know get emails they're like there's

an error and i'm like are you using the right code are you did you pin your repositories like

you know i have 99.9 sure you're making a mistake but i just like sometimes i have to just like

wait a couple days and be like you sure you want to check you know um so it's good and bad it's

like i'm pretty sure i'm right on this um because i've put it in this little box um but anyways

yeah yeah you don't have those it's a little i envy the like yeah like you could do this like

it might work for you um so when i go through my updates yeah yeah like i literally i've thought

of running a script to like check everything but i also for now manually go through everything

um as much for like the text part because i you know i just kind of whinge when i read

old things i've written and i'm like oh i could rephrase that or improve it so it's like a forced

update mechanism i don't know if that's what you do if you go through and read the text or you

you probably have a test suite you can run and just see what's broken uh you think too much for

me i i definitely go through and read things um in in speed up your django tests at least because

some features were still under development i sign posted hey you know django 4.0 you'll be able to

do this so there would just be a matter of updating it and checking that what i wrote there is still

applicable yeah i do that too yeah i try to say i think this is coming um but i mean also i think

You know, it's good to have some distance from it, right?

I mean, eight months is hard, which this is the first, last year was the first time I missed an update cycle.

But it's almost too close because you sort of need some distance.

And yeah, and things change to like revisit the same stuff.

Anyways, yeah, that makes sense.

I wish you well on the update challenge.

It's like a band playing their greatest hits is how I look at it.

You know, it's like some for me, like new stuff, and then some for everyone else, which is updating things I've already written.

Greatest hits, I like that analogy.

yeah it's like you know if nobody cared i wouldn't do it so anyways not to make this

about writing books carlton i know you love those discussions it's just i i like them

because they remind me not to do it anyways we should um we should wrap up okay let's wrap up

there adam thank you so much for coming on um good job with the book launch everybody remember

you can pre-order until the 10th um boost your django dx 10 discount well there you are um thanks

for coming on and thanks for everything you do to the django for the django community and well done

again on the mancom dredelic ward couldn't think of someone who deserves it more thank you very

much for having me all right everyone we will are back on our every two weeks schedule with new

episodes on django chat.com and chat django on twitter and we'll see you all next time bye bye