Transcript: Django’s Rough Edges - Keryn Knight

Hi, welcome to another episode of Django Chats, a fortnightly podcast on the Django webframe.

I'm Carlton Gibson, joined us by Will Vincent. Hello, Will.

Hi, Carlton.

Hello, Will. And today we've got Kez Knight with us, who's a longtime contributor to Django,

member of the Django triage and review team and, you know, general member of the community.

Hello, Kez. Thanks for coming on.

Hello. Thank you very much for having me. Appreciate it.

No, it's a real honour. I wanted to have you on for a while.

Let's start. We always kick off with your backstory.

So tell us about you and how you got into programming and how you found Django.

I got into programming accidentally by, when I was a teenager,

we kind of was looking at doing various things on the web

and we kind of learned a bit about it during school.

And so I learned HTML and then nascent CSS stuff that was happening.

And accidentally formed a career around that, managed to get my first job doing that kind of thing.

That led to more kind of full stack work where, you know, be involved in the back end and a lot of the front end.

Back when the front end was much simpler than it is today, I must stress that.

And started out doing like PHP back end stuff.

And ultimately, in around 2010, sat down to try and make something more maintainable when we were working in a web development agency.

Obviously, there's a lot of juggling of projects there.

We were looking for something that kind of fit the bill and was easier to onboard and revisit later.

and we revisited a whole bunch of different frameworks, cross-languages,

and ultimately settled on Django as having the least cons of all of them.

And I've been using it ever since.

Okay, and so in an agency environment, is that primarily where you were?

Yes, yeah, we were an agency for a number of years,

and then we pivoted towards doing a specific product later in the time.

Okay, and the key bit for the agency stuff

is you've got lots of projects on the go,

so you're always cutting and coming back and coming,

but they've got to be the same in some sense

because you can't relearn something bespoke

for every single project.

Yeah, I mean, a lot of our work was very bespoke.

Obviously, everyone supplements these things

with your relatively standard CMS or CRM kind of work

because that's reasonably quick to turn around

if it isn't too complex.

But you had things like WordPress kind of taking control of those kind of markets, certainly for the CMS stuff anyway.

So that meant that we moved more into kind of bespoke aspects, whether that was tailoring a whole back office system or building a REST API on top of some complex job system.

We really kind of ended up running the gamut in terms of the range of things we would take on.

Some more successfully than others, inevitably, but it did become very varied, I would say.

Okay, okay. And so I wanted to ask you then how you got into contributing to Jenga,

because you've, you know, you've been, you're just very active and you come in on particular issues

and make very insightful comments. And as I look through the issue history, you've been doing that for a long time.

Yeah, I can't remember when my first ticket got opened.

It was probably around 2013, 2014 when I actually found something that might be considered an issue.

The longer I've worked with it, obviously I've become more familiar with the kinds of things that are likely to be considered bugs

and the kinds of things which are likely to be, no, this is actually a problem in user land.

I would say that, you know, one of the things is to just keep trying.

Fully 50% of my tickets over my career must have been closed as won't fix because they were some misunderstanding I had or something that had been retread elsewhere in a previous ticket or something.

But it's a case of just continuing to kind of try your hand at it.

It's not a difficult thing particularly.

And the most that really happens is someone closes it as inevitably not relevant.

That's the worst that happens.

The best that happens is you either get a new feature that you're really after

or you fix a bug that affects you and a bunch of other people.

And I think how I ended up starting that contribution process

was really just a couple of things i encountered that i thought were errors and they weren't

they were my errors or they were a third party library's errors but it begins that process of

actually talking with the community about how things progress and whether it's right that

they're not in call or are in call and it's you know it's a potluck for me sometimes as to whether

it's uh it's going to get accepted and that's always interesting if nothing else because you

can't have everything in but much as you'd like to and there's obviously whole cohorts within the

community that have divergent ideas about what should be in there and what shouldn't be in there

and they can become quite polarizing so straddling that middle ground of

what makes the cut is is always an interesting thing to try and establish yeah okay

one thing that you've um quite often done i've seen in recent discussions which has been

you know super handy is i don't know we'll be looking at adjusting the settings object or

something and you'll come in with uh well you know but if you use it in this particular way which

for me it's sometimes quite a left field way like you know to manually be configuring so

calling settings configure with manual see there's something that you know i do but not

very often and yet you're quite frequently there with ah you know if you configure a script this

particular way it you know this happens so yeah i think that's just a case of having been around

the block so to speak for so long that i've encountered an awful lot of not uncommon but

less frequent use cases and tried so many various things.

You know, there's whole aspects of Django and indeed Python

that I'm not as well versed in as I might hope to be.

The ORM has come on leaps and bounds over the years

and there's bits of that now that I've never used because of that.

But there's also other things where I have used settings.configure

to do something outside of a normal management command and i've had to become aware of the

foibles of doing so and because that's the less common approach it's more valuable for me to

step in in those kind of things where it hasn't necessarily been considered because it's off the

beaten path if it's if it's the if it's the common 80 use case then everyone's already going to be

thinking of those things anyway so there's there's not much i'm going to say that's

interesting or poignant towards it yeah fair enough fair enough i want to ask an open-ended

question which is so you've been working with django for coming on 10 years what how has it

changed i mean just big picture because a lot of people are new to django and they don't have that

history you know just what have you seen that you like and dislike in terms of the change in that

time i would say i came in just after the magic removal branch which i think was 0.96 to 1

something like that so i've already missed the boat on probably the biggest change in

django's history um i think some of the kind of bigger ticket items that have really

helped the overall community have been things like the bringing into core of a migrations module

which was previously south and the new migrations module is more flexible and also more robust and

that's certainly not to do disservice of of south we all used it prior to the inbuilt migrations

and the splitting out of like the template template backends so that you can use ginger or

mako or whatever else you'd like to use i think that was a that was a great step along with

the introduction of app configs i think by and large those aren't necessarily used as much as

people might have expected yeah not much at all i mean for certainly for beginners

it's like because i cover it in my book and i'm like you can do all these things but

You know, you won't necessarily, like, I don't see it used a lot at the, you know, more beginner level that I teach.

Yeah, and I think there's some reasons internal to Django that that's the case, and some reasons dependent on them being an external library that doesn't get the same focus in the documentation as the built-in Django template language, for example.

And of course, Ginger specifically being so similar in look and feel to Django, people don't necessarily know why they might want to reach for that.

And the answer to that is usually just a sheer performance one because it compiles bytecode differently and magically and I don't understand it, but it's cool.

But I think the app config one specifically didn't happen in the way anyone foresaw,

because there was a long time where the concept of a default app config meant no one had to adapt.

And I think the overall adaptation has now happened in Django itself to force that and finally deprecate that.

So I think we may now see more dependence on things like an app config that wasn't there before.

But that also comes with its own slight tricky intricacies.

But those are the big ticket items I can think of in terms of what's actually kind of progressed.

And I don't mean incrementally in terms of like, as I say, the kind of modeling of database stuff.

that's improved leaps and bounds uh you know the introduction of subquery syntax to actually

manually build those was a pretty revolutionary thing because it allowed a lot more flexibility

in what you could do that wasn't magically done behind the scenes for you but i don't know

i don't know enough about those because they keep improving incrementally and i might be on the last

long-term support version, I don't necessarily see,

oh, okay, I can use this for that until I get around to the next one.

So there's bits I'm less familiar with.

I think the whole expressions API stuff,

it's just phenomenally powerful and the constraints that have come in.

Yes, yeah.

And it's like, ah, the ORM has become like, oh, wow,

it's very sophisticated compared to what it is.

And it's like, it's kind of how to learn these things because the docs can be a bit terse.

Yes, I think because it's become more powerful, I think it has also become harder to learn.

And that's, I think, true of Python in general.

But the more we can focus on what these improvements actually bring to the table, the more the users can get their head around it.

But yes, the Expressions API is exceptionally powerful

and I probably don't make as much use of it as I'd like to

because I haven't fully thought through how to do it in that way

because I have that long history of using it.

I'm used to doing it a certain way to work around the fact it wasn't there

and now I can do it, but I'm not thinking about it

because it's too new, so to speak, in terms of my overall trajectory using it.

Isn't that the curse of learning more?

absolutely it is yeah it's uh in in many ways it's easier for a new learner to come on board and get

used to the modern way of doing these things because by the time they are starting it the

incremental journey has already gone so far but it also means that yeah you can end up not out

of date necessarily but stuck in your ways there's nothing wrong with being stuck in your ways that's

we defend that we defend that vigorously on this show well i'm not oh sorry carlton you guys well

i was just gonna segue off um you mentioned that the new features of python so um while we while

we're talking about new features of django and don't change what do you what's your sort of take

on the way python's evolved and i think it has become a lot more complex especially over the

3.0 to now release cycle where there's a lot of additional syntax there's a lot of additional

things that people are looking to add into the code base that makes sense from a maintenance

perspective which is always a great thing to have but doesn't necessarily make sense in that first

kind of almost prototypal phase where you're first building something out and i'm thinking

especially of type annotations they've become very popular recently through things like

mypy and there's a couple of others whose names i've forgotten that do type checking

those add a huge amount of verbiage to the average python project for the safety of

making sure it's doing the right thing but that's that's exceptionally difficult to

reason about for a new learner and there's also things like the walrus operator turning up

which i've used it more than i thought i would which is a good use it in django the other day

for the first time it's great it's really good um you know there's i've there's use cases for it

that i didn't think i'd have but by the same token it is the introduction of another type of

declaring a variable and now a new user has to learn that so i think python's in a tricky

position overall in that it's trying to adapt and become more modern but it may ultimately be

leaving new users in the dust as a result i think it was david beezy had this line that python fits

in your brain maybe it's not his line maybe it's you know just but one of the great things it fits

the right size for your head and you can hold it all yeah it certainly was i don't know if it still

is uh especially with the kind of introduction of the whole async io ecosystem and i think that's

somewhat fractured, which is always going to be a difficult

one to deal with. But intrinsically through the kind

of 2.0 to 3.0 release cycle, you could hold it in your head.

and that became slightly truer again

when 3.3 came around

and we had things like Unicode literals again.

But it is still expanding

and there may come an inflection point

where you've alienated your new audience

and you're then left with power users,

I suppose we would be,

but who know all of that syntax

and have internalized all of those things.

It's a difficult boundary to straddle.

Well, Django is, I mean, specifically on Async, you know, Django is having it basically be like a feature flag where it's there, but it's not a default and you can turn it on and off, which I guess for beginners would be nice if Python had that.

But I also see why for maintenance and power users who dominate the discussion, they'd want to just have some of these things in there and not be a separate garden within Python.

Yeah, I think ultimately it would have been nice if we could instantly just flip a switch

and become async IO all the way down the line, because that would have made everyone's lives

so much easier from, you know, the Django fellows, the people implementing these things

all the way through the pipeline and the actual users in terms of talking about the deployment

strategy and the deployment strategy for python has always been a difficult one compared to say

php or go where you're just compiling a static binary and that's always been one of python's

problems and i think that hasn't changed particularly it's got better in some ways

it's got worse in some ways now we talk about the async io situation and we talk about red and blue

functions and how those affect the overall ecosystem. And ultimately, it is still a

transitionary period for Django. I know at some point in the future, it will be entirely async IO

all the way if you want it. And at some point in the future, it will become that by default.

But it's still a process we're going through in terms of getting that adaptation all the way through the whole chain, which is, you know, substantial.

It's a full stack framework for the back end.

So there's a lot of moving parts and there's a lot of things that can change at runtime, which makes it slightly complicated to even figure out if it is synchronous or asynchronous.

Do you agree with that, Carlton? It's somewhere down the line.

Oh, well, the end point.

That was the bit.

Yeah, maybe.

I mean, in principle, why not?

Because in principle, it should be slightly less resource intensive

to use asynchronous IO.

There's a context switching cost.

But, you know, if we can make it async all the way down,

then you should be able to declare a sync view at the top if you want to.

I mean, why not asynchronous code?

It's because it's more difficult to write,

it's more difficult to reason about so 90 of view code that you write should you should just be

using sync django like you've always been using because that's going to be the simplest to write

the simplest to maintain and it's going to be as performance as you need it then why do we need

async an asynchronous story well django's you know full stack web framework it's there to to have all

the batteries and you don't want to change web framework just because you need high throughput

for you know a particular endpoint or you you want to make some parallel api requests or something i

don't know what you know but we need a story there do we need to be the fastest web framework in the

block you know no we don't um so i don't know i don't know where it'll end up i do see though

that you know for instance to to get we've got an async view dispatch um thing in place now but we

still have to um kind of wrap the synchronous signals that you know request started request

finished those still synchronous so we have to wrap those in a sync to async um wrapper and that

goes out to a thread and okay there's a slight context cost there um could can we get rid of

that yeah there's a pr to get rid of that but there's a performance overhead it's really complex

and it's like okay and it takes a long time to sit down with these prs and really work through

exactly what's going on and exactly why it's slower or exactly why it's quicker or exactly

what the error conditions are. And so it's going to take a little while. I'll agree with that bit.

Well, what you're saying about the evolution of Python, that's pretty interesting to me. I mean,

I studied history in college, so I always like to look at past examples. I mean,

there is a life cycle to programming languages and the fact that Python was a teaching language and

then kind of got adopted early by Google. And in many ways, the data science people

made the shift from two to three happen because the website, that was kind of a mess, but we had

all these new people come in. But that does seem a fundamental tension for a language, right? It's

like who drives the use? It's power users. You have more, I think, background than I do. I'm

just trying to think if there's examples of other languages that have gone too far in the power user

side and lost people i mean i think at php php has gotten a lot better um over time i would i would

say but i'm sure yeah i would i would think i left just before the php kind of grew up um you know i

left before it had a package manager and composer i left before de facto standard frameworks came

out in terms of like laravel and symphony yeah so they kind of revitalized its uh kind of

availability in terms of doing big back-end systems i would say the php always had the

benefit of being exceptionally easy to write and easy to run and it mixed concerns within

one file often but that was fantastic because it allowed people to actually see their results

it grew up with the advent of php 5 through 7 it developed all of these extra features

taking them from other languages and other frameworks and other ecosystems and i think it

very much is in a good place now it's you know it's reasonably performant it's got a great story

across any ecosystem niche you might want to fill these days i would think in terms of the kind of

language familiarity i have pearl would probably be the one that i

would think has sort of entered into the end of its lifetime

and that's not to say it's a bad language and it's not to say people

aren't still using it to do incredible things it's a great text processing

language but it's one of those things where it's

time in the sun has waxed and waned whilst python is

still kind of currently over the last 10 years very

much in that middle ground python has seen a huge increase because of things like the machine

learning and uh all of that kind of stuff that that i don't really understand i it's it's an

entire part of the ecosystem that is beyond my understanding but there's a lot of cool stuff

happening with like pytorch and tensorflow and people are using it as a as a glue language

and that enables the pickup where people don't necessarily have a full understanding of

the language they can still write the things they need to write i think you know in the in the long

term over the next 10 years we may find that some of that dissipates because those use cases will

be fit by a different language but we may also find that it just gets codified as python is the

language to do x y or z because it's been doing it for so long because it has a decent api for it

what have you it's it's still i think too early to say that python is on its way out in any fashion

i i wouldn't uh i wouldn't bet on that particularly but it's it's an interesting period and it's

become more interesting over the three point x release cycle with all the new stuff we've got

as a result of that time will tell you can still write it without all the you know the type

annotations and things you can absolutely you can bang out your 30 line pi script that does what

you want it to yeah i find i find i have a uh love hate relationship with type annotations

in that if i if i'm encountering some gnarly bit of code and it doesn't have them i really wish i

had them but i also don't necessarily want to write them because historically being nominal

typing they've seemed like the wrong kind of fit for what i'd like to do python is a duck

duck typing based language and more recently we've got structural typing in terms of protocols

and i think that that's a great addition but unfortunately it's still relatively recent so

a lot of the type annotations you see out there will be the old this is an integer this is a class

of x y or z where perhaps they could be more flexible and i think that's that's probably part

of the reason that django itself doesn't have type annotations if you look at the various stub

projects for it they are exquisitely detailed but exceptionally complex because of that duck

typing that means you can pass anything in that fits a certain structure yeah but even if you

just take to um so take the channels project there was a pr to add type hints onto there

and just taking the example of a user you know where a function takes a user object a django

user object like the type hint was horrific um because and because it can be an abstract user

an abstract base user you know it's just too many types and so you know you might create an alias

where you say a user is in fact this this horrible thing but it's like wow that's that really is quite

flexible in what's accepted in this function yeah and i think that's one of python's strengths

ultimately yeah yeah yeah well on um php i have i have a mate who runs an agency actually though

he just joined a company full-time but he's uses laravel and so he's always forwarding me

things about Laravel and there's huge advancements there. Sometimes I feel like the people I know in

the Django world, we're a little bit myopic in terms of those fast API, but non-Python language

frameworks like Laravel has a lot that's interesting. But the model where it's just

one person owns it and runs it so they can just make decisions is sort of this balancing act.

Sometimes it's nice. They can just do it and monetize and they don't have a Django Software

foundation board um but there's less of a community or it's a different type of community

so that's interesting for me to see he's constantly sending me all these features that

they can do and monetize and um sometimes i'm jealous that they can move faster but

i i also think about what you know what's the cost i mean because it's a one it's a one-man

operation ultimately still um and i wonder how sustainable that can be yeah i mean i think

it's it's a difference in that community aspect because absolutely and obviously i don't have a

huge amount of experience within you know the larryville community as i said it came much later

but having that strong personality at the top defining this is what we're going to do

this is what's coming next is a great thing and it's it allows you to push forward your state of

the art in terms of what you're producing quite quickly but you have to balance that with the

amount of churn that produces uh django is an exceptionally old framework by modern standards

it's been around like 15 16 years or something now at this point so there's there's a lot of

users and a lot of us are used to doing things in a certain way certainly there are things i wish

django had done differently and or i wish django would jump in and produce something that hits the

80 case that i see it's missing but it's more democratic in that it's all just ultimately

contributions from the community putting them into track and the mailing list and talking about

those things and we have the django enhancement proposals thing to make sure that if something

big is coming into core it's well specced out and i don't i don't know what laravel has in terms of

that but you can see that they are moving fast across a lot of different fronts and it's it's

an interestingly different community in that aspect and yeah absolutely there's parts of that

i'm super jealous of because they move fast but there's also a part of me that's like thank

goodness django doesn't move fast yes i might wish it had x y or z functionality or it had done

this differently but it's a very stable framework overall the number of times i've had real issues

changing things on an upgrade cycle are because of third-party dependencies most of the changes

internal to django are one they're in the release notes and well covered there but additionally they

are relatively minor because it's got that incremental path and it doesn't really subscribe

to semantic versioning so you just upgrade to the next one the biggest change is when you go from

long-term support to long-term support and even that's mostly just warnings so it's uh yeah i

would say i wish there were things django would do and faster but i entirely get why it doesn't

and certainly it hasn't stopped me using it over these last 10 years so hopefully great effect okay

the traditional sort of answer there though is to do things as a third party package you know as an

add-on do you think do you think there are limitations to that approach i think doing it

as a third party add-on is often the best case initially that's certainly how migration started

out that's how static files started out and there were various community provided ginger integrations

for example to cover off the things we talked about earlier i think the problem with relying

so much on third parties is one you're then beholden to an even smaller cohort of maintainers

because a lot of these third-party packages just one person won't have yeah there's just one person

absolutely um so so you become dependent on a smaller number of maintainers you become dependent

on a larger divergent set of maintainers who obviously have different priorities and they have

you know lives that they want to live independent of their open source contributions

And ultimately, whilst it does sometimes pave the way for things to come into Django proper,

it does also mean that Django itself doesn't have an answer for that.

And the biggest, you know, one of the biggest areas now served by Django itself

I would say is rest api's using rest framework previously we had things like tasty pie very

early on there was a couple of others but rest framework seems to have taken the mind share

overall and that's great because it means the community works entirely within that ecosystem

but it's still a disappointment that when you come to say to a new user let's use Django for this

then they ask about that story how to how to REST APIs work what does it have there

the answer is it doesn't have that I mean it has it absolutely has the kind of underlying primitives

but it doesn't have the full story so you have to reach out for that third package

and then you're into that kind of maintenance aspect that i talked about so as with everything

it's a it's a difficult balancing act and the notion of increasing what's in core i think has

been diminishing over time because core has had some things ejected out of it i think

there's a couple of little packages form tools i think was one that got made into a separate

package because it's not the most common use case but i think there are some things where

django is missing a story for the 80 in that 80 20 rule that it could be serving and

i don't think there's a good answer to that because it's a lot of extra burden to take on

for a very small kind of core set of people who are working on it day in day out and it's a lot

of extra burden but when you evaluate it against saying to people there's nothing built in for that

especially when the web has changed so much over these last 15 years when when tango first came

onto the scene it was revolutionary for the python community in the same way rails was for the ruby

community and i think in some aspects that's less true now and you see that in the competing

frameworks that are kind of capturing mindshare today like fast api or black sheep or what have

you they need a black sheep there's there's always another one yeah this is framework of the week

yeah absolutely and they are interesting and they serve a different purpose

to what the core of Django now provides.

Django now provides a template language

and always has,

but that's not even necessarily

making up the majority of use cases.

It may still be.

I don't have any data on it,

but certainly there's a lot of REST-driven APIs

for a given value of REST,

and those aren't going to be using

the kind of template language backend

anywhere near as much.

So it's always a hard thing to balance that.

And I don't have to sit in a position where I make the judgment, which is fantastic for me.

I mean, just on the REST framework type side, there's a couple of tickets, one to extract the request parsing into a kind of separate class.

We've got the multi-part parser, so form data comes in.

But to make that kind of pluggable so you could have a JSON parser in there.

And then we could have content negotiation.

And then at that point it becomes, okay, you could accept JSON requests and form encoded requests.

And, okay, that starts to look more RESTy.

And then, you know, the serializer story we don't quite have.

But, you know, there's no reason why that couldn't be brought in.

And then you've got 80% of what REST framework gives you with the, you know, the generic views.

And that's very much something I've been thinking about recently too, because I think, as you say, half the story is already there and the missing half is difficult to do in a backwards compatible way, but not insurmountable.

as you say the content negotiation the parsing of the request body these are all things that

we can make work and once you do even the forms api i think could probably be adapted successfully

because one thing people don't necessarily fully understand about the forms api in terms of

internally is it's a reasonably good API for validating a dictionary which is perfect because

most web requests these days are perhaps nested but they're just dictionaries of here's some data

now the forms API itself doesn't currently allow you to have nested dictionaries but there's no

reason it couldn't and I've been thinking about how I might tackle that because yeah I think once

you have that and you've got body parsing and you've got content negotiation you've got like

three quarters of what people actually want and the forms api then has enough rich metadata inside

of it that you could do stuff like generate an open api schema for the whole thing and you get

you know a rolling stone effect it'll have knock-on effects for everything rest framework

would be able to become lighter as would tasty pie and so those extra dependencies you do have

would have less surface area and more eyes would be on those other bits because they're now in

django proper so that that's you know a promising thing it's no do you have a question will otherwise

i'm going to swing to always but you go ahead but well okay well if you've got a question cutting

because like not as good as probably your question so you go ahead okay all right fine so

the other thing i wanted to really talk to you was about was um you've done lots of tickets on

optimizing you know little bits of django or on profiling bits on the the template engine you're

working on speeding that up you mentioned that earlier that ginger's faster well okay let's make

the dtl faster um can you talk talk about you know how you go about how you approach an optimization

ticket and you know your thoughts there and the work the process you go because i find it really

interesting you open these really detailed tickets that spell out you know some real clear gains that

then make it easy to review and easy to think oh yeah let's get that in the process is exceptionally

hit or miss the uh the tickets you see that i'm asking you to consider in terms of merging

are the ones that have proved successful i've been down so many dead ends that it's unreal um

i have so many notes just floating around with numbers in that don't prove anything

yeah yeah that's important that's important to hear as well though because if people are thinking

yeah if you're looking at my tickets uh the performance related recently they probably

represent maybe a 25 success rate in terms of things i've actually found there's whole other

there's pages of notes around okay well can i try and speed this up not targeting any specific area

but just a thing that looks like maybe i could speed it up and the answer is no because because

of consideration x y or z that i hadn't seen elsewhere or because i'm not super familiar with

that bit of the api and the way it's used and i have to wait 500 seconds for the entire test suite

to fail essentially and say no that was wrong so by no means is is it a 100 success rate it's mostly

a case of finding a section of the code that i think probably could have some speed up applied

and seeing if after that speed up is applied is it even worth it because again even if i make a

change and it's 50 nanoseconds faster on my machine if it's in a piece of the framework

that's not a hot loop essentially it's a waste of time to really deal with it if it's called

once per process and i'm saving 50 nanoseconds it's probably not worth actually wasting the

time on because everyone's already got a whole bunch of things to look at and work on so it's

got to be relatively frequently used or a pathological case and then it has to show actual

meaningful value and deriving that meaningful value is it's it's difficult because measuring

things is not easy i won't say i get it right 100 of the time but if it looks repeatable

across multiple runs if it looks like i can come back and rerun it in two days time and it may not

have the same numbers but it's got the same kind of overall difference that's when i consider

opening a ticket and so that that means a lot of data collection and it means writing any individual

ticket about it takes quite a lot of time because i need to make sure i'm getting the data right

and explaining why that data is relevant and ultimately all of that might be to save a couple

of microseconds here or there but it happens a lot and so that build up allows it to actually

have a real difference in in in production so to speak but it is exceptionally it is exceptionally

hit and miss i won't try and paint it as always 100 successful no no no i think testing is reminds

me of there's the xkcd about compiling code where you know it's like the number one excuse of

programmers goofing off and they're fighting with swords and the manager is like what are you doing

it's like code's compiling we don't really have that in python but the test suite you know if it's

we do when you're running yeah i mean that's and you know i mean we adam johnson we've had on

numerous times he has a whole book on speeding up your django tests but and a larger code base

yeah if it takes more than 30 seconds you know it's like well that's a 15 minute break right

there just every time so yeah i try and make sure i do as much work that i think will run

successfully before running all of the tests as i say i think it takes me about 500 seconds to run

them uh without using the parallel runner so it's a significant chunk i'll go and have a coffee

and uh see what i come back to and if it's good news maybe there's a ticket if there's not

then i will hang my head in shame and move on to something else and see or see if there's another

way to do it i wanted to ask about django technical errors which is a new third-party package you have

up which has subjectively better four or four or five hundred pages i was actually gonna i i forget

where i saw maybe i saw this just looking at you before this interview but a couple weeks ago i

wanted to put in the django newsletter um and jeff trippett was like oh it seems kind of alpha but i

wanted to put a note being like still still alpha but um that looks very cool like can you talk

about where that's at like can i put it in the newsletter are you comfortable with a lot of

people looking at it i mean i would say i'm comfortable with people looking at it but i'm

less comfortable with people using it um it's it's still in that alpha stage of i'm still thinking

through all of the ways i'd like to display the information and figuring out all of the different

pieces of information that django's existing error pages display and the only reason i'm

looking at it is because those error pages have stayed substantially the same for most of django's

lifetime but the web has moved on we have better understandings of things like accessibility we've

got things like dark mode people who have preferred reduced motion all of these things

mean that there's probably something to be done in terms of improving the error pages

and the reason i call it subjectively better is because i'm not as well versed in accessibility

as i'd like to be and it's a lot of experimentation that i'm doing there hopefully you know hopefully

one might be able to say they're objectively better but i certainly wouldn't make that claim

because i don't have a full scope of how it should be and how it actually works for the users

it's currently just me playing around with it it's it's in a promising state it's currently using

tailwind to output all the css mostly as an exercise in getting familiar with that because

that's relatively new and i hadn't had a chance to play with that particularly but ultimately i

be ditching that in favor of actually just embedding all the CSS in the file.

It's in progress, it's very much alpha, I think it's cool, I'm kind

of enthusiastic for it, but if nothing else I'd like it to spark a conversation

about improving the error messages and all of that surrounding stuff. I don't

necessarily think as it is it would ever end up in core though i you know would be thrilled to

have that but if it at least if when i park it it has achieved the job of someone better

investigating and providing new error messages and new error pages then that would be a job well done

and i've in doing it i've looked at a number of other error pages whether they are kind of

from competing frameworks or what happens if you google error page or debug page on google images

and just seeing what other people display and how they might display it so there's probably some

aspects of it that are cribbed from somewhere else but i don't know where and it's just trying

to make something in general that bit easier to pause and i think that was easier for the

django error pages under django 2 when we didn't have stacking exceptions whereas now an exception

being displayed can be very long in terms of the individual frames because zero division error was

caused by unicode error somewhere else so yeah i'm hopeful for it but we'll see where it goes

i wouldn't recommend people use it yet it's a brave thing to take on the the error pages because

They're kind of venerable.

We had Karen Tracy on a few weeks ago

and her book was written like a decade ago

and they're almost unchanged

from what she covers in that book.

I've tried hard to point out in the readme

that it's subjectively better,

but the current error pages

have stood the test of time exceptionally well.

I think anyone who comes back to Django

having five years off

can still read that same error page.

It's not drifted by any substantial amount.

It's just improved incrementally over that time, which is fundamentally what Django does.

That's Django in a nutshell, no?

Yeah, incrementally improved, and I don't think that's a bad place to be.

Fantastic.

Well, I know we're coming up on time, I had one more, but do you have something you want

Matt, I was just going to say, what you're saying about Python and its evolution, I mean, we, the Django Software Foundation, think a lot about Django and how to make it secure.

And I'm the treasurer, so I'm looking at the books and trying to think, how much money do we need?

You know, we always talk about expanding things or keeping it small.

You know, I would say big picture, pulling APIs in at some point to core in some way, that's a no-brainer.

I think that will happen.

we'll probably need to expand the number of fellows i mean we we are we the dsf is lucky

that carlton and marius are so good and so efficient that it's not an issue right now but

at some point we'll have other fellows at some point we might want to expand

we could always raise more money if we needed to to fund things um so these are kind of the

questions uh from the dsf level we think about you know how do we make jango better but i would

say for beginners anyway so my wish list i would say bringing apis in in some form cms i mean you

know laravel has apis and cms wagtail is amazing it's kind of the default now um i don't know if

that needs to be brought in the same way apis does um but then deployment deployment's the

deployment and maybe a i should at some point write a another official tutorial that kind of

gets people kind of what i have in my books but something that says here's how to write and launch

a Django project for beginners, because that's missing a bit now, partly because deployment is

tricky, and also because subjectively saying, well, what's the default? What are the settings

configs? What's good enough to put live? But I think those three things, if there was more about

CMS and the official docs, bring the APIs in, and a way to build and deploy a project from the

official docs i think those would all help and certainly some of those are doable but i'm curious

what your list is i put put that out there is that's me ruminating publicly i mean the list

of things i'd like to see changes on a daily basis depending on on what i'm dealing with i suppose

but certainly having an answer for building the majority of an api would be one of the top ones

i think it would be nice to have and it's a risky proposition but i think it would be nice to have

an answer to how to do the front-end ecosystem there's there's a number of competing packages

in terms of how do i bolt webpack onto it how do i make use of babel or typescript and

And because the front end has become such a separate beast than it was previously, and it's got its own ecosystem and its own configuration to deal with, that's still a difficult thing to get your head around if you're not fully deep within it.

Now, I think Rails has had more attempts at doing that.

I think it has sprockets in the asset pipeline and stuff,

and they change things more frequently, as far as I'm aware.

And the front-end ecosystem has moved at a very quick pace

over these last five or six years, ultimately.

So adding something like that into core would be a risky proposition

because you may end up switching out two versions later for the current in thing

but perhaps that's settling down with things like ES build and SWC that allow

you to kind of make use of JavaScript modules or tailwind packages so maybe

that is becoming something where you can finally add something in and not just

risk it needing tearing out in two versions time i think that would be a good thing to have an

answer for and i think at the moment it's fine that the answer is use this popular third-party

package because it's so changeable but i think one day perhaps when when it all settles down and

things have stabilized fully if it ever happens that would be a cool thing to have i've been

waiting for javascript to settle down i was gonna say we always have to ask us now about htmx have

you used that do you have thoughts on that i haven't used htmx i have used the previous version

of carson's intercooler intercooler yes i used that uh on a number of projects and i think i

have a package for that somewhere as well um just to kind of make the django side of it more useful

and more built-in uh i i really liked intercooler and i think whilst htmx isn't a big change from

what intercooler was i think it's happening at the right time in terms of he seems to be having

much more success uh finding an audience for htmx than intercooler which was always possibly a

little too early for its time it was you know competing with angular and ember and react as

the things that had the zeitgeist at the time and now i think we're not necessarily seeing a push

back on that but with things like htmx and alpine js and then you've got the various live view things

that phoenix has and i think larabelle's got an equivalent you're seeing more

decoupling again. So where we might have had a full offline first React front end,

some people are pushing back into we can do a lot of it on the server and just add in that

interactivity we need via this small package. And I think that scales relatively well. Your

system has to be very interactive or very complex to not be able to get by at the very least with

some combination of the two and i think most most people with a middling amount of development

experience of something of that ilk would know when to stop doing that and move to a fully

separate front end and a fully separate back end or merge the two completely and that's always a

different inflection point because it depends on the project depends on what it's doing is it so

interactive that it's web sockets and streaming diffs all the way down or is it just an occasional

ajax update to refresh some chart or other but i think it's a super cool project and whilst i've

not used it having used the previous one to do a couple of projects i'm really enthused by that one

yeah it feels like it's it's moving up the that gap in terms of having to switch over because

like i'm just about to well by the time this comes out my updated apis django apis book will be out

a lot of newcomers come to me and say i have to use apis and react or view and just every

you know bag of tricks and i always say well you incrementally get there you know start with start

with just django do stuff in the templates only switch to apis later unless you're experienced or

know you need it and so htmx just keeps pushing that out which certainly for people learning or

on smaller projects if you don't really need it it's a whole other world to to manage so i think

that's that's good for all of web but especially for django that helps there's more you can do

with just default django yeah there's a simplicity to it that's powerful enough to get a lot of stuff

done but also means you are just progressively enhancing progressive that's the word yeah

and yeah i think there's there's a lot of value in that and i think it dovetails quite nicely with

the way django works in that django has always been about getting something that works and works

well and hdmx and things of that ilk that are small but powerful allow you to do that on the

front end as well and that's certainly not to criticize react or anything you know but yeah

i i'm a big fan of the kind of progressive enhancement route carlton you had well no i

i mean i totally agree with all of that obviously the um i i think as well they you know you talked

earlier about um extending the forms api maybe into you know a resty serializer type um domain

and what i find is really nice with htmx is the ability to push oh i'm still using form oh i'm

still using form oh i'm still using a form and you get to the you know maybe with a sprinkle of

alpine and instead of instead of sending json data you just send form data you know the javascript's

got the form objects it's like oh i'm still using a form you know it's like i haven't actually got

to the point where i needed json at all yet and then you know if we can then just get the forms

that one step further out such that they can sensibly interact with json you know in and out

then actually it's like oh wow the surface area of what we're able to cover with that core

traditional Django stuff just grows to cover the 80% I think I think it does cover most use cases

at that point then yeah okay your specialist endpoint you know blah blah blah but that's why

I'm excited by it I think it it knows me to do really quite straightforward Django for much

longer than I've been able to do for years like literally I don't know half a dozen years of oh

no now you've got to you know you've got to jump into this front-end world and it's straight to

the api no i wouldn't why would i go straight to the api it's it's really changed um the

phenomenology of building a website you know in a way that i find very attractive so yeah

yeah i i agree entirely having having been around for long enough that that's where i started doing

it and then there's that middle ground of everything becoming an api and now we're maybe

seeing it go back the other way slightly there was a simplicity to it and whilst you know a lot

of users now expect more interactivity and expect all of the extra stuff that the front end can do

that it previously couldn't there is there is that successful middle ground where yeah it's still just

a form coming back and it's just augmented with the validation error of this field and that

validation is correct for their current content language and it's in the right position and it

hasn't had to be duplicated in the front end there's there's a simplicity to that and it's

it's really nice to see that kind of thing come back and i think for django specifically that's

been helped somewhat by the whole forms api becoming template based where previously it was

all just strings being all strings made it difficult to add classes it made it difficult

to add these extra data attributes that alpine and htmx want to add on to things

whereas now with the forms api fully within the template system it's relatively easy to

always augment it with this tailwind class this bootstrap class this htmx attribute this

alpine x data and so it's allowed it to become more seamless than it otherwise would have been

i don't think i'd have enjoyed working on a project that used alpine or hdmx or intercooler

if i had to manually output all of the things or augment them all in the python widgets themselves

so i think i think that's dovetailed quite nicely well just as there's the jam stack and javascript

Maybe we can get the DAM stack going, you know, Django and APIs and markup.

Yeah, let's get our own cool stuff now.

Well, we had here in the U.S., we recently, there was a, the postal service put up a site to, so everyone could get COVID tests, you know, two years late.

But it was a static site and they just had, I think it was jQuery or something for the forums and connect to a backend and they put it together quickly and it worked and it scaled to, you know, 350 million people or whatever.

And a lot of people are like, wow, that's that's amazing. You can do that. And it's like, yeah, you can do that. Like that's it doesn't need to be this big beast. They had to like subdomain the U.S. Postal Service website because it's its own thing. But it's like, yeah, you can be simple and scale and it doesn't need to be, you know, because everything is so much of the web is just a form. And yes, I think there's there's a tendency because we're all driven by interesting new things.

there's a tendency to add complexity for the sake of either learning it or because it's

the thing to do and yeah i think sometimes the performance and scalability thing falls by the

wayside you know django is not the most uh scale excuse me not the most performant thing in the

world because it's backed by python python's getting faster so django will get faster cool

that's fantastic but it's never going to compete with something written in c++ or something but

it doesn't need to because the actual performance and scalability for a lot of the web is as you say

just some form over a database or over an external api and so much of it is data entry in in some

fashion that as long as you can horizontally scale out reasonably well your big bottlenecks

probably still going to be your database yeah well that's why that is the last thing um crud

the concept of crud for beginners new to web development when i tell them and show them that

everything is 99 of the web is crud that's a real aha moment for people when they realize

ah, Facebook, crud, Twitter, crud, like every site. It helps kind of piece together and provide

a context that newcomers, you know, you don't think about if you're just using the web. And

once you get there, it's like, oh, it's not that hard to create these sites because you're doing

90% of it is the same. You're just changing the skin and it's a little bit different. But

anyways, that's something I enjoy showing people. And, you know, it takes sort of five or six, but

when they see, oh, wow, like it's all kind of the same. It's like, exactly. Like that's what I'm

trying to get you to it's not overwhelming like there are these niches and deep areas but it's

kind of the same thing over and over again but you don't until you build it you don't think about it

no and that's that's pretty fundamental to the web itself you know if you exclude things like

web sockets and all of that the web's verbs are just other words for crud possibly they're better

words than crud because they don't form a bad sounding acronym but uh the acronym is at least

catchy that once you hear it you understand what it means fair enough right let's call it a time

to hit up um hit up on our ai so okay thank you for coming on and thank you for all that you do

for django it's you know you're silent hero but you know thank you very much anyway you call you

call me a say silent hero you're the you're the silent hero you do all of it yeah i don't think

i don't see it no it's marius it's marius right anyway right that was uh django chat uh thanks

for joining us folks join us again next time we're um django chat.com and chat django on twitter

take care bye-bye