Transcript: Kolo for Django - Lily Foote

Carlton Gibson joined us over by Will Vincent.

Hello, Will.

Hello, Will.

Thank you for joining us, Lily.

Yeah, it's a pleasure.

We always ask.

So, you know, it's kind of the origin story.

So let's go with that.

How how how come you're here?

Um that led to me thinking, uh now that that led to me uh picking up Python and um then I was like um how do I uh how do I make my own website?

And I looked around a bit, saw that Django was quite recommended for websites with Python, so tried that and then ended up uh turning that into a career.

Um yeah, I uh uh had a couple of um junior level jobs and then um ended up in Oxford uh uh for a company called Incuna where there were a bunch of um other Djangoe people, um uh Mark Tamlin, who was a uh core dev for a while, um and uh uh Charlie Denton and uh George Hickman and

just being around them uh really kickstarted my involvement in open source and um as very inspiring.

Because I think Torchbox uh

Um uh yeah, I um never worked with them, but I am aware that they're around.

But there was um were there meetups and things like you know, is there a scene?

Um uh he's since moved away, so it's run by other people now.

But um I remember I uh tried running some Django girls events in Oxford.

Uh we managed to do it successfully twice, which was good.

Well that was super.

So you know, was was that your first?

Were you been before?

Had you been uh seemed to be quite well established

Yeah, so I think the first DjangoCon I went to was um Cardiff in 2015.

Okay.

And then yeah, I've made pretty much every DjangoCon Europe since, um except the first online Porto one I didn't go to.

Um I one of my early jobs was half Django and half uh Kivi, the um Python uh mobile um framework.

In recent years I've um uh got into Rust a bit.

Let's make it to more difficult.

The um so okay, so the the way I kind of got to know you mostly was you're like, oh yeah, I I'm I'm working on this this DB defaults ticket,

Yes.

Can you tell us a bit about it?

Listen, don't know what it's what's it for, you know, and perhaps the backstore of getting it in because uh you know, spoilers that store that ticket was open forever.

Um for the most of Django, um you were able to define a default, but that would um only be used at the Python level.

So Django's RM would um set that explicitly before sending it to the database to be saved.

Um but now we let we have a way of telling the database that's this is what the default is, and whenever something's saved, uh the database can put that default in for you.

Um this is really useful if you're doing um stuff that's not um Django with the same database as the one that you are using Django with.

And um so it's it's quite a um useful feature.

Um I uh I think I picked it because I was looking for a um a RM ticket that was um wanted.

Um I think I was looking at the old tickets um because that's always an interesting place to look.

And um by the time it uh landed it was the second oldest open ticket.

Fantastic.

I mean, DB defaults aren't just useful, like they um so it's all right.

So um the classic example is the date now, right?

I need a date added now, I need that something.

Um, but then if I go in and I happen to be editing not via Django, if I access you know via pandas or I access via any other um system, that that um default is not enforced.

And to have it at the db level saves me a bit of work at the at the Python level, that's one thing.

But I think you know, if if I take my database seriously, which maybe I do, maybe I don't, but there are lots of people who do.

If I take my database seriously, having it at the DB level adds a whole layer of security that's significantly better, right?

Yeah, it's it's um I I will take your word for it that I'm under selling it.

Well, just easy to know it's a little, yeah, I don't know.

Okay.

Was is did you also progress through other open source Django things before that, or that was something you had you were dealing with for work and so decided to just jump right in?

Um and that's something that I needed for work.

And then I just kind of stuck around and was like, yeah, let's let's keep doing ARM things, and I think the vast majority of my contributions have been in the ORM at some some more somewhere or other.

I mean like the DV tickets, how many how many DjangoCon sprints were you just chipping away at that, chipping away at it, chipping away at it?

I I do do um uh conference sprints based development.

I go to a conference, get very motivated, hack on it for a bit at the sprints and for a few weeks afterwards, and then take a break until the next conference comes around.

Helps balance my um uh against burnout and stuff.

Um as to how many it looks like I um assigned myself to the ticket in 2020, though I did CC myself earlier, so maybe I was thinking about it in earlier than that.

Maybe I'm not 100%, but I've got it in my way.

So okay, so let's talk about check constraints because it it um what so Django defaults um in the database, in and of itself, it may, you know, whether you think that's a groundbreaking feature or not, I think it it's part of a long line of things that have come in the database from indexes through constraints, through comments in the DB, through database defaults, through hopefully we're gonna get cascades coming up.

And all of those are kind of enriching the the ORM.

And yeah, you'll you know you you're probably as good as capable of anyone who's speaking to that topic.

Um now when would that have been probably around 2015 2016 um expressions lookups lookups transforms um I think everything that I've worked on since has been made possible by that

yeah what is the learning pathway there like how how do we it's kind of like there's this there's this clip that people don't jump off and how do we how do we help them to do that?

I think the thing that really helped me was I mean one thing that helped me was I got in early ish which is a bit tricky for other people but the other thing is just I went to a lot of sprints where a lot of experienced RM people were already so I was just around them and able to talk to them and just learn from them.

Right.

I've often wondered if you don't need to know this the SQL in order to really understand how the ORM is working

I suspect that's part of it.

Or maybe I'm just being naive and at a certain scale all these things come up in your day-to-day work

documented but you so you spend a lot of time learning it and I just ah anyway it's but everyone hits those issues right the

and then you've got something for your toolbox or maybe it's a colleague at work.

I think it's a lot of this is just community knowledge that you pick up by being around and talking to people who've been there before you.

Yeah I mean

I think Django

Um it's I think the maybe the biggest problem is that there's so much documentation that it's a bit difficult to find the right bit of it but search engines are not too bad for that.

And then it's like well what do I use this for I mean like window expressions are kind of quite powerful and there's quite a nice dock on window expressions and how they work but actually you need some good examples of how to use them and and whatnot that I kind of think are missing.

I think there's a there's a gap there for that kind of um worked examples of how of these more powerful ORM features.

I don't know

I mean obviously we're talking about the ORM but have you like primarily APIs and back end have you done templating any front end stuff or do you have a niche?

Yeah I have dabbled with front end but it is not my comfort zone.

Um I think from for what I've done with it REST framework has been pretty good.

Um there's a few bits where I'm like okay that's a not quite how I would have chosen to do it perhaps though I can't think of any examples this is thinking back to last time I ran into that um but Rest Framework does a pretty good job.

I I remember uh what was it that I used before Rest Framework?

Tasty Pie I think and Rest Framework works so much better for me um it was a more explicit Tasty Pie was a bit too magical if I remember correctly I I like the reasonably explicit nature of Rest Framework.

but then you know same with atters same with um pydantic look if you need um the more complex cases where you need to add transformers or validators or you know whatever on then you end up adding a field descriptor anyway and it doesn't end up looking too much different than what you've already got um so it's kind of interesting as to if there's a sweet spot for bringing in the the newer sort of syntactic sugariness but you

know it's no point throwing out the whole stack for just a little bit of syntactic sugar so there's a that's kind of where I'm at mulling it over in my mind.

I don't know.

I've not actually um really had a chance to play with that stuff.

Um the past few years I've been um not actually really doing Django itself that much, though very much in the ecosystem.

Um so colo I describe it as a dynamic code analysis tool.

Um it was um invented because Will um Klopp was frustrated with traditional debuggers, um, which you have to know where to set the breakpoint, you um you have to understand your codes a certain amount to be able to use them well.

And he was like, I want something that'll just give me a lot of that context without having to understand the code beforehand.

So he started building colour and we um met online at the um Junk on Europe 2020 uh online sprints uh online conference for um uh in Porto, and um we got to talking and I was trying out contracting at the time, so I um uh so we decided to give that a go and it went really

well.

Um it was a lot of fun um uh working with Python's introspection um features and uh working out how to um extract in useful information from people's programs without being too slow and um yeah uh I eventually went full-time with uh colo because it was just simpler and

I was having so much fun with it.

I um and that's also how I got into Rust as well.

Um we were looking for a way to reduce the overhead of um having written it in Python originally, um, tried Scython at one point but didn't really get anywhere.

But yeah, tried Rust and was able to get that speed that we were looking for, which is really good.

Um Will's talk at DjangoCon Europe this year, and it's we'll put a link to it.

I mean, he got a standing ovation for that talk.

I've never I've never seen that at you know, all all the Django Cons I've been to, I've never seen that before.

I mean it's quite a jaw-dropping like moment to see it in action.

That was um because so what he did was demoed Colo's ability to generate um tests, Django tests for you.

So um made some requests.

So you have Colo running.

Um, you know, is it in the VS Code debugger as it's running there?

So you're clicking around in your application, and then it does the whole um, you know, gets the clo the test client, configures it with the right parameters, you know, does the request, puts in some some dummy asserts, and then you can customize those to check the exact things that you you want to check.

But it's like, oh yes, that's the really sort of repetitive bit of writing tests for Django, right?

And it's just went boom.

And it's not um one point he made it's not um it's not probabilistic, it's not generative, like um, you know, you might get out of Copilot where it's quite often wrong, it's fully static, right?

It's fully deterministic.

It's it's really good when it works, but there's an awful lot of edge cases and stuff, so right.

Any chance of getting it as a standalone app or is it tied

to the VS Code except for the same thing?

Do you want it with BB every

well exactly?

I'm like, come on, come on.

Um we have some work in progress stuff around these areas, but I can't say uh when they'll be launched.

Testing is one of those, you know, it's like testing, deployment, those are kind of the two big things that I always want to check in with people on how they're doing it.

How how have you seen testing?

Like, how do you test Django uh projects and how has that changed over time?

If if you could speak to that, right?

Because that's I'm I'm curious your example because I feel like everyone has a little their own flavor of how it's actually done.

Um I mean, I have long been a um fan of test-driven development.

Maybe not purely strict.

I often bounce back between writing a bit of code and writing some tests, but I try to have it have them inform each other.

I think Django testing is often in the integration test um realm, especially when you're doing pretty standard CRUD stuff.

Um because the stuff that you really want that fine-grained uh unit test um stuff for is the um custom business logic that is um not so um entangled with um the database and so on, which is relatively rare in my experience in Django projects.

It's all a bit entangled with the database at least, so you end up with some sort of integration test in my experience.

So yeah.

Um yeah.

I personally am a fan of um the request factory and um uh skipping middleware and just putting the request into the view and so on.

Um but for Colo's auto-generated test, we've been relying on the test client a lot um because that matches what the what we recorded better.

Um because when you record a um a local request response that would go through the middleware, so it makes sense to test it going through the middleware as well when auto-generating.

I think people testing views in isolation is kind of more advanced or more niche.

And yeah, I I like PyTester fair amount.

Um I mostly like the tiger asserts and I use the fixtures a little, but I wouldn't call myself a PyTest expert.

There's a lot that is in there that I just don't use.

Coverage is amazing.

Um frustratingly, um for Colo, we it's actually really difficult to get um 100% uh test coverage for the same reason that coverage itself really struggles to get 100% test coverage, because the um Python API that we use, uh in our case it's .set profile in coverage's case is .set trace.

They're similar, but with a few minor differences.

Um they can't introspect code that is running in one of those hooks.

So coverage can't introspect its own code while it's running.

I've had to work around that, and it's um a bit frustrating, but oh well.

There is the in in 312 the um um uh new uh is it I forget what it's called, is it monitoring, which might allow for introspecting that stuff?

Uh it's been a while since I've looked at it, so that might have changed.

But fingers crossed we can get better coverage out of that.

Do you do you do anything with Playwright or any of those tools, or is that less applicable?

I mean that's more for like testing JavaScript and uh like you know, did my Stripe payment go through trying to uh there were some talks at DjangoCon US about it.

Gotcha.

I think that's the dominant tool.

I um uh pa Andy Knight um is is very big advocate of it and had a keynote, I think, one of the days on it.

Have you have you uh I just uh push that once more, HTMX, have you at all played around with that or is that cropped up in this template only project?

Honestly, I think if I had a reason to do some front end, I would look to see if I could use HTMX for it.

But I mean my front end is VS Code at the moment, mostly, so it's what works in VS Code, which is a lot of TypeScript.

Oh fun.

Can I ask you a couple of linked questions?

Can I ask you

And then swing swinging from there, just keep in mind it's like how do you how do you find it in relation to coming back to Python and then the typing efforts in Python?

Because I I'll tell you my response, then you can just bang off.

Is spend all time writing properly typed code in Rust, and it's good fun, it's hard, and that you've got the compiler and it's all this nice.

And then I find typing in Python to be deeply frustrating in that it doesn't do it's all it's all work and no promise.

That's a comeback.

So with that as a as a sort of blur, how do you feel about Rust?

And you know, what do you like?

And what lessons can we draw?

Um it's it's high level enough in a lot of places that it wasn't too big a change coming from Python for me.

The thing that really helped with that as well was that I had a very concrete use case that I understood inside and out.

Um so I was just being able to gradually port the bit the Python implementation of Colo into Rust.

It's a really smooth way to learn Rust.

And I just iterated on that.

So the amount that I had to learn at each point was actually very small.

It's just okay, what's the next little thing I need to learn?

Okay.

Um I I have discovered that I really like um statically typed programming um when it's got all these modern ergonomics that Rust has.

Um sometimes it's still a bit tricky to work out what the right typing should be, but once it's once you've got it, it really just works.

And I absolutely agree with you that the experience of optional typing in Python and TypeScript to the extent that I've worked with TypeScript, it's it's just second class compared to something that's thoroughly integrated in language.

And then I get deeply frustrated with it and just like no, I'm ripping it out because I just don't want this.

It it takes all the fun out of writing Python for me.

And it's not that I dislike Rust, I love Rust, but Python typing ain't it.

I mean

So it's a view and you assign a class-based view and you assign a model class as the model attribute, and then it's got um at various points you might use self.object.

So you know, on a uh create view at the end, you've got self.object, or on a detailed view, you you fetch the object to begin with, and then you render it to the template.

So self.object has got the set is a is an instance of the type that model is, and there's just been literally no way of typing that in Python um without typing something like class my crud view square brackets my model, you know, model equals my model object my model.

And it's like you've typed my model 40 times in three lines.

It's like this is not I'm not doing it.

I can't, I'm not making any person write that.

I wouldn't write that.

No one should ask anyone to write that.

It's just it's not good.

Yeah.

In in Kalo uh in uh we do actually still use my pie and typing, and my experience is that like 80-90% of the time it's just busy work to tell cut uh to tell my pi what I already know.

Um five percent or something, um, it's okay.

My pie is actively stopping me doing something that I know is right, and then the rest of the time it actually catches a small mistake.

Okay, yay.

It's uh now maybe I maybe the benefits would be more apparent if I was working with more people also working on the code base, but it's me and Will basically.

Um, and I pretty much own all the Python and Rust code, so it's

Interesting.

Okay, let's move out before the before the lynch model arrives with

the flaming.

And uh yeah, anyways, but I don't have a

Oh yes, you uh one of the navigators on the on the on the first pilot program that's just finished.

So can I ask you about your your experience there and you know your thoughts?

Because from the outside looks like a wonderful thing.

Um I think uh Sarah Boyce uh was the person who um asked me if I wanted to be a navigator, and I said, yeah, alright.

Um and yeah, it's um it's been good.

Um it was I had uh three mentees and they all got stuff landed in Django, and um I I'm glad I did it.

It was um good to see them uh getting this stuff in and having the confidence to work with Django and so on.

Yeah.

Because I think there's a billion people listening to this podcast who would be qualified to do it.

And we've had the organizers on you know, Sarah come on and talk about it and pick it up.

So, but you who played the navigator role, what did you have to do?

Well, if you know someone's like, Well, maybe I could be a navigator.

What so what's involved?

Um so that was fairly low commitment.

The other thing that I did was I tried to keep an eye on the pull requests that they had and give an initial review, um, and uh I think that went fairly smoothly as well.

Um the time commitment from my point of view wasn't wasn't too onerous.

You can always um drop out from doing it again if you don't like it.

I also said that if if they had too many people wanting to be navigators, I was happy to step back.

So and I haven't heard yet whether that is the case.

So we'll see.

Okay.

So I imagine there'll be uh yeah, we should have one of them on to mail.

Well, actually, we're gonna have Sarah on uh coming up soon, so we can ask her to give us a recap of it.

I think in yeah, Sarah, Sarah's gonna come on and talk.

So I should have Sarah B before.

Oh yeah, sorry, yeah.

Two Sarah.

So many Sarah's gonna have Sarah.

Nice.

I looking at other other work in this space is something that does sound valuable, but it's not something I've got around to doing.

Well that would be it would be.

I mean, Carlton, you can probably maybe speak to this.

I mean, SQL Alchemy is certainly historically has been spoken of as having some features that the Django ORM does not, but I don't know if it's as active as the Django ORM is.

I suspect not, but I I can't speak to that.

Um, I'm not an I'm not a big ORM depth expert, so I've used SQL Alchemy a few times and thought, yeah, it's it's it's an ORM.

Um like I think the big thing is that like um Django's the Django RM is slightly more um object-y, if that makes sense.

Like SQL Alchemy that doesn't hide away the SQL quite as much.

Like you're aware

Um so I don't know.

I don't uh yeah.

I mean, all the expressions are now more directly named after the equivalent SQL concept and um check constraints, that's named after the database name for it.

Whereas early stuff like filter is more our own naming.

So there's some value in matching up well to the database concept that you're implementing that we've been doing in more recent years.

But uh so Colo, uh obviously it's uh so we had David Kramer from Sentry on a little while back, and he was talking about how Sentry developed initially as an improved Django debug toolbar.

And in a sense, Colo is also like that.

So my question to you is beyond where it is to now, where do you see potentially going to you know in the future if things keep going swimmingly?

Um I want us to have a good story for not just Django, but um Flask, Fast API, maybe even Scientific Python, though I don't know if that's possible.

Um that's where I'd like to go.

But um I don't know if it's where we'll end up going.

Um I'd also like to be able to I'd like to be able to have a way of people contributing plugins to it so that we don't have to do all the integrations.

Um that's something that's always simmering in the back of my mind.

How are we gonna do that?

But at the moment the test generation is our main focus, I think.

So that's where a lot of our thinking and um development time is going at the moment.

I mean it's you know it's quite an impressive little tool to be honest.

Yeah, well, no, I mean just thinking on that, like the the question is like at some point it has there has to be a business towards it.

So it's moment colo is free.

Um, and I guess that the the sort of um the big one is teams, right?

You it's uh is to be able to provide some kind of you know help at a team level where you know start up five, ten developers, they can get a subscription, give it to everybody.

Um you know I you know

Um, but yes, I would I would love for Curlow to be self-sustaining.

Well, we've been not doing nothing but asking you questions.

Is there anything you wanted to speak of since you're on?

Any in the anything that are of interest or with work or the community you wanted to mention?

Um I know that's a big big one I just threw at you, so it's fine to it's it.

Um uh I guess there's there's been a fair bit of um conversations on um the forum and Discord recently about uh consensus and how stuff that um a lot of um suggestions to for changes to Django go in at the issue tracker and a fair number of them just get it accepted and that's great.

But when but when they're rejected the response is okay, if you don't disagree, take that to the forum or used to be the mailing list or whatever.

And then it feels like it's a much it can be a lot more of a draining process to um try and make your case and um get something in.

And I think my wish list for Django going forward is that we make we make that uh less draining on contributors somehow.

Um when they've they've got an idea and it's not immediately met with yes, let's do that find a way to make that smoother and I'm hoping that we can improve that experience somewhat in the future.

The the default has been for many years that um a sort of change required a consensus in favour of it.

Yeah.

Yeah.

I mean, you know, to basically any change.

Um and that's that can be massively frustrating when that's the feature you want.

Yeah.

Um I I think one of the things that has kept me around in Django is I've largely only picked things to work on that were already accepted.

Yeah, yeah.

Is the the the the the reason why the historical policy was you know uh requiring the consensus for the change was uh the keeping Django stable.

And if you I was on a the Django developers mailing list searching for some things, and I've came across a thread from I don't know 2015 where there was massive gripes about the the massive churn between versions and the difficulty of updating.

And looking back now at 2023 to something from 2015, you think there's no churn.

Like you might, oh yeah, I have to tweak a setting and I've got two and a half years to tweak that setting before you know it becomes pressing.

It's really not uh it's not a burning, and most times it's pip, you know, pip install dash u Django, run your tests.

Oh, it's done.

Look, I'm on the latest version, took me 10 minutes.

That wasn't the case.

Yeah, I I I have some memories of upgrading back in the 1.x days.

I do remember it, but I think perhaps we've overshot and got a bit too conservative now.

Yeah, possibly, possibly.

I'll just say uh uh to what you were saying, Lily, one thing I've taken heart with is um Paolo has this idea of Django sprints.

Oh yes,

which he's

So perhaps that's one way.

Um yeah, there's yeah, there's been a number of things recently.

I mean, I guess grand scheme actual raw numbers, probably not that many, but the it feels like a lot because there's so they generate so much discussion and it feels like such an uphill battle.

It can be just the like long grass, and it's like, how do I move this feature forward?

Yeah.

It's not not sure I can, or not sure I can quickly.

Yeah.

Yeah and they don't so there is some but you know, jjango not programs and others, we can see that there are people and faces, and the community is friendly, even if the interfaces aren't always as friendly.

Actually, that was one of the points that Sarah Boyce was making about um she's a big fan of the the Discord and how that sort of interaction more aligns with, I guess, modern communication and certainly is more friendly than just an email service, email-less serve.

Um I also think that for better and for worse, Django has a reputation of I mean for better it's really stable and um uh slow moving, but also for worse it's maybe missing out on um opportunities it could um it could be provid

ing to um is Django going to continue to be picked for new projects?

Some people I've talked to think no, you're only gonna pick Django if you already know it or it's a maintenance.

Is there a d default though?

But I mean Yeah.

Django still exists and it works.

Yeah.

I mean, I mean certainly there's just the marketing of Django.

This is one of the topics on the the members' forum about just the website and things, which there is work being done.

Um uh so yeah, how do we get across?

I mean, that's one of things I've been most heartened by is um Sarah Boyce has really taken the lead on the upgrades to Django section on the Django news newsletter and just showing it's just it's so active.

There's I feel like it's 12 to 15 a week of changes and you know, so some sort of marketing sheen on what's actually happening, but because we're mainly volunteers, we don't have the muscle to do it or the time to do it.

And that will lead to it continuing to flourish.

I mean, there was this or being on the board during COVID, it definitely felt a little bit like end of days holding on.

But you know, now that we have the conferences and everything and new initiatives and new people, and so I mean, so many people are going out for the board now that there's uh the new board that there's even some discussion of perhaps we should expand the board or you know find a way to take advantage of people who want to contribute in a way that we're not.

I mean it's a good like really strong position to be in.

Um Carlton, you had something to say.

I I interrupted you about

I just I totally, you know, I agree.

I I don't I think it's I don't really buy the old Django uh mortal danger line.

I've heard I've been hearing that for 15 years and it it's no true.

And it I I I don't buy it's um there's an existential threat.

I look at the changes that come in every version, I think, yeah, this is positive.

I do hear, you know, um what Lily's saying about you know, can we can we tweak the the way we um discuss the features?

Can we think about ways of pushing forward?

I think you know um there's not enough, there's too much of a barrier to getting to a full sort of formal depth, this this Django enhancement proposal that it's too it should be a lightweight thing.

It should be look, here's my here's my proposal, can we discuss it?

Can we feedback on an actual written spec?

Because you know, if you want to make a major feature change in any project, to write up a little spec's not a big ask, right?

That should be easy.

It should be we should be encouraging people to write writing third-party packages.

We should be saying, look, look, this is what I've done.

How can I can I pull it in?

And yeah, we should be open and you know, let's do all of those things.

That's all great.

I don't buy that Django's index essential threat if we don't, but yes, let's let's take the new energy and let's go with it.

And it is wonderful to see just so many young faces.

Like we went to the that first conference back after the COVID two, two, three years where we didn't have it, and then we were back, and it was like, yes, all the old faces, yes, all the new faces too.

And it's like, oh wow, yes.

I think it's a really healthy place that we're in, and that's just wonderful.

So that was I took a lot of heart from that, as did I think most of the people there.

I Right.

Well, I think we're we're pretty much um out of time at this point, unless there's any is there anything we didn't cover, Lily, or managed to throw enough questions your way?

Well, we'll have links to everything we discussed, especially Colo for Django, which everyone should check out.

And um Is there is there any way that would you like if anyone doesn't already know about it who's listening and tries it, is there a way to provide feedback or what would you ask someone to to do who downloads it and for the first time?

Um we have some documentation.

If um if the documentation is unclear, please let us know it's something that um I want to be good but I'm not a technical writer so um uh we have an issue tracker as well um and um I think you can also uh schedule time with Will to um come and actively help you um uh get started or

work through a particular problem or something like that as well.

There's lots of different options.

And yeah thank you for coming on really good

Bye bye bye bye