Transcript: Django ORM - Simon Charette

I'm Carlton Gibson, joined as ever by Will Vincent.

Hello, Will.

Hello, Will.

Simon, thank you for coming on the show.

Simon.

For me, like okay, so I I'm like flushed to have you on the show, but you're one of the big contributors to the ORM, and nobody knows who you are.

So before we kick off, could we tell us who you are and how you found Jiang and what's your backstory and how do you come to be in the community?

So when I um I was in uni, uh I discovered Python by uh there was this very pesky HTML thing that was popping up when I was um trying to connect to the Wi-Fi and was a pain to kind of open my my browser.

And you know, like the old if you were trying to load on the GPS site, it wouldn't work.

So uh I figured out that there was a way I was using Ubuntu at that point, and there was a way to use one of the signals emitted by the OS to kind of like book into it, and and Python was um the way to do it.

So that's that's start that's how I started using Python, just trying to avoid having to uh fight with my uh university Wi-Fi uh portal.

I love

that story.

So my uh my dad was working for a uh company called Citigroup uh Diners Club.

It's a kind of like credit card thing, and they had an office in Montreal, and a couple of times uh per month you needed to like work on the weekend uh because you need to like backups of like credit card stuff, so they had to like to print everything.

Uh all like the credit project they were printing that on paper for others.

Yeah, exactly.

Okay, wow.

Uh, and I got to kind of play with um computers at a very early age, got to uh program uh kind of like using my visual basics, so like writing um uh bash and uh what's the name of the Windows equivalent?

Um like DUS.

I don't I don't remember all the like I was writing like bat files and things like that.

So uh so I I started playing with that.

There was I played video games after that uh and uh trying to automate a few things that maybe I shouldn't have, but like I um I was kind of like obsessed with these games and wanted to like it turned into uh I like more the programming aspect than playing the game itself.

So um so yeah, I had like a background in in programming before using Python.

I worked in a web agency developing um websites in PHP, and maybe my first contact with um ORM Lite stuff is that we had this in ALS uh PHP framework, like a lot of agency ad at the point.

And um we had a database connector uh logic that was mainly talking to MySQL, and we had a customer that wanted to use um, I think it was Access uh was or SQL server, I don't remember exactly, but had to write a kind of like connector uh for that, kind of like port to the logic that was dealing with kind of like pagination and all of that stuff.

So um yeah, that was kind of like before Python was doing mainly PHP, and um there was a a framework as well that was popular before jQuery called Prototype.js, and um the community was not that open, but I kind of like I contributed to Switch Next and was reporting bug to it.

Uh but uh and I remember um we're going on a uh camping trip with my parents and kind of like being obsessed with this thing and just printing the uh the source code on the paper and just carrying it with with me because I wanted to read it all.

Um so yeah, that was that was all um kind of like pre-Python stuff.

Uh I did PHP and JavaScript, I guess.

I've got this vision of you being like the girl from Jurassic Park being like, this is a Unix system, I know this.

Like I didn't I didn't I didn't add like a lot of the focus on kind of like the OS things at that point, but uh yeah, I I like computer stuff and I was uh exposed to it at the early age because of the the job my my dad had.

And so yeah,

I was like, no, that's not a and that's I think that's for people learning program, that's the the thing is that some people come in with years of experience and some people it's their very first programming class.

And so it's sort of like if you can get through that first year of programming, most people can catch up, but you don't know that someone has, you know, in no other subject does somebody have five, ten years of experience.

Uh, but in programming that can be the case.

Um so when I I got to college and uni, I mean I was I didn't know what to do really.

Uh there's was tons of things I liked, and um, but I had kind of like this this pull towards it because it was interesting to me and I think it was rewarding as well because um I'd say what was easier, but I had like uh add facility uh doing these things.

And um that was in college I went to um a college called Collège de Maisonneuve, and uh my I was working in uh my my uh degree was in uh multimedia.

So basically we're doing kind of like uh drawing uh of like 2D animations, uh 3D animations, and a little bit of programming.

Um but since I had like facility in programming, it was um I was very interested in because I was able to do uh focus on the other things basically that um I eventually didn't end up working with, but kind of like gave me a foundation in kind of like some form of like video uh editing and um like the the part the part that liked the most was really about just the drawing.

I was not that good, but kind of like being able to spend some time drawing at school was kind of like I was not expecting that when I was uh in high school that it you could do that.

Yeah, yeah.

Uh and yeah, I it was really cool.

And how did you find Django then?

And um the consultant was uh someone coming from a Python and Django shop, and um he was uh kind of like trying to push for that, but they wanted a uh an external assessment.

So I was there and I was kind of like this PHP guy, I didn't know much about Python except for the script that I wrote for Wi-Fi stuff at the Uni.

And they um yeah, they asked me to kind of like do an assessment because they had a trust relationship with me, and we evaluated both Symfony and Django, and uh we decided to do a complete one ED moving from a PHP shop and uh going to uh towards Python and uh and Django, and uh yeah, that's how I ended up using it.

I think my first contribution was a um translation that was not properly uh done in, I think that was even like before Trans FX, but uh I ended up kind of like creating a ticket for it and started contributing uh to a few more things, issue that we ran into, and uh eventually uh Claude Paraz wrote to me on Claude uh wrote to me on on uh IRC and asked me if I if I wanted to um help with contributing a bit more because I was kind of like

submitting something every week and I guess that was how it worked at that point.

You just had to be uh to be so pesky that they uh they wanted you in so you uh you could help instead of uh booging them.

Yeah.

But as you found at an early age, right, if you show up, it's uh especially open source, like people are dying for help.

So it's that that leap though is a big one for people I find, you know, later in life, right?

Because they have imposter syndrome and all these things as opposed to just doing it incrementally and then um so yeah, that's a smooth transition from you know being like, oh, here's this new thing, and oh, something's wrong.

Like of course I'll fix it.

I think that's still something people don't think about or or don't know where to look or don't don't think that it could really help them.

Um that's something that I think made me fall in love uh into Jen with Django, is that but my experience with prototype JS, it was kind of like a wall.

It felt like I I couldn't um I was suggesting something and it was kind of like there was not kind of like no discussions.

Um I I get that Django is is as a large project as well.

Um it's it's possible that uh newcomers sometimes run into more of uh maybe a pushback against on some things, but I I still do feel and I I try to embody that to a certain extent, to try to be as welcoming as possible, uh, while maybe maintaining some some boundaries as well, because we're humans and we we don't want to burn out, obviously.

So we had like uh Tortue developers that we knew that yeah, we had similar projects uh using our framework, and um there was like a new engineer as well, or in there like it took a lot of time to onboard on the P on the previous PHP projects.

We wanted to compare that, and uh even with the fact that with the language bar basically they needed to write uh learn a new language and um deployment as well at that time was not as easy as it is now.

That was kind of like maybe 15 years ago now.

Um, I don't know, like it was a long time ago, and we still ended up being kind of like flush uh while learning uh all of these things along uh the way.

So uh then a few more projects were added and uh they ended up switching entirely.

Obviously, they needed to maintain the PHP projects, but uh the new projects were created using Python.

And I think that for me is the is the big one.

It's like why why use a framework?

Well, because if you don't, you're gonna end up inventing one.

And the one you invent just won't be as it won't be as good.

Um a lot of like the batteries input there we buy kind of like a peace of mind that allows you to focus on not being in the business of um maintaining um this kind of project.

So uh yeah, it's it's a big plus.

Uh and I I think a lot of people realized it at that time, kind of like the rise of Symfony, Rails, Django, kind of like it was that time when uh folks started to realize that yeah, there's there's a benefit in kind of like putting all that together.

Because that was also the you know the other big one.

Did you did that come up or no?

I played a bit with it, uh, but I'm uh the part I did the most was kind of uh I'm I'd say reverse engineering, but kind of like looking at how they were doing some things, just because I was curious, because I mean it's kind of like let's say a sibling of Django to a certain extent, right?

They they grew along over the years, so it's always interesting to see how they're doing things and how they're they're taking the project.

Um, but no, I I've never kind of like chosen it and use it for a project myself.

Which is a bit, you know, it's a big company and a big Django shop.

And you know, can you tell us about your your life there?

Yep.

y and make sure they didn't do anything they should they shouldn't.

And um there's also this dynamic um in in Quebec and kind of like maybe more the eastern part of Canada around uh just bilinguism, uh be able to send uh email campaigns in both languages, tools like uh MailChin and such to do um a bit of that, but uh it was not we were kind of like trying to target the niche there and uh focus on these kind of like two big features that um users in um Canada and

mostly in like uh Quebec were uh eager to get.

So I worked there for five years, and that's kind of like the time I started computing uh more and more to Django because we were doing things with um kind of like dynamic models uh and kind of like all sorts of crazy things that needed me to dig into like what the ORM was doing, or like bug um NC, a contributor at the time, about oh well this thing is you have an idea of how we can do that.

Um but yeah.

So the the real trick there is that you're trying to you're trying to push it to its limits for a long period of time, and you slowly creep the those that knowledge, that that depth.

Like every time I try to look at them, uh read about them, I I find it fascinating.

Um, the way they implement these kind of like low-level primitives that um can be used to do.

Like nowadays, tools like Postgres, I mean, you can use it for so many things when you start.

Um there's everything in there that relates kind of like data or synchronism uh across um um you know, like distributed service.

So um yeah, it's it's fascinating to me, and it was also fascinating to be dig in there and um see the discussion around um like the expression APIs uh that were those added, uh the work that uh was done as well for NS by uh NC by to try to um just make things better.

Um that was very interesting to me.

And had the chance to be um exposed to um all sort of like these discussions because I was invited to uh Jengo under the hood for uh I think the the three years in Amsterdam, that was kind of like very kind of like um I felt like I I needed to give back because everything that was kind of like uh given to me there in terms of I mean it completely changed my life.

I was kind of like included in in this group of of uh folks that were even I'm just in directly mentors to a certain extent, kind of like being able to watch how they work, how they approach problems, um, and just maybe get things done in in a context where it's hard sometimes to drive consensus, then there might not be like a right answer, there might be some that are better than others, and at the end of the day, we need to make a choice.

Um, so yeah, that was that was the thing that pushed me towards the uh ORM, trying as well maybe to fill a bit of the void uh in this area, um, because it's it's kind of like an important part of the framework, but it's if there's no kind of like keeper of the knowledge there, it's easy to um maybe repeat errors of the past.

Uh tests can only do so much in terms of telling you to not to do a particular thing.

So I mean I guess that leads in so how do we um it's something you talked about in your DjangoCon um keynote um in 2022, but how do we how do we encourage contributions to the ORM?

Because this is it is quite big and scary and it is quite complicated, it is quite difficult, but how do we how do we keep it so that there can be new people coming on board to pick pick up and pass the bat on onto and keep it fresh?

I think the code base is once once you wrap your app around it, um there are obviously parts that are more intimidating than others, for example, things that relate to join pruning and things like that.

These are kind of like non-trivial to reason about, but most of the time you don't have to go that deep.

Um, there's two layers, and they could be better documented.

Something that I like, I mean, three things that I mentioned the at the end of the talk was um a form of like mentoring.

Um, I was uh I I've been kind of like looking on the side about what like the projects like Django Nuts are doing, trying to maybe find a way to uh contribute there and uh chime in, find a way to uh mentor uh developers that might be interested to dig into that.

So that's on my kind of like 2024 um to-do list.

I will see if I ever get to it.

But um other things we could do that um I I was against at first because of the burden mind post, but any form of like typing as well in terms of like self-documentation, I mean I guess it could help because if you have like your IDE and tools are so good nowadays in terms of like generating codes or like guiding you to a certain extent that you can get 90% of the way there, and and that's enough to kind of like just start wrapping your head around um all of these pieces

and um obviously documentation, but that's a large effort, and it's it's a there's a tax with documentation.

If we need to like to keep it uh up to date, it's one more barrier to kind of like trying to change things up in this area.

So you said you said a couple of things in in there.

One you said earlier on dynamic models.

Let's come back to that in a sec.

You just said about typing.

Is do you see that a way of incrementally introducing typing into Django?

So that might that might be one area where we start typing it.

Um maybe trying to incorporate uh things from projects like Django Stubs uh could be one.

Um, I mean a lot of the work has been done there.

Um to me, the the immediate benefit in terms of like maintaining the RM to be uh more around adding it at a lower level, uh so more in terms of like inside.

Uh so we can at least kind of like play with it, see how much of a burden it is, setting up CI pipeline as well.

And my provides like also other solutions uh for enforcing typing, but also a bells and whistle to kind of like do it in an incremental way because it was developed at the point where there was pre-existing code base that didn't have typing.

So, from my experience on other projects, uh at Zapier, for example, it it works pretty well, like the gradual um typing.

Sure, there are new things that you need to learn about.

Uh typing can be uh intimidating, partially in the case of like Django, where we do kind of like some meta uh class magic in some cases.

So you have to it's it's not as easy, but I think there are real benefits there in terms of self-documenting code.

So um, Zapier, for I think seven years, what is what kind of projects do you work on?

So we I you've mentioned that they're very supportive of you work doing open source work, but is it like one big project for multiple years?

Do you have different ones?

Like what is what does the day-to-day look like?

Um there might be um a couple of times where I I have to write a piece of code because it's I'm uh I I have knowledge about many systems, and just getting the teams together to do it would be uh hard.

So sometimes I just kind of like create proof of concept or actually deliver um, let's say kind of like eye-impact code that um allows for um the unblocking of teams or prevent like lifting of dependencies between uh between two teams.

So um otherwise, a lot of mentoring, a lot of uh trying to come up with the um a way to drive alignment within the company uh around the technical vision is something I've been working on uh in the past few months.

So uh that's thank that's changed a lot.

When I joined, I was uh my title was product engineer, and I just kind of like revolved around the uh workflow uh part of the product, which is like very interesting again.

It's kind of like the let's say the equivalent of the ORM for Zap here.

It's just fascinating.

This kind of like huge distributed system that you can just throw a workflow definition at it, and um you say you want to trade graph something, and it's just it does its thing, and it's uh resilient in the face of like the thousands and thousands of APIs that can return very various response code, and you need to make sure that some things are idempotent, and like it's it's very interesting uh from a um software engineering perspective.

Is that accurate?

I think it's uh it's about that.

I think it's uh um I I wish I could code more, and I think that Django is the kind of like escape from that, but I've realized that um the more I code, the like and others as well, and they they try to make it explicit to me because I I like coding obviously, but uh I can have way more impact by empowering uh engineers, developers within Zap here to actually um ship code themselves and kind of like be a vector uh of that.

It's it's you know, like the the uh the writing code and shipping it and seeing it deploy and like ultimately this machine, the reward cycle is so immediate that it's it's yeah, it's easy to get trapped in there while you can you can be way more impactful doing other things.

But then you need, yeah, it's almost like you need open source just to code code or code for fun because you're removed from it.

It's a it's a weird thing, right?

Like there's no other maybe because I didn't grow up programming.

There's most fields you just do what you're doing and you get better at it and you just do it.

But coding, and I I guess most engineering fields very quickly, three, five years in, you're just really pushed to provide guidance rather than do the work yourself.

So it's an interesting interesting dynamic, again, not coming from a programming background growing up.

But sounds like you're managing it well.

It's kind of like like explicitly extract myself from there, but kind of like as I said, the reward the reward cycle is is not immediate, it's kind of like longer now.

But when you see these things happen, even if you just had like this indirect impact on you know, like putting this little cog in uh this whole delivery of thing, it's it's so much rewarding to um to see that you you've kind of uh you've you've helped uh still to a certain extent, because that's the thing that I believe is rewarding from even for Django.

It just feels like I don't know, you help to a certain extent uh someone with with the problems they're facing and uh just the the impact of it as well.

So yeah, it's different reward cycle, but still um very rewarding.

So Zapier must be supportive of your contribution.

And um, so yeah, I do I I I like I like the Django stuff, I like the RM thing, so I gravitate towards that.

Um and sometimes it's just kind of like just a nice break of um sometimes the problems I'm dealing with at jug are more kind of like organizational.

So kind of like going back to Django is is just a nice way to kind of like clear my um clear my mind and just maybe put things in perspective as well.

One is that obviously your knowledge of the RM is an asset to you know their their ability to maintain the product and your ability to maintain the products as a team, but there must be and Simon gets a mental health break as well by you know contributing a bit.

Like, I mean, I would think that'd be a big draw to work with you if anyone's interested in Django.

I would right.

I would if I was a manager, if I was a company owner, recruiting is always the problem.

Uh so I think it's hard to quantify, which is sort of the issue.

How do you quantify someone's time, you know, on the steering council or the security council?

But uh, you know, if I was looking to apply for Django work, if someone was was a you know very involved, that would be a huge draw.

Uh so I think that that's one of the reasons why they're they're motivating, uh, they're motivated by that.

But it's it's great to have this fixability.

It works well for me.

Um so I'm I'm glad.

But yeah, you're you're right.

Like it's I've I've I'm able to help and out for sometimes non-trivial uh questions about how things should be done, or um, we have like a very large uh monorepo, uh like a very like old Django project, like created like I know like 12 years ago now.

Um and uh it uses like a huge MySQL cluster and tons of other databases.

So um there are stuff that comes up that uh we need to come up with um uh solutions that are uh non-trivial and having some knowledge about the RM certainly helps.

Okay.

So we uh we we we use it for like all the hysterical models are are there, uh and uh all the new ones were kind of like trying to break the the monlet, kind of like have each domain use a separate database so we can kind of like maybe stop its broad to a certain extent um because you start to run into yeah issues with um team boundaries, uh the kind of like domain violations that uh Django kind of like allows you to um basically cre

ate a foreign key across like everywhere.

Um I saw I heard you called in the previous um uh Django chat talking about the old kind of like um user slash user profile thing as well.

This is something that uh we kind of like go went back and forth about um because the project was created before uh custom user model was supported.

So we we've got this model, but when Django switched to having a custom user model, it felt like an atrocity, but uh the more I think of it, I think it's great that it's it's kind of like separated.

So I all have to say that I I support you in this uh uh on this front.

I feel like it's there are pros and cons basically, and um, because these models can grow very large.

That's it.

It becomes oh let's just throw another field on the user model, and then that's fetched every request.

It's like no, this isn't this we're not encouraging a good pattern here, I think, is the bottom line.

And so there may be some projects where customers you know don't get rid of the ability to swap it by all means, but as the recommended path, I just I don't know.

And again, it's as you say, trade-offs.

This I mean, so I'm giving a talk in two days at Django Boston, largely just cribbing Carlton's ideas.

So I'm gonna give Carlton credit about the Django user model.

And I mean, 12 years ago, uh Russell Keith McGee put up a whole wiki with the five, it's more than five options when they were talking about what to do.

There was like one, 2A, 2B, 2C.

And even then, you know, just pros and cons.

And you know, I think Jacob at the time was sort of in favor of option five, which is more what Carlton's going for, with with just like just uh basically just an identifier.

Um, but you know, there had to be a uh BDFL benevolent dictator for life decision, and they chose um I guess the custom user model approach.

So it's just fascinating that it's not anything new and it's documented, they had to make a decision.

And I I think it's but you know, late this coming back to it.

I'm like, let's change it.

Anyway, anyway, that's a uh thing.

Simon, I wanted to ask you about, or unless you've got have you got something to say there?

Sorry.

Uh I mean like users have been asking that we lean in to that a bit more.

I think the migration framework is ready to a certain extent, and Drew spent and all the contributors around the migration stuff as well, spent a lot of time kind of like making that better over the years.

Um, so I I think there's value in kind of like Rave at providing abstract-based class.

Uh and in most cases, like having a single user model is going to be fine because you're not going to end up with like 50 fields that really that makes like the tuples in the database so large that fetching them for every request is going to be problematic.

So I I think it makes sense.

Um and I I wish we were leaning more of it towards swappable models.

Uh I know that people not everyone likes them, but I think they're they're a clever idea.

So I've got this package that um just um for makes the email uh field unique on user model.

And all it does is it comes along on the app installs, it adds a constraint, and then it's got a migration that you run one time, and it just all works smoothly.

And if you wanted to uninstall it, you just reverse the migration and you know pretend it was never there.

Do you think it would be feasible to do a bit more like a bit more along that?

I mean, it's a kind of a horrible monkey patch, just a proof of concept that you can do login by email very simply with the with the default model.

But do you think there's a possibility of kind of optional migrations that third-party packages could provide?

Say, look, if you want this, do that.

Uh I guess you could have like different app configs that possibly.

Uh so we we've not done that extensively, but I think the idea when uh AimRick worked on app configs with D that there would be different kind of like ways you could install the app.

Uh you could install it multiple times if you wanted to with a different label by subclassing it, uh, and they would add kind of like a different label.

I I don't think uh a lot of package lean into this idea, but I think that was the idea behind the uh some of the refactor there.

So maybe there's a way there by using like a separate app config that have a different migration module or something like that.

Um but the the challenge still that will always be there is that if you start applying migrations for one path and you want to move to the other one, then

Yeah, yeah.

Okay.

Uh yeah to the test it up.

But it's

I'm gonna ask you one more thing about hot take, then I'm gonna let Will have a go.

I will ask you about um SQLite because there's been some changes recently that were gonna come in.

And I remembered um uh after I don't know, DjangoCon something or other, 2022.

We were just talking because um Simon Willison had done some benchmarks on you know using SQLite.

And I know um Tom Dyson has spoken about using it in production as well.

And I remember you sort of saying, ah, but you know, if you need access from multiple servers and and multiple front-end servers and things like this.

So I'm gonna ask you about SQLite and you know your take on that and then the recent changes that have come in.

So I think it has the perception of SQLite in the industry has changed a lot in the past 10 years, where it was previously more of like a I wouldn't say like a toy database, but more something that you would use for development or uh so on to a moment now where folks are like uh building solutions that use SQLite in a distributed manner where you you can have like multiple things that are writing at the same time, distributed amounts.

So the game changed a lot with regards to SQLite, and uh there's a lot more interest in trying to use it um in scenarios that you would have possibly used uh more of like something like a Postgres or MySQL in the past.

So um the benchmark that um Simon uh ran were very interesting in terms of like seeing how far we can um push it.

And uh there was some uh at the time there was some discussion as well in terms of tweaking some mode um there.

Um it was the something that relates to how uh transaction behave when you open a transaction in terms of like when the um the lock is acquired.

Uh it and a recent change that was merged in was to uh allow to specify through a database option how do you want it to behave basically.

So previously it was working when you open a transaction, there was no um lock that was immediately acquired because you could have like read-only transaction.

In this case, you don't you don't need to add this contention, but the moment you try to do a write, um if the database was already led by something, you would get an error.

And what this thing is um this new flag is allowing you to configure um is to add the lock being immediately acquired, which can have like some impact on contention and is not going to be working well if you use the atomic request setting, because that obviously opens a transaction for every request.

So even if you're only doing reads, which is the case for a lot of um web application, uh you're going to run into issues.

But if you want to um do a lot of uh if you have like a difference, now you're gonna try to push the limits in terms of delivery, that might help because otherwise you might get errors that are you basically have to litter your code with try statement uh handling possible exception when doing writes because the database is locked.

So um, and there's something else as well that's being discussed with regards to changing the uh write-iad log or something setting.

I have not followed that one that closely, but um yeah, two settings that should make it easier to use Django with SQL Lite to a uh larger scale.

Uh I think we're uh there's not um we we use SQL Lite for something that was very interesting.

We moved away from this solution, but um we have a system that injects webhooks that are coming from all over the place.

And obviously you need to make sure that you never lose a webhook, right?

Because if you do, then zap might not trigger and um the service might not be smart enough to kind of like retry it and run into a 500 or something like that.

Um so one thing we used to do was to uh we used to mount volumes directly on these uh um boxes and pods that were um ingesting these requests, and we were writing it kind of like a nodded log in sharded um by minute, I believe, SQLite databases.

Uh so we would, if we in the catastrophic cases where the queue was down or something like that, this thing could keep ingesting and we could open the machine and run a script that would open these thousand, thousand, thousand database and replay.

Um so kind of like a write-the-add log in front of a queue written in SQLite.

And that worked very well for us uh until we we moved to another technology eventually.

Speaking about APIs, I need to ask: have you played around at all with Django Ninja?

Have you had a chance?

Yep.

Do you so what so I what do you make of the current API landscape in Django, right?

Because there's Django Rest framework, which is great, pretty feature complete, maybe not so active.

And I hear a lot about Django Ninja.

Where do you how do you analyze that situation?

A lot of them use Django Ninja because um like most of the time when you spin up new projects, they're I'm obviously they're going to be uh prototype, but they're like you're going to try out things and um this the ease of development uh around uh Django Ninja developers like it.

They feel like they're more productive using it than uh Django Rest framework.

For large projects though, um I we haven't run into a point yet where uh we we need to um kind of like we miss the batteries include of D IRF.

Uh the large project that we're treated with them use it, and it's even if it's sometimes it's very explicit in terms of like uh like not that board play, but like there's there's way more like bells and whistle that with Django Res framework that you get with Django Ninja.

Um uh in these projects we we keep using Django Res framework, but uh in in the new ones, um like a lot of them are opting for Django Ninja instead because um the it it works better for them and uh it's uh we're we're eager to kind of like try things that makes development uh easier uh for some of these projects.

So what's your experience there and your thoughts there?

Um but I think that the yeah the appeal of Pydentic is is it's good.

I mean people like enough, like it feels like the Django was developed in a in a different time, right?

Like with meta classes, there was no nothing of like uh the nice DSL that you can get today with typing.

So I get where it's it's um it's easier, it's it's faster.

And yeah, if we're honest here, like like this Django was was developed um today with like the typing things, it would things would would be very different in terms of like what the DSL looks for model definition.

So um I get why uh people don't want to pay the tax of um well this software was developed when typing didn't exist, so I still need to kind of like um do these things.

And I mean, even with projects like Django stubs and such, you don't get as much um help from your IDE and things that you get from Pyden Tick because I mean the typing is obviously easier to interpret.

Um so yeah, that that's the feedback I've heard from from uh folks that prefer to use it using Django Ninja.

Okay, good.

What what do we say about these?

I feel like well, let's start with the steering council, right?

Because this in many ways um replaces the benevolent dictator for life, and it's the most prestigious position to have.

And yet, like so what how is it how has it been being on there?

And then where would you like would you like to see it do more or less around making decisions like typing for Django?

Like there was a lot of discussions around like the technical board should get involved, more in that, kind of like drag the technical vision of the project, um, and so on.

While it never really happened uh over the years, I think uh it was first interpreted as more of a just replacement for the fact that we didn't have kind of like more, um didn't have like the core was dissolved, and we needed some folks to be there to kind of uh write the um status quo in terms of like we need to take the technical decision here, so kind of like ping us and we'll be able to provide some guidance.

So um I'm seeing our Django kind of like organically evolves and kind of like try to add a few things.

I'm not sure like the involvement from the technical board in terms of like driving the vision would would be necessary beneficial.

I I do feel like it happens organically to a certain extent, and trying to like the features that are going to be highlights uh are kind of like self-imposing to a certain extent, and do we really need them uh in terms of like do we need like a shiny feature every release?

Uh is Django at a point where we need to kind of like reinvent things um over kind of like being stables and keeping up with the latest development.

Um so yeah, and it's it's interesting because if you look like other other projects, like for example, Rails and such, they they're very much driven this thing in terms of like developing new things and new paradigms and so on.

Django is has not been that um in is it okay?

Um feels okay to me to a certain extent to have Django kind of like take this position.

Um do we fear I I don't I don't think it will make Django irrelevant by just keeping it its current pace, is what I mean.

The fact that Django hasn't been changing so much means that now with HTMX it slides in nicely, right?

Whereas Django could have easily gone off in these different directions and now maybe be questioning that.

So that's just one example that comes to mind about being a little slower, maybe pays off.

And there are tons of projects, like, for example, like just the async uh Tishano Cartel, you work a ton of that uh with channels and such.

There are these kind of like new paradigms that are being included, and and we slowly get there.

Um even if the technical board wanted to kind of like say, like, oh, we need to kind of like go full fully async, or we need we need to do this or that.

I mean, we've got fixed kind of like release dates.

Um, it's all contributors, uh, benevolent uh folks that are there.

I'm I'm like, is it going to help?

Kind of like trying to put these things?

Is it going to strain the community even more uh in terms of trying to meet these objectives?

Um, I feel like it's the way it happens, is it's kind of good.

Like if it's ready, we're gonna make there.

Yes, there are some features we're trying to get in and focus more because we want to uh put a highlight there, but it's it's happening at more of a sustainable pace than uh than if we were to try to set these objects.

At least that's that's my take on it.

And it's it, you know, it's not the fully async, you know, brand new web framework.

If you want one of those, there are there are those to choose about.

But if you want to add JSYN, async to your Django project, it's really there and it's good.

And the streaming responses and disconnect handling we've added, and you know, all these things, they they're great, they're wonderful.

Um, and yeah, it's matured over time.

So that's

I just so the security team which which you're on, I just want to shed some light on that because that's often held up, I think rightly so, as one of the gold standards that Django is fantastic with security.

So if I'm a developer and I find what I think is a bug, I email the security team, and then on your end, what does that process look like?

Uh we try to kind of like assess uh whether or not it's it's a it's a false positive, um, uh whether or not there's an exploitation vector there.

Uh and uh assuming there is, uh we're we go we we will let the uh submitter know and we will work be working on our side uh in terms of providing a patch or with the submitter in terms of uh providing a patch, the the problem.

Um in in some cases, uh like if we look at like the recent ones that were uh we ship like security um corrective for, uh a lot of times it's kind of like around uh regexes, uh things that are uh could they look benign, but like if you start kind of like pushing kind of like crazy user input in there, they start to misbehave.

Um but yeah, that that's a process basically.

Um it gets submitted uh and uh before we release uh anything, we uh submit uh a request for a CVE uh to um tie it to the release itself and the commit message.

And from there, all the supported Django uh version supported version of Django are going to get backboards uh and uh assuming they're affected by the vulnerability, and the other is going to be uh accredited.

But yeah, the what the security does basically is uh look at these emails, make some assessment, and um from there determine is this a problem or not.

Um we sometimes we get reports for Django-related projects as well.

I know that in the case like Django Res framework, we had some recently as well, and we try to redirect that to the um folks that are involved with these projects, or um try to see like, oh, is there any uh other part of Django that might be affected by uh a similar problem?

Um but yeah, that that's about it.

There's also kind of like a recent discussion about um the inclusion, uh, how we can get more folks involved in there and ensure the parentity of the of the team uh going forward.

The uh I think Carlton, uh you can talk more about that, but the the the the push uh thereby Michael um menfree was was about possibly using the um the the pipeline of new contributors as a way to get more people involved with security.

Well, you need to be known and and trusted.

So the triage and review team is a nice on-rim for that because you've already been around the Django project for a release or two and you're known, and you that might be a nice way of doing it, but I think again, it's important that it we think about the stability and about the continuity of of the security team as well as you know, just as much as the rest of the Django ecosystem and project.

Um, so that's uh but thank what you know one thing, Simon.

We have to say is thank you for your service there, because that's you know, it's an on it's it's an it's a thankless task at times.

Like you're driving the community like that.

I know how people in Django get very excited about just listening to Django chat and uh seeing you just revolve around the community as well.

You've been just uh yeah, uh stars of uh of Django.

That's that's the thing, is that the real work is happening in the dark.

Um it's black hole.

Uh some of it is uh more facing, but uh yeah, it's it's awesome to know that there's it feels great to be uh kind of like in this distributed um entity and seeing it uh kind of like continue, uh continue to work, evolve over time.

Uh obviously there are things would like to evolve faster and so on, but it's it's great to see it to see it grow basically.

I've been kind of like revolving around it for for more than 10 years now, and it's uh yeah, it's it's an awesome project.

So I have I promised a couple of weeks ago we had um two fellows from uh uh Pin Planet, which is uh an app built with Django, and Andrew there's the expert, and he he he asked I had a question I promised him I'd ask you, which is he's from an SQL background and he's used to you know working in raw SQL, and then when you jump to the ORM, and he he's very um capable and pleased to be able to construct equivalent queries in the ORM, but his point was they often don't look very much like the underlying SQL that's gener

ated.

But you know, perhaps you have to use values in a weird way in order to get the grouping right.

Or is there is there any way that we can we can make the the relation between the ORM query and then the the SQL more transparent is in your view?

Uh so everything relates to models.

Uh sure they are kind of like aliases that represent tables, but one thing that SQL Alchemy has is that there's these two kind of like layers, like you've got the um kind of like table representation, you can bind models to them, and you really have a query builder.

You have like a lot of flexibility in terms of like you can use tables directly and use like um exist clause here, kind of like this join here, and so on, and uh you can then kind of like pipe that into a model representation.

The org doesn't have that, the the kind of like compiling part is every coupled to the notion of a model, which make things like um any form of like union or uh addition of like eventual um uh common table expression, the width statement uh in SQL or things like um uh the discussion about a subquery wrapping and things like that.

Um it's very there's a fine line in terms of like providing very high-level abstraction while still allowing um low-level injection of logic.

And that's I believe the big challenge with uh Djangona.

You look at like primitive that were recently introduced, like um uh filtered relation, for example, that tries to kind of expose that.

It's it tries to do it and it it for complex cases it fails to a certain extent because um the ORM very much needs to be able to know what kind of joint it's dealing with, uh, with regards to elighting them and optimizing them and um making sure that the query still makes sense in the context that you make.

So I think that's the kind of like biggest challenge in terms of uh providing more flexibility to folks that are maybe have a DBA uh background or very familiar with SQL and want to turn the URM into doing what it wants.

Um so that's that's a big problem or um design issue with the current state of things.

I know NC brought up a few uh like years ago that might be beneficial for the ORM to kind of like be built on top of um SQL alchemy so people could like drop back to just doing these kind of things, but um that's uh too big of a right for me.

I'm not I'm not ready to venture there and support this multi-year venture.

So then the the I guess the question is if do you have if you put that on the side as something that we're not gonna do, do you have a kind of magic one that you would you know if this if I could fix this, I'd fix that.

If I could change this, I'd change that.

Um I wish the um we we we were doing a better job at kind of like buying more into uh removing a bit of like the extra uh stuff.

So today most of extra is obsolete.

What but one thing that remains is around the um so t course that extra allows you to do a few things like select new columns, uh but order by or where um and there, but all of that can be uh emulated using raw SQL, the expression itself.

Um you can like import from Django DD models raw SQL and wrap it and pass it directly to filter in order by.

And the fact this is carried around all over the URM makes it um and the fact that it's like kind of like not um that well supported in all cases makes it very awkward to uh to use.

So like finally pulling up the band-aid on uh extra, kind of like deprecating most of its option and coming up with a nicer way to do um reference to tables, either being through joints, common table expressions and such, I think could could um push the a bit even further in terms of like what is doable without going through a massive refactor by introducing an intermediate uh table representation.

Okay, great.

Uh just wanted to say that uh it was it was great to be here, it was great as well to listen to you all.

Uh it's great to be uh part of the community.

And um, if there are uh again, I I want to reiterate that, come into it.

I I want to kind of like get closer this year to either like Django uh Django Knots or any form of like similar projects to find a way to kind of like uh do form of mentoring.

If you need you have questions about DORM, uh about the PR uh uh pull request, you can you can ping me on the on the and on on uh on the on track as well and do my best to try to support you through that.

And um I I learned about DORM by just trying and breaking things and moving things around.

So um it's only a matter of like trying, I guess.

So it's it's a lot of stuff, but it's it's certainly manageable once uh you you you you play a bit with it.

thank you for coming on.

I it's truly when we started this podcast, one of the guests that Simon uh that Carlton wanted was you.

You were like this mythical figure that hadn't been to like recent conferences and worked on the ORM.

And so you know to meet you at DjangoCon US two years ago, and then now that you're on, like you were like one of the top five people Carlton was like, we need to get this person on.

So we started this.

It's it's glad to be here.

Uh glad to be here.

I think that the past years with like uh I had like another surgery last year and other things.

It's just uh and I played Ultimate Frisbee before during the exact season where uh all the the conference were happening, so it didn't work out.

But uh yeah, looking forward to possibly meet you in person uh at the next events.

Well, thank you.

Um so everyone, we are at DjangoChat.com and we'll see you next time.

Bye bye.