Transcript: Django ORM - Simon Charette

Hi, welcome to another episode of Django Chat, a podcast on the Django web framework. I'm

Carlton Gibson, joined as ever by Will Vincent. Hello, Will.

Hi, Carlton.

Hello, Will. Today we've got with us Simon Chalet, who's a long-time contributor to the

OAM and all-time hero of mine. Simon, thank you for coming on the show.

Yeah, thanks for having me, Carlton and Will.

Brilliant. Simon, for me, like, okay, so I'm like flushed to have you on the show,

but you're one of the big contributors to the ORM and nobody knows who you are. So before we kick

off, could we tell us who you are and how you found Jang and what's your backstory and how

do you come to be in the community? Yeah, sure. So when I was in uni,

I discovered Python by

there was this very pesky HTML thing

that was popping up

when I was trying to connect to the Wi-Fi

and it was a pain to open my browser

and if you were trying to load an HTTPS site

it wouldn't work

so I figured out that there was a way

I was using Ubuntu at that point

and there was a way to use one of the signals

emitted by the OS

to book into it

and python was um the way to do it so that's that start that's how i start using python just trying

to avoid having to uh fight with my uh university wi-fi uh portal fantastic i love that story

well it sounds like you maybe you'd already done a little bit with programming languages before

yeah correct so my uh my dad was working for a uh company called city group uh diners club it's

kind of like credit card thing and they had an office in montreal and a couple of time uh per

month he needed to like work on the weekend uh he's into like backups of like credit card stuff

so he had like to print everything uh all like the credit card they were printing that on paper

for others but was it on the paper with the little dots down the side yeah exactly yeah yeah okay wow

boxes and boxes of that and um so i and there was like a crazy server room there as well

uh and i got to kind of play with um computers at a very early age got to uh program uh kind

of like using my visual basics or like writing um uh bash and uh what's the name of the windows

equivalent uh like dust i don't remember all the like i was writing like bat files and things like

that so uh so i i started playing with that there was i played video games after that and trying to

automate a few things that maybe i shouldn't have but like i um i was kind of like obsessed with

these games and wanted to like it turned into uh i like more the programming aspect than playing

the game itself so uh so yeah i had like a background in in programming before using python

I work in the web agency developing websites in PHP.

And maybe my first contact with ORM-like stuff

is that we had this in-house PHP framework,

like a lot of agency ad at the point.

And we had a database connector logic

that was mainly talking to MySQL.

And we had a customer that wanted to use,

I think it was Access or SQL Server.

I don't remember exactly, but I had to write a kind of like connector, uh, for that kind

of like port to the logic that was dealing with kind of like pagination and all of that

stuff.

So, um, yeah, that was kind of like before Python was doing mainly PHP and, um, there

was a, a framework as well that was popular before jQuery called prototype JS.

And, um, the community was not that open, but I kind of like, I contributed to a certain

next 10 was reporting bug to it uh but and i remember um going on a camping trip with my

parents and kind of like being obsessed with this thing and just printing the source code of the

paper and just carrying it with with me because i wanted to read it all um so yeah that was that

was all uh kind of like pre python stuff uh i did php and javascript i guess yeah cool cool cool

i've got this vision of you being like the girl from jurassic park being like this is a unix

system i know this uh i i didn't i i was more kind of like uh eye level uh before like i didn't i

didn't add like a lot of the focus on kind of like the os things at that point but uh yeah i i like

computer stuff and i was uh exposed to it at the early age because of the the job my my dad had

yeah and like the give the tell there is that you were already using ubuntu when you went to

university and so yeah that's what tripped me up i was like no that's not a and that's i think that's

for people learning program that's the the thing is that some people come in with years of experience

and some people it's their very first programming class and so it's sort of like if you can get

through that first year of programming most people can catch up but you don't know that someone has

you know in no other subject does somebody have five ten years of experience but in programming

that can be the case yeah that this experience kind of like oriented my let's say career choice

as well um so when i i got to college and uni i mean i was i didn't know what to do really uh

there was tons of things i liked and um but i had kind of like this this pull towards it because it

was interesting to me and i think it was rewarding as well because um let's say what was easier but

I had like, uh, at facility, uh, doing these things.

And, um, that was in college.

I went to, um, a college called Collège de Maisonneuve.

And, uh, my, I was working in, uh, my, my, uh, degree was in, uh, multimedia.

So basically we're doing kind of like, uh, drawing, uh, of like 2d animations, uh, 3d

animations and a little bit of programming.

Um, but since I had like facility in programming, it was, um, I was very interesting because

I was able to do a focus on the other things basically that, um, I eventually didn't end

up working with, but kind of like gave me a foundation in kind of like some of like

video, uh, editing and, um, uh, like the part, the part that I liked the most was really

about just the drawing.

I was not that good, but kind of like being able to spend some time drawing at school

was kind of like i was not expecting that when i was uh in high school that you could do that

it's something nice about um those kind of uh the other bits which aren't so you don't you don't

think of them as your core bits but those other bits that sort of round you out a bit it's like

make you a bit more of a fully fleshed human being yeah yeah it was yeah it was it was it

was a great time like being able to just explore uh that was kind of like the the college part uh

and yeah it was really cool okay and how did you find django then so uh the agency where i was

working at was um they had a consultant and one of the recommendation the consultant gave them is

that in order to speed up their development process they should adopt the framework because

like a lot of agencies they were maintaining the old php framework and um the consultant was

uh someone coming from a python and general show and um he was uh kind of like trying to push for

that but they wanted a an external assessment so i was there and i was kind of like this php

i didn't know much about python except for the script that i wrote for wi-fi stuff at the uni

and they um yeah they asked me to kind of like do an assessment because they had a trust relationship

with me and we evaluated both symphony and django and i decided to do a complete one ed moving from

a php shop and uh going to uh towards python and uh and django and uh yeah that's how i ended up

using it i think my first contribution was a uh translation that was not properly uh done in i

think that was even like before trans effects but i ended up kind of like creating a ticket for it

and started contributing to a few more things,

issues that we ran into.

And eventually Claude Paroz wrote to me on IRC

and asked me if I wanted to help with contributing a bit more

because I was kind of like submitting something every week.

And I guess that was how it worked at that point.

You just had to be so pesky that they wanted you in

so you could help instead of bugging them.

You just had to keep turning up, right?

Yeah.

Oh, this guy again, right?

Okay.

Well, it's interesting hearing that story

about how even when you're younger with Prototype.js,

like it didn't occur to you not to reach out,

you know, because I think a lot of people think

frameworks or software is, you know,

people over there do that and I do this here.

But as you found at an early age, right?

If you show up, it's especially open source,

like people are dying for help.

so it's that that leap that was a big one for people i find you know later in life right

because they have imposter syndrome and all these things as opposed to just doing it incrementally

and then um so yeah that's a smooth transition from you know being like oh here's this new thing

and up something's wrong like of course i'll fix it i think that's still something people

don't think about or don't know where to look or don't don't think that it could really help them

And I think the Django community and the Python in general is a good example of, like, trying to do things right and being welcoming in terms of trying to provide guidance, being open.

That's something that I think made me fall in love with Django was that, but my experience with prototype.js, it was kind of like a wall.

It felt like I couldn't, I was suggesting something and it was kind of like, there was kind of like no discussions.

And I get that Django is a large project as well.

It's possible that newcomers sometimes run into more of maybe a pushback against some things.

But I still do feel that I try to embody that to a certain extent, to try to be as welcoming as possible, while maybe maintaining some boundaries as well, because we're humans and we don't want to burn down, obviously.

So did the entire consultancy then switch for all its projects

from PHP Symphony to Django, or is it just on that specific one?

It was a prototype project, so we wanted to...

Thinking of it now, it was great.

It was kind of like metric-based.

We had two or three developers that we knew that we had similar projects

using our framework, and there was a new engineer as well,

And there, like, it took a lot of time to onboard on the previous PHP projects.

We wanted to compare that.

And even with the fact that with the language barrier, basically, they needed to write, learn a new language.

And deployment as well at that time was not as easy as it is now.

That was kind of like maybe 15 years ago now.

I don't know, like, it was a long time ago.

And we still ended up being kind of like flush while learning all of these things along the way.

So then a few more projects were added and they ended up switching entirely.

Obviously, they needed to maintain the PHP projects, but the new projects were created using Python.

The thing that you said a couple of times during the introduction, during the story there,

was that they were like everybody else at the time.

Everyone was maintaining their own in-house framework.

And I think that for me is the big one.

It's like, why use a framework?

Well, because if you don't, you're going to end up inventing one.

And the one you invent just won't be as, it won't be as good.

Yeah.

And the old kind of like onboarding was kind of like spoke to me a lot during that time

because you don't have to maintain like information like documentation.

You don't have to keep up with like the latest security vectors discovered.

um a lot of like the batteries include there by kind of like a peace of mind that allows you to

focus on not being in the business of uh maintaining uh this kind of project so

uh yeah it's a big plus uh and i think a lot of people realize that at that time kind of like the

rise of symphony rails django kind of like it was that time when uh folks started to realize that

yeah there's there's a benefit in kind of like putting all that together yeah exactly did you

did you poke around rails at the time as well because that was also the you know the other

big one did you did that come up or no uh not during the the review there uh when we kind of

like compared the technology i played a bit with it uh but i uh the part i did the most was kind of

uh it's a reverse enduring but kind of like looking at how they were doing some things just

because i was curious because i mean it's kind of like let's say a sibling of django to a certain

extent right they grew along over the years so it's always interesting to see how they're doing

things and how they're they're taking the the project um but no i've never kind of like chosen

it and use it for a project and myself so wanted to ask you about your work because you work at

zapier right which is a bit you know a big company and a big django shop and you know

can you tell us about your your life there yep um so i previously prior to django i was working at a

uh i found a startup that was um yet another email marketing platform that was focused on

new laws that were coming into effect in canada about kind of like spam laws with regards to

explicit consent that you need to renew when sending email marketing and company were eager

to be able to just comply and make sure they didn't do anything they shouldn't and there's

also this dynamic in Quebec and kind of like maybe more the eastern part of Canada around

um, uh, just body wisdom, uh, to be able to send, uh, email campaigns in both

languages, tools like MailChimp and such to do, um, a bit of that, but, uh, it

was not, we were kind of like trying to target the niche there and, uh, focus

on these kinds of like two big features that, um, users in, um, Canada and

mostly in like, uh, Quebec quarter, uh, eager to get some work there.

for five years and that's kind of like the time i started computing uh more and more to django

because we were doing things with um kind of like dynamic models uh and kind of like all sort of

crazy things that needed me to dig into like what the orm was doing or like bug and see a contributor

at the time about oh well this thing is you have an idea um but yeah so all right so this this is

the secret is basically you you get to be because you're you're one you're basically the main or one

of the main contributors on the ORM for the last few years and your depth of knowledge is just

astounding so the the real trick there is that you're trying to you're trying to push it to its

limits you know for a long period of time and you slowly create the those that knowledge that that

depth yeah I got a good interest as well in just a relational database like every time I try to

look at them i read about them i find it fascinating um the way they implement these

kind of like low-level primitives that um can be used to do like nowadays tools like buzzcrest and

you can use it for so many things when you start um there's everything in there that relates kind

of like data or synchronism uh across um um you know like distributed service so um yeah it's

fascinating to me and it was also

fascinating to be digging there

and see

the discussion around

like the expression APIs

that were those added

the work that was done as well

for NC

to try to

just make things better

that was

very interesting to me and

had the chance to be

exposed to

all sort of like these discussions

because i was invited to uh jungle under the hood for uh i think the three years in amsterdam

i was kind of like very kind of like um i felt like i i needed to give back because everything

that was kind of like uh given to me there in terms of i mean it completely changed my life

i was kind of like included in this group of of of folks that were even i'm just indirectly

be mentors to a certain extent kind of like being able to watch how they work how they approach

problems um and just maybe get things done in in a context where it's hard sometimes to drive

consensus then there might not be like a right answer there might be some stuff that are better

than others and at the end of the day we need to make a choice um so yeah that was that was the

thing that pushed me towards the uh ORM trying as well maybe to fill a bit of the void uh in this

area um because it's it's kind of like an important part of the framework but it's if there's no kind

of like keeper of the knowledge there it's easy to um maybe repeat errors of the past uh test can

only do so much in terms of telling you to not do a particular thing yes yeah no exactly so i mean i

guess that leads in how do we um it's something you talked about in your django con um keynote

in 2022 but how do we how do we encourage contributions to the ORM because this is

it is quite big and scary and it is quite complicated it is quite difficult but how do we

how do we keep it so that there can be new people coming on board to pick

pick up and pass the baton on to and keep it fresh? I think that we need to put

help with some guardrails I think the code base is once once you wrap your air around it

um there are obviously parts that are more incendiating than others for example things

that relate to joint pruning and things like that these are kind of like non-trivial to reason about

but most of the time you don't have to go that deep um there's two layers and they could be

better documented something that i like i mean three things that i mentioned at the end of the

talk was um a form of like mentoring um i was uh i've been kind of like looking at the site about

what like the projects like django nuts are doing trying to maybe find a way to contribute there and

chime in find a way to mentor developers that might be interested to digging to that so

that's on my kind of like 2024 to-do list we'll see if i ever get to it but other things we could

do that i i was against at first because of the burden mind post but any form of like typing as

well in terms of like self-documentation i mean i guess it could help because if you add like your

id and tools are so good nowadays in terms of like generating codes or like guiding you to a certain

extent that you can get 90 percent of the way there and that's enough to kind of like just start

wrapping your head around um all of these pieces and um obviously documentation but that's a large

effort and it's it's a there's a tax with documentation if we need to like to keep it

up to date it's one more barrier to kind of like trying to change things up in this area yeah so

you said you said a couple of things in in there one you said earlier on dynamic models let's come

back to that in a sec you just said talk about typing is do you see that a way of incrementally

introducing typing into django yeah um i think that in like the past releases uh we've we started

to maybe like the past four or these five years we start to kind of like document change to the

um database uh backends change even if like if they're not documented we're still pointing like

oh if you use this particular method uh you might want to do that so that might that might be one

area where you start typing it um maybe trying to incorporate uh things from projects like

Django stubs could be one. I mean, a lot of the work has been done there. To me, the immediate

benefit in terms of maintaining the RMTL would be more around adding it up at the lower level.

So more in terms of-

Inside.

Inside, yeah. So we can at least kind of play with it, see how much of a burden it is,

setting up CI pipeline as well. And MyPy provides other solution for enforcing typing

provides all sorts of bells and whistles to do it in an incremental way because it was developed at

the point where there was pre-existing code base that didn't have typing. So from my experience on

other projects, at Zapier for example, it works pretty well, the gradual typing. Sure, there are

new things that you need to learn about. Typing can be intimidating, partially in the case of

Django, where we do some meta class magic in some cases, so you have to

It's not as easy, but I think there are real benefits there in terms of self-documenting

code.

Well, let me ask a question.

So Zapier for, I think, seven years, what kind of projects do you work on?

So you've mentioned that they're very supportive of you doing open source work, but is it like

one big project for multiple years?

Do you have different ones?

Like what does the day-to-day look like?

I would say it's kind of at this point, it's more like multi-month project where I'm more

kind of like try to, um, support many teams in terms of features they're trying to deliver,

uh, engineering problems they run into. And there might be, um, a couple of times where I,

I have to write a piece of code because it's, I am, uh, I, I have knowledge about many systems

and just getting the teams together to do it would be, uh, hard. So sometimes I just kind of

create proof of concept or actually deliver let's say that like eye impact code that um allows for

um the unblocking of teams or prevent like lifting dependencies between uh between two teams so

um otherwise a lot of mentoring a lot of uh trying to come up with the uh a way to drive alignment

within the company around the technical vision is something i've been working on uh in the past few

months so uh that's changed a lot when i joined i was uh my title was product engineer and i just

kind of like revolved around the uh workflow uh part of the product which is like very interesting

again it's kind of like the let's say the equivalent of the orm for for zap here it's just

fascinating this kind of like huge distributed system that you can just throw a workflow

definition at it and um you see you want to trigger something and it's just it does its thing

and it's uh resilient in the face of like the thousands and thousands of apis that can return

very various response code and you need to make sure that some things are idempotent and like

it's it's very interesting from a software engineering perspective so it sounds a little

bit like a software architect would be like a title i've heard for what you're doing and that

you're not directly managing people but you're overseeing large groups of engineers and providing

guidance and occasionally coding but largely helping them figure it out on their own is that

accurate it's a yeah it's a it's a it's a good definition i think it's a it's about that i think

it's a um i i wish i could code more and i think that django is the kind of like escape from that

that i realized that um the more i code the like and others as well and they try to make it

explicit to me because i i like to think obviously but uh i can have way more impact by empowering

uh engineers developers within zapier to actually um ship code themselves and kind of like be a

vector uh of that it's it's you know like the the uh the writing code and shipping it and seeing it

deployed like all this machine the reward cycle is so immediate that it's it's yeah it's easy to

get drafted there while you can you can be way more impactful doing other things well that sounds

a lot like um so andrew godwin uh has also most recently had a software architect title and it

seems like that's about as good as it gets as you progress in a company because companies want you

to manage manage people but if you want to still code a little bit and have impact but not be doing

as much managing managing the software architect role is the way to go but then you need yeah it's

almost like you need open source just to code code or code for fun because you're removed from

it it's a it's a weird thing right like there's no other maybe because i didn't grow up programming

there's most fields you just do what you're doing and you get better at it and you just do it but

coding and i guess most engineering fields very quickly three five years in you're just really

pushed to provide guidance rather than do the work yourself so it's an interesting interesting

dynamic again not coming from a programming background growing up but sounds like you're

managing it well yeah it's uh it's at first it was hard it's kind of like like explicitly extract

myself from there but kind of like as i said the reward the reward cycle is is not immediate it's

kind of like longer now but when you see these things happen even if you just had like this

indirect impact on you know like putting this little cog in uh this whole delivery thing it's

it's so much rewarding to um to see that you you've kind of uh you've you've helped uh still

to a certain extent because that's the thing that i believe is rewarding from even for django just

feels like you help to a certain extent uh someone with the problems they're facing and

just the the impact of it as well so yeah it's different reward cycle but still um very rewarding

but you do have quite a lot of availability for django it seems like you know when there's

an issue you you're always you're always there to to help field it so zapier must be supportive of

your contribution yeah the the feedback i've had was basically as long as it doesn't affect my

my kind of like performance at work it's it's okay and um so yeah i do i i like i like the

jingo stuff i like the aura and things so i gravitate towards that um and sometimes it's

just kind of like just a nice break of um some things the problems i'm dealing with at job are

more kind of like artisanal so it's like going back to jango is it's just a nice way to kind of

like clear my um clear my mind and just maybe put things in perspective as well i mean so from

dapier's point of view there must be two sides to it one is that obviously your knowledge of the

ORM is an asset to you know their ability to maintain a product and your ability to maintain

the products as a team but there must be so and simon gets a mental health break as well by you

know contributing a bit well i'd say number three is also simon anyone who works with django enough

is going to come come across you and it's another it's a recruiting tool right like i mean i would

think that'd be a big draw to work with you if anyone's interested in django right i would if i

was a manager if i was a company owner recruiting is always the problem uh so i think it's hard to

quantify which is sort of the issue how do you quantify someone's time you know on the steering

council or the security council but uh you know if i was looking to apply for django work

if someone was was a you know very involved that would be a huge draw yeah it's um yeah i think

there's a yeah obviously there are kind of like benefits for zap for zap you as well uh so i think

that's one of the reasons why they're motivating uh they're motivated by that but it's it's great

to have this flexibility it works well for me um so i'm i'm glad but yeah you're you're right like

it's i've i'm able to happen elsewhere sometimes non-trivial uh questions about how things should

be done or um we have like a very large uh mono repo uh like a very like old django project like

created like i know like 12 years ago now and uh it uses like a huge mysql cluster and tons of

other databases so um there are stuff that come up that we need to come up with solutions that are

non-trivial and having some knowledge about the rm certainly helps okay so i think zapier is

primarily mysql is that correct um the project that was created a long time ago uh that's kind

of like the, let's call it like the mullet,

is MySQL and Postgres.

So we use it for like all the historical models are there

and all the new ones,

we're kind of like trying to break the mold

with kind of like add each domain use a separate database so we can kind of like maybe stop its

growth to a certain extent um because you start to run into yeah issues with um team boundaries

uh the kind of like domain violations that uh django kind of like allows you to

basically create a foreign key across like everywhere um i saw i heard you call to the

the previous Django chat talking about the old kind of like user slash user profile thing as

well as something that we kind of like go went back and forth about because the project was

created before a custom user model was supported so we've got this model that when Django switched

to adding custom user model it felt like an atrocity but the more I think of it I think

It's great that it's kind of like separated.

So I'll just say that I support you on this front.

I feel like they're pros and cons, basically,

because these models can grow very large.

That's it.

They're pros and cons.

They're trade-offs.

And it's not like it's clearly the case that, you know,

but in my experience, many much smaller scale projects than Zapier,

but it becomes a dumping ground.

comes up let's just throw another field on the user model and then that's fetched every request

it's like no this isn't this is we're not encouraging a good pattern here i think is

the bottom line and so there may be some projects where customers you know don't get rid of the

ability to swap it by all means but as the recommended path i just i don't know and again

it's as you say trade-offs but it's so it's so um well documented this i mean so i'm giving a talk

in two days at Django Boston, largely just cribbing Carlton's ideas. So I'm going to give

Carlton credit about the Django user model. And I mean, 12 years ago, Russell Keith McGee put up a

whole wiki with the five, it's more than five options when they were talking about what to do.

There was like one, two A, two B, two C. And even then, you know, just pros and cons. And, you know,

I think Jacob at the time was sort of in favor of option five, which is more what Carlton's going

for with just like just uh um basically just an identifier um but you know there had to be a bdfl

benevolent dictator for life decision and they chose um i guess the custom user model approach

so it's just fascinating that it's not anything new and it's documented they had to make a decision

and and russell's talk he gave a jango con about it that when it was introduced yeah red user blue

user yeah and it's a great talk and you're nodding all the way through and you're like yeah that's

right and i i think it's but you know late this coming back to i'm like let's change it anyway

anyway that's a thing simon i wanted to ask you about oh unless you've got have you got something

to say there something oh yeah i was just uh i think it's a idea of swappable model i think it's

great i mean like users have been asking that we lean in to that a bit more i think the migration

framework is ready to a certain extent and drew spent and all the contributors around the migration

stuff as well, spent a lot of time

making that better over the years.

So I think there's

value in kind of like

providing abstract-based class.

And in most

cases, having a single user model

is going to be fine, because you're not going to end up

with like 50 fields

that makes the tuples and

database so large that fetching

them for every request is going to be problematic.

So I think

it makes sense, and I wish

we were leaning more of it towards

swappable models uh although that people not everyone likes them but i think they're they're

clear idea one thing i wanted to ask you is talking about migrations is is i have i'd like

a kind of concept of sort of bolt-on migrations where kind of an optional one so that i've got

this package that um just um for makes the use email field unique on user on user and all it

does is it comes along on the app installs it adds a constraint and then it's got a migration

that you run one time and it just all works smoothly and if you wanted to uninstall it you

just reverse the migration and you know pretend it was never there do you think it would be feasible

to do a bit more like a bit more along that i mean it's a kind of a horrible monkey patch just

a proof of concept that you can do login by email very simply with the with the default model but

do you think there's a possibility of kind of optional migrations that third-party packages

could provide say look if you want this do that um i know there's a way uh i guess you could have

like different app configs that possibly uh so we've not done that extensively but i think the

idea when uh amric worked on app configs would be that there would be different kind of like ways

you could install the app uh you could install it multiple times if you wanted to with a different

label by subclassing it uh and it would have kind of like a different label i don't think uh a lot

of package leading to this idea but i think that was the idea behind the uh some of the refactor

there so maybe there's a way there by using like a separate app config that have a different

migration module or something like that um but the the challenge still that will always be there

is that if you start applying migrations for one path and you want to move to the other one

then how do you come back yeah yeah the migration framework is likely going to trip on something

uh because it's it's that design in a way to when you can in fact remove pieces that easily

uh from from what i've seen it has to sort of be off from the main trunk it has to be clean apply

and clean unapply otherwise you're going to go into a world of pain yep quite quickly okay i

just wanted to get i just wanted to get your first take on that as to and yeah that's my my

hot take uh yeah so god i'm going to ask you one more thing about hot take then i'm going to let

will have a go i will ask you about um sequel light because there's been some changes recently

that were going to come in and i remembered um after i don't know jango con something or other

2022 we were just talking because um simon willison had done some benchmarks on you know

using sqlite and i know um tom dyson has spoken about using it in production as well and i remember

you sort of saying ah but you know if you need access from multiple servers and multiple front

end servers and things like this so i'm going to ask you about sqlite and you know your take on

that and then the recent changes that have come in yeah so um i mean that's interesting kind of

like the evolution of sqlite and its recognition just within the industry in terms of like this

like very solid tool that you can basically write to this faster than and organize it than you would

otherwise for trying to do your own file system interaction so i think it has the perception of

sqlite in the industry has changed a lot in the past 10 years where it was previously more of like

a i wouldn't say like a toy database but more something that you would use for development or

So on to a moment now where folks are building solutions that use SQLite in a distributed manner where you can have multiple things that are writing at the same time distributed amounts.

So the game changed a lot with regards to SQLite and there's a lot more interest in trying to use it in scenarios that you would have possibly used more like something like Postgres or MySQL in the past.

So the benchmark that Simon ran were very interesting in terms of seeing how far we

can push it.

And at the time, there was some discussion as well in terms of tweaking some mode there.

It was something that relates to how transactions behave when you open a transaction in terms

of when the lock is acquired.

And a recent change that was merged in

was to allow to specify through a database option

how do you want it to behave, basically.

So previously, it was working.

When you open a transaction,

there was no lock that was immediately acquired

because you could have a read-only transaction.

In this case, you don't need to have this contention.

But the moment you try to do a write,

And if the database was already locked by something, you would get an error.

And what this thing is, this new flag is allowing you to configure is to have the lock be immediately acquired, which can have like some impact on contention and is not going to be working well if you use the atomic request setting, because that obviously opens a transaction for every request.

So even if you're only doing reads, which is the case for a lot of web location, you're going to run into issues.

But if you want to do a lot of,

if you have like a difference

and are going to try to push the limits

in terms of delivery,

that might help

because otherwise you might get errors

that are, you basically have to litter your code

with try statement

and link possible exception when doing writes

because the database is locked.

So, and there's something else as well

that's being discussed

with regards to changing the write I add log

or something setting.

I've not followed that one that closely,

but yeah, two settings that should make it easier

to use Django with SQLite to a larger scale.

So Zapier will be moving to SQLite?

I don't think so.

We used SQLite for something that was very interesting.

We moved away from this solution,

but we have a system that injects webhooks

that are coming from all over the place.

then obviously you need to make sure that you never lose a webhook, right?

Because if you do, then Zap might not trigger

and the service might not be smart enough to kind of like retry

if it runs into a 500 or something like that.

So one thing we used to do was to,

we used to mount volumes directly on these boxes and pods

that were ingesting these requests.

And we were writing it kind of like another log in sharded

by many, I believe, SQLite databases.

So we would, if we, in the catastrophic cases

where the queue was down or something like that,

this thing could keep ingesting

and we could open the machine and run a script

that would open these thousand, thousand, thousand databases

and replay.

So kind of like a write-to-add log

in front of a queue written in SQLite.

And that worked very well for us

until we moved to another technology eventually.

Okay.

Speaking about APIs, I need to ask, have you played around at all with Django Ninja?

Have you had a chance?

Yeah.

So what do you make of the current API landscape in Django, right?

Because there's Django REST Framework, which is great, pretty feature complete, maybe not so active.

And I hear a lot about Django Ninja.

How do you analyze that situation?

We had this exact debate a few months ago in terms of what should be used.

New projects that are spin up at within the organization, a lot of them use Django Ninja

because like most of the time when you spin up new projects, they're, I would say they're

going to be prototyped, but they're like, you're going to try out things.

And this, the ease of development around Django Ninja, developers like it.

They feel like they're more productive using it than Django REST framework.

For large projects, though, we haven't run into a point yet

where we need to kind of like we miss the batteries inclusive of DRF.

The large project that we're treated with them use it,

and even if sometimes it's very explicit in terms of like,

not to say boardplay, but there's way more like bells and whistles

with Django REST framework that you get with Django Ninja.

In these projects, we keep using Django REST framework, but in the new ones, a lot of them are opting for Django Ninja instead because it works better for them.

And we're eager to kind of like try things that makes development easier for some of these projects.

And a sort of related question there is about the kind of the serialization part of it.

like the explicit serializers

which sort of were inspired by the Forms API

versus the kind of newer Pydantic-inspired approach.

What's your experience there?

What are your thoughts there?

I really like, you know, like,

I'm more of like the DRF person myself

because I've used it more

and I know how to find my way around it.

But I think that the appeal of Pydantic is good.

I mean, people like it.

But it feels like Django was developed in a different time, right?

Like with metaclasses, there was nothing of the nice DSL that you can get today with typing.

So I get where it's easier, it's faster.

And if we're honest here, Django was developed today with the typing things.

Things would be very different in terms of what the DSL looks for model definition.

um i get why uh people don't want to pay the tax of um well this software was developed when

typing didn't exist so i still need to kind of like um do these things and i mean even with

projects like deck jangle stubs and such you don't get as much um help from your ide and things that

you get from pidentic because i mean the typing is obviously easier to interpret um so yeah that

that's the feedback I've heard from, from, uh, folks that prefer to use it,

use a Jekyll and Jekyll.

Okay, good.

So the last question I wanted to ask is about the security team and the

steering council. What, what do we say about these? I feel like, well,

let's start with the steering council, right? Cause this in many ways, um,

replaces the benevolent dictator for life and it's the most prestigious

position to have. And yet, like, so what, how has it,

How has it been being on there, and then where would you like to see it do more or less around making decisions like typing for Django?

Yeah, I think there was maybe a misinterpretation of what the role was.

There was a lot of discussion about the technical board should get involved more in that, drive the technical vision of the project, and so on.

while it never really happened over the years,

I think it was first interpreted

has more of a just replacement for the fact that we didn't have kind of like more um didn't have

like the core was dissolved and we needed some folks to be there to kind of uh break the uh

status quo in terms of like we need to take the technical decision here so kind of like ping us

and we'll be able to provide some guidance so um i'm seeing how jingo kind of like organically

evolves and kind of like try to add a few things i'm not sure like the involvement from the

technical board in terms of like driving the vision would would be nicely beneficial i do feel

like it happens organically to a certain extent and trying to like the features that are going

to be highlights uh are kind of like self-imposing to a certain extent and do we really need them uh

in terms of like do we need we need like a shiny feature every release uh is jango at a point where

we need to kind of like reinvent things um over kind of like being stables and keeping up with

the latest development um so yeah and it's it's interesting because if you look like other other

projects like for example rails and such they they're very much driven this thing in terms of

like developing new things and new paradigms and so on and django is has not been that um and

is it okay um feels okay to me to a certain extent to have django kind of like take this position

do we fear i don't i don't think it will make django irrelevant by just keeping at its current

pace is what i mean i think we've seen that with like templates to make one example right the fact

that django hasn't been changing so much means that now with htmx it slides in nicely right

whereas django could have easily gone off in these different directions and now maybe be questioning

that so that's just one example that comes to mind about being a little slower maybe pays off

yeah and there are tons of projects like for example like just the async uh the channel

carton you work a ton of that uh with channels and such there are these kind of like new paradigms

that are being included and and we slowly get there and even if the technical board wanted to

kind of like say like oh we need to guys like go fully async or we need to do this or that

i mean we've got fixed kind of like release dates um it's all contributors uh benevolent

uh folks that are there i'm like is it going to help kind of like trying to put these things it's

going to strain the community even more uh in terms of trying to meet these objectives um i feel

like it's the way it happens is it's kind of good like if it's ready we're going to make there yes

there are some features we're trying to get in and focus more because we want to uh put an

highlight there but it's it's happening at more of a sustainable pace than uh than i if we were

to try to set these objects at least that's that's my that's a take on it no i think async's a good

example in that it's it's it's taken quite a few years to go from nothing to where we are but it's

actually it's quite mature now and it's it you know it's not the fully async you know brand new

web framework if you want one of those there are those to choose about but if you want to add

async to your Django project it's really there and it's good and the streaming responses and

disconnect handling we've added and you know all these things they're great they're wonderful

and yeah it's matured over time so that's so I just one more I just so the security team which

you're on I just want to shed some light on that because that's often held up I think rightly so

as one of the gold standards that Django is fantastic with security so if I'm a developer

and I find what I think is a bug,

I email the security team.

And then on your end,

what does that process look like?

So we review the report.

We try to kind of like assess

whether or not it's a false positive,

whether or not there's an exploitation vector there.

And assuming there is,

we will let the submitter know

and we will be working on our side

in terms of providing a patch

or with the submitter in terms of providing a patch the problem.

In some cases, if we look at the recent ones

that we ship security corrective for,

a lot of times it's around regexes,

things that look benign,

but if you start pushing crazy user input in there,

they start to misbehave.

But yeah, that's the process, basically.

It gets submitted, and before we release anything,

we submit a request for a CVE to tie it to the release itself

and the commit message.

And from there, all the supported version of Django

are going to get backports,

and assuming they're affected by the vulnerability,

and the other is going to be credited.

But yeah, what the security does basically is look at these emails, make some assessment,

and from there determine, is this a problem or not?

Sometimes we get reports for Django-related projects as well.

I know that in the case of Django REST framework, we had some recently as well.

And we try to redirect that to the folks that are involved with these projects or try to

see like, oh, is there any other part of Django that might be affected by a similar problem?

But yeah, that's about it. There's also kind of like a recent discussions about the inclusion, how we can get more folks involved in there and ensure the parity of the team going forward. I think Carlton, you can talk more about that, the push there by Michael Manfrey was about possibly using the pipeline of new contributors as a way to get more people involved with security.

yeah i think um my thought that i threw in there was that the triage and review team might be a

nice onboarding route there because to there's a kind of there's a certain trust element in being

on the security team in that it's it's confidential and so it can't just be oh i want to be on the

well you need to be known and trusted so the triage and review team is a nice on-ramp for

that you've already been around the django project for a release or two and you're known and

that might be a nice way of doing it but i think again it's important that we think about the

stability and about the continuity of of the security team as well as you know just as much

as the rest of the django ecosystem and project um so that's uh but thank you what you know one

thing simon we have to say thank you for your service there because that's you know it's an

it's an it's a thankless task at times thank you both as well thank you driving the community like

that i know all people in django get very excited about just listening to django chat and uh seeing

you just revolve around the community as well you've been just uh yeah uh stars of django

well they should be excited about you know the security team that's that's the thing is that

the real work is happening in the dark it's um it's black hole yeah i think work is happening

everywhere uh some of it is uh more more facing but uh yeah it's it's awesome to know that there's

it feels great to be kind of like in this distributed um entity and seeing it uh kind

of like continue uh continue to work evolve over time uh obviously there are things would like to

evolve faster and so on but it's it's great to see it to see it grow basically i've been kind

like revolving around it for for more than 10 years now and it's uh yeah it's it's an awesome

project okay so we're coming up on time simon so i i probably a couple of weeks ago we had um two

fellas from uh pin planet which is a an app built with django and andrew there's the expert and he

he asked i had a question i promised him i'd ask you which is he's from an sql background and he's

used to you know working in raw sql and then when you jump to the orm and he's very um capable and

pleased to be able to construct equivalent queries in the ORM, but his point was they

often don't look very much like the underlying SQL that's generated. Perhaps you have to use

values in a weird way in order to get the grouping right. Is there any way that we can

make the relation between the ORM query and then the SQL more transparent, in your view?

The big challenge with the ORM and something that SQL Alchemy doesn't have, well, it has, the ORM query builder doesn't have this intermediate notion of tables.

So everything relates to models.

Sure, there are kind of like aliases that represent tables.

But one thing that SQL Alchemy has is that there's these two kind of like layers.

Like you've got the kind of like table representation.

you can bind models to them and you really have a query builder you have like a lot of flexibility

in terms of like you can use tables directly and use like um exist clause here kind of like this

join here and so on and you can then kind of like pipe that into a model representation

the ORM doesn't have that the the kind of like compiling part is every couple to the notion of

a model which make things like um any form of like union or addition of like eventual um

common table expression the width statement uh in sql or things like um uh there were discussion

about a sub query wrapping and things like that um it's very there's a fine line in terms of like

providing very high level abstraction while still allowing low level injection of logic

and that's i believe the big challenge with uh jangona ice you look at like primitive that were

recently introduced like um filtered relation for example that tries to kind of expose that it's

it tries to do it and it it for complex case it fails to a certain extent because um the orm

very much needs to be able to

know what kind of join it's dealing with

with regards to aligning them

and optimizing them and

making sure that

the query still makes sense in the

context that you make so I think that's

the kind of like biggest challenge in terms

of providing more flexibility

to folks that are maybe have a DBA

background or very familiar with SQL

and want to turn DRM

into doing what it wants so that's

that's a big let's say problem

or design issue with the current state of things.

I know NC wrote up a few years ago

that might be beneficial for DRM

to be built on top of SQL Alchemy

so people could drop back to just doing these kind of things.

But that's too big of a right for me.

I'm not ready to venture there

and support this multi-year venture.

Right, okay.

So then I guess the question is, if you put that on the side as something that we're not going to do, do you have a kind of magic wand that you would, you know, if I could fix this, I'd fix that.

If I could change this, I'd change that.

Oh, yes.

There are tons of things.

I wish we were doing a better job at kind of like buying more into removing a bit of like the extra stuff.

So today, most of extra is obsolete.

what one thing that remains is around the um so of course extra allows you to do a few things like

select new columns uh but ordered by or where um and there but all of that can be

uh emulated using raw sql the expression itself um you can like import from django dd models raw sql

and wrap it and pass it directly to filter and order by and the fact this is carried around all

over the ORM makes it, and the fact that it's not that well supported in all cases makes

it very awkward to use.

So finally pulling up the Band-Aid on Extra, deprecating most of its options and coming

up with a nicer way to do reference to tables, either being through joins, common table expressions

and such, I think could, could, um, push David even further in terms of like what is doable

without going through a massive refactor by introducing an intermediate, uh, team of

representation.

Okay.

Brilliant.

Anything else that we didn't ask you about, or you want to plug before we sign off?

Um, no, that was great.

Uh, just wanted to say that, uh, it was, it was great to be here.

It was great as well to listen to you all.

it's great to be a part of the community and um if there are uh again i want to reiterate that

commit to it i i want to kind of like get closer this year to either like django uh django knots

or any form of like similar projects to find a way to kind of like uh the form of mentoring if

you need you have questions about the orm uh about the pr uh uh pull request you can you can ping me

on the on the on the on track as well i do my best to try to support you through that and um

i i learned about the orm by just trying and breaking things and moving things around so um

it's only a matter of like trying i guess so it's it's a lot of stuff but it's it's certainly

manageable once uh you you you play a bit with it okay awesome that's very sure thank you for

coming on i it's truly when we started this podcast one of the guests that simon uh that

carlton wanted was you you were like this mythical figure that hadn't been to like recent conferences

and worked on the orm and so you know to meet you at jango con us two years ago and then now that

you're on like you were like one of the top five people carlton was like we need to get this person

on so we started this only took me 140 episodes to pluck up the courage to actually invite you

yeah i was i i'm glad that i'm glad that you uh you invited me it's glad to be here uh glad to

be here and regarding conference possibly i'd be able to be more present i think that the past

years like i had like another surgery last year and other things it's just uh and i played ultimate

frisbee before during the exact season where all the the conference were happening so it didn't

work out but uh yeah looking forward to possibly meet you in person uh at the next events brilliant

well thank you um so everyone we are at jango chat.com and we'll see you next time bye bye bye