Transcript: Python Tooling - Hynek Schlawack
Hi, this is Will.
A quick note that my website, learnjango.com, now supports free tutorials and premium courses, meaning my three books, Django for Beginners, APIs, and Professionals, are now available.
I'm running a 50% off Black Friday sale until December 1st.
More information in the show notes and later on in the show.
Hi, welcome to another episode of Django Chat, a podcast on the Django Web Framework.
I'm Carlton Gibson joined as ever by Will Vincent.
Hello, Will.
Hi, Carlton.
Hello, Will.
And this week we've got with us Hinnek Dwalek, who's um Python core contributor and all-round, well, you know, big, I'm a big fan.
So hello Hinnick, thank you for joining us.
Thank you for having me.
And I love how everybody has problems to peg me anywhere because I'm so all over the place that uh it's just this dude in Python and we love him.
Yeah, yeah, yeah.
No, that's
exactly it.
Because I was gonna say you're a Python core team, but you are you are on the Python core team, but that's not where you hang out mostly these days, right?
Yeah, I'm not, sadly.
Like my time is finite.
Are you more on the Go side of things, or you're still equally in the Python world, or how how would you divide that time?
No, no, no, no.
I'm I'm on the Python side.
Okay.
But a vast majority.
Like we use Go only uh when we need something that is faster than Python, which is very rare.
Like Python is defined for us most of the time, or when we need a single binary, which happens more often when we need a set UID or something like that.
But otherwise it's all Python.
Okay.
And so I'm just gonna get jumped straight into there because um you talk about being past faster than Python.
Um now there's this whole Python 3.13 and free threading and things like that coming along.
Is that gonna make a lot of difference?
Because I'm quite excited about it.
I'm like, this means I can stick with Python, I can do my threading, my template rendering in a thread, and it won't crack slow down.
But what's your view?
Because you're a bit more closer to the metal, I think.
Um I think I'll give you the most senior software engineer answer that there can be.
Uh, it depends.
So good.
Yeah, because
let's be honest, we've been doing pretty fine for like 30 years without free threading, right?
So um I am absolutely pro free threading.
I love it.
I'm very excited.
I was one of the people advocating for uh this experiment to start.
Uh but you need um a parallelizable problem to take advantage advantage of parallelization, right?
And uh people are also not great at writing parallel code, which will probably change because um in Python we do not have good APIs for parallelism with threads because it doesn't make a lot of sense, they are just not great.
Um at some point you just run into the gill, so nobody ever took the time to like build really good abstractions for free threading.
And this is something I'm really excited about when we will build that.
But uh I don't enjoy I don't expect any world-changing uh things happening right now or for most people, but I believe for many people it will be life-changing like when they have computational things, right?
Like the data science people, I think they will be much happier than us.
Although their stuff is already usually in fault random C and uh doesn't have legal, so we will see.
But um it is
okay, and so to be
clear, it is necessary to future proof Python.
I think that is true, but I don't expect that uh our code is suddenly gonna be super fast or something.
It's it's just we will see.
There will be an evolution.
I think it will be more boring than most people think.
Okay, so no no instant new world.
I was gonna ask them, what about the sub-interpreters thing?
Because it it was all sub-interpreters, sub-interpreters, sub-interpreters, then suddenly suddenly free threading came out of the blue, and then I've hardly heard about subinterpreters for a while.
Yeah, I mean, uh Eric Snow's still working on them and talking about them, and I think they are also exciting, and I think they are orthogonal to free threading, and I think if they ever reach a certain stage and don't get like the whole all the oxygen uh pull out of the room, they can also be very interesting in the future.
But they have some limitations, right?
Like you can only exchange um uh um certain types of data between subinter sub-interpreters while you can change uh you can share anything using threads, which is both good and bad.
So which which
which is of course the problem with that.
So there's been this this presentation by Yuri Salivanov at LastPyCon, where he talked about this amazing um I don't even remember how the proper name for that is, but it's these are the structures that are uh that don't change, but you can change them in a way that they remember the changes.
So you're basically applying patches to dictionaries and to lists, and they're that makes them frozen, and you can share them across uh barriers like uh subinterpreters.
I think the correct name is STMs or something like that, but uh didn't go anywhere yet.
It's still not public.
I think it's in their asyncpg or in their um edge db report somewhere, but you you would have to copy it out.
So I guess my one more question in this sort of way, then we should cut back to the beginning because we didn't even ask introduce you or ask anything, we just jumped straight into performance.
Is do you think then that these these over the medium term these techniques uh will keep Python competitive with you know the goes and the rests of the world?
I mean I
mean this two questions, right?
Uh what is medium term for you?
And Python right now is the most used programming language in the world, right?
And the most popular one.
So like asking if it's competitive is is kind of weird.
Um I mean But
there is this thing at the moment where it's like if you don't write it, rewrite everything in Rust, the world will end.
And I know that's what we have that every time there's a new technology or a new option, but yeah, you know.
Yeah, again, I I think it's it's important, it will enable new things, but it is, and I said it before when we were like advocating for these changes.
We do not have the fantasy right now or the imagination what we could get out of this because we don't have it, right?
Um, we cannot imagine from the primitives that we have now what could be.
We will have to do, we have to try it, we have to play with that, and then probably someday out of some company will come some amazing framework that will make everything easier and everything will just fly and everybody will be happy.
But um, I don't think you can expect like there's gonna be something in a year or something.
There is gonna be prog some nice things.
Um, I think async.io works a little bit better when you have free threading and like things like that.
But it will take something big that will add an abstraction that people love to use to be an actual change.
But again, it's great for the future, right?
All right, I'll ask a question.
So there we have a lot, we have a big Google Doc of things we want to discuss, but I wanted to make sure we ask about your day job, which is at is am I Vario Media?
I'm probably saying that wrong.
It's it's not a real word, right?
So we are German companies, so we call it Vario Media.
Okay.
Um nobody's getting offended if you pronounce this English.
Yeah, we are a web hosting company, we do email hosting, we are domain registrar, and uh we run a kind of tight chip with not a very big technical team, which informs a lot of the things that I do and say.
Like I cannot afford having um janky systems that page someone every night or something like that, because that someone is always me, so I cannot just uh push something to production and hope someone else will have to deal with that.
And uh yeah, we are only serving the German-speaking market, so uh that's why the probably not a lot of your listeners have heard of us, and our hosting is mostly LAMP.
So for the web hosting part, like exciting times right now with WordPress.
Wow, you I can't I can't if you go if you go a whole cycle.
Come on, you you can't you can't leave us there.
Yeah, how is uh is that a model for you know the Python and Django worlds, you think?
Have a dictator who goes crazy.
What do you mean?
Have a dictator who goes crazy.
Yeah.
I mean Guido was apparently smart enough to just abdicate before he can get crazy, right?
It seems like it just maybe it's almost I mean, that's the thing is that Matt had a reputation, deserved or not, for being relatively sane and uh you know, good in quotes person.
And so it's yeah,
I remember early Mastodon, he was like this he was kind of celebrated, right?
Like you took over Tumblr and uh integrating things into the Fediverse.
Right.
I mean, give it enough time and enough money and power, and I guess this is the the lab experiment playing out.
Yeah, yeah, yeah.
Hell hath no fury like a billionaire scorned or something like that.
Right, I know.
Well, it's I mean it's interesting in the Django space for you know Wagtail, right, which is has been a big thing for a long time, but especially as people are starting finally to think maybe WordPress isn't isn't all it's you know, maybe we should look beyond WordPress.
I'm curious to see if that brings more more momentum to things like Wagtail.
What do you Carlton, you're making it?
Uh the thing with WordPress is that it's PHP and it's dramatically easier to deploy than anything that will ever be with Python.
So um you can just drop a bunch of files on a server and it works, and this is why it's so successful.
Like uh you cannot win the war against this one on technical merits, like with with feature lists or something.
That's just not gonna work, I think.
I think that's it's more likely that that's gonna be our open source, a new open source fork of WordPress.
My sort of experience with it of trying to sell alternatives was always it was about plugins and themes.
It was you know, people didn't care whether it was PHP or Python or or Ruby or whatever, but what they cared about was plugins and themes.
And uh
and only because I didn't know how much of a pain it is to run with the
slow because
well I'm I'm curious your so your your day job, I mean clearly it's interesting and challenging to you.
Like what are kind of the things that keep you engaged with it, right?
Because you could be doing we're gonna talk about all these open source things that you're doing, um, but of all the challenges you could take, like this seems to be one that you seem to you know enjoy and stick with.
Um I'm just curious if you could talk about running a web hosting kind of company, because that's something a little different than most of our guests are working on.
Um I mean, my open source projects are results of my work at the web hosting company because I don't do open source just for fun.
Like these are projects that I need that I use myself because I don't have time to maintain software that I'm not using.
So this is all our stuff that we use.
Um so uh you can see by that alone that my challenges are quite wide.
Yeah.
Um this is probably the reason why yeah, this is probably the reason why I've stayed there for over 20 years, is because uh thanks to the fact that we are a small team, um I have a lot of different jobs in a way.
Like I I don't think I could work in a company where I just work on one one Django application uh eight hours a day.
So this is not my life.
I I'm I have a wide variety.
I mean there's some Go, uh I write um web services, I I write async IO code, like all our um domain registry thing is done in async IO and uh works pretty well actually.
So yeah, I I wear a lot of hats and I can kinda sort of decide myself when I change them.
So there's a lot of freedom.
I mean, there's a lot of pressure and responsibility because again, if something breaks, it's my fault and I have to fix it myself.
But uh I have a lot of challenges to choose for from, and of course, there's this thing that I the thing that I I do, I am the best at the company.
There's nobody over me.
I cannot get promoted or anything like that.
So so I've hit the ceiling here, so which is why I have this understanding with the company that uh they let me work on open source projects and engage in this community.
That was like literally what I told them to either you give me opportunity to learn from others there by serving a community, by having exchange with the community, or I will have to go somewhere where I can grow professionally.
Well, and uh yeah, here we are.
Yeah, that kind of autonomous kind of autonomy is deeply rewarding, right?
That's it is it is it is.
I'm a big fan for like uh these uh slow productivity books from Carl Newport and those things, and just like this career capital that you can build up, and um it is not that I'm working less, it's it is just that I work more on things that interest me right in the moment, right?
Like I could take like two or three weeks off and uh transfer our projects to UV and just do this deep research of UV.
I don't like if I don't have a deadline, I have time for RD basically.
Yeah.
On the other hand, nobody else is doing the RD.
So this is the other side, right?
Like what I don't do, it doesn't happen.
Well, that's sort of the dream, right?
I mean, Carlton, for you, Carlton, right?
Like coming off of being a fellow for five years, speaking of autonomy, you know, look at what you and Mariush are doing.
Like you're not going and being engineer a thousand at Fang, right?
Because it just doesn't translate.
No, so yeah, I'm just working on um a single Django app at eight hours a day.
But it's just
you, but it's just you though.
But yeah, but it's just it but it's me, and I get to choose it and I get to set the the the thing, and I'm a you know, it's it's not a 10-year-old project yet.
It's so I'm just you know it's still very much fleshing out, very much enjoying it.
But it is you know me driving the direction, I and that's the rewarding bit.
It's yeah, fantastic.
Um, you mentioned Cal Newport, Hinnek.
I've heard you talk about hustle culture and the you know, yeah.
Um yeah, I mean whatnot.
Not a fan, right?
Like uh neither him nor nor myself.
Like um I mean there's
treading that balance between doing your thing and promoting it and but and just being out there sort of slimily selling it all the time.
Yeah, I mean that that's another hustle thing, right?
Like hustle culture can mean different things.
Like uh the thing I usually talk about is that uh just do more, do it now, do it as fast as possible.
And then there's of course this constant hustling and chilling, and uh I just don't feel good about that, so I don't do that.
And uh yeah, it might be a that's just performance not great for my success.
That's just like performance, coding and art, right?
If you're spending all your time talking about how hard you're working, how hard are you actually working?
Is like right?
It's like when you go to when you go up to meetups.
When I lived in San Francisco for a while, I tried to go to meetups, and they're great at first, but then you realize it's like the same people saying, having the same conversations, and everyone who's actually doing something is like hunkered down doing the work rather than
Bro, I don't even sleep anymore.
You know, I'd I'll say one of the things I found a little disappointing about the Bay Area was realizing that even though everyone was, you know, a a lot of people were, you know, nerds in quotes, they were still super all the negative male stereotypes.
So it wasn't just like a you don't have to be a jock to fulfill some of those.
Anyways.
I think we have a name for that.
Um yeah.
Okay, I want to ask one more question.
Then I I know Carlton has a bunch, but so I w I have to bring up you have a 2011 blog post where you talk about quote Django's ORM is still a joke and reference that you preferred Flask and Pyramid.
So I have to ask, I know it's 13 years ago, do you still feel that way and maybe provide more context around that?
Okay, that quote is very much informed by a buck in Django that, by the way, is still open and can uh
Yeah, you did link to it.
It's can legally drink soon.
Uh it's not the one that is linked in a post, but um we'll put it in.
It's yeah, I have a fun story.
Yeah, 6614.
I have a fun story for you.
So um we as part of RD, I was trying out Django to use in our uh systems and it ended very, very poorly.
Um mostly like the key reason is that Django is uh a great framework for greenfield projects, but we do very few of those.
We have this existing architecture.
The um company exists for like 25, 26, six years now.
Uh we run a weird database, and um I had to fight Django a lot more than it was helping me to make it fit our infrastructure.
And what happened there is that we have a we have a database table that has a composite primary key.
Ah that's nearly coming in.
And I'm just make I make the long story short, I changed the password of everybody in our company because the table was uh the the primary key was username and domain.
And the ORM never told me that it's only gonna have the last primary key you define.
So when I when I set the the password hash for a certain user, I changed it for everybody in the company and all of our bots and whatever.
So that was a lot of work to fix again.
Yeah, yes, I can imagine.
I can imagine.
When not one once upon a time I forgot a wear clause on a delete query clause that uh went a delete query that went very wrong.
Um I didn't
forget anything, I just didn't know, and Django didn't tell me.
So you know, so that little negative story about Django from um time to a multi, by the way, but um Goo the Google Summer of Code project from this year is gonna get finally get the composite primary keys moving forward that's going in, and um it after I don't know 20 years almost, it's 2019.
You don't have the the buckets open for 90 years.
I check today.
Exactly, exactly.
So, but that ties into you know a lot of what we're gonna go on and talk about is that Django does things a particular way, and it's not necessarily the way you might do it from a kind of pure perspective.
So let's just put that out there and then go and talk about your open source work because I you've done so much, and I do want to get onto it and cover it with the with the listeners.
I think the thing you're probably most famous for is the Atters package.
Is that fair?
Yeah, I I that's definitely the one with the most downloads and with the biggest impact on Python itself, yes.
So I mean that so the Atters for people who don't know, you neighbor shooter but define a kind of very um very succinct um class, and it was the inspirate inspiration for data classes, which I think most people would have known of.
Yes, that's correct.
Like um I mean, adders predates type hints, so that's a thing.
Uh there's this funny syntax most people have forgotten about, but people back then really loved where the package name is just attr and the decorator is at edder.s, so adders, and uh you define attributes by assigning them to edder.ib like addrip.
Uh hilariously, not everybody got it, and people just imported those symbols, and there was like add as class and then x equal ip.
But yeah, uh so anyway, this was like the first good class building toolkit for Python, and it was uh really um popular too.
Um at some point, people started asking to get something like this into the standard library, and um I still remember I got the email from Guido when I was at Amsterdam Airport, which is a good coincidence, traveling to uh PyCon US, and we met up there with Eric Smith and yeah, did the first planning stage of data classes.
Okay, I mean it's uh it's difficult to get something into Python, right?
I mean, like something big like that.
It's because normally it's the talk is always about trimming it down and having less.
Yeah, it's it's usually not a good idea to put an existing package into the standard library.
Like we have this moniker of saying that uh packages go to the standard library to die, and it's just liberal.
That's a good one.
So we have this same debate in in Django, right?
So Python was is batteries included.
And I remember when I first picked up Python, I came from PHP world, I picked up Python, and it was just amazing because you know the standard library provides I don't know, most of the things I want wanted.
Um, and Django is very much a batteries-included web framework, but we have this same tension between keeping things maintainable, keeping them fresh, and being batteries included.
Because if you trim everything away, then where do you go?
You know, the example I always have is a um is a production web server.
I'd love it if there was a production grade web server in the standard library so I could just sort of import it and create, you know.
But I understand why it's not there, but uh in the same time I want it because I want I like that batteries included philosophy.
I kind of miss it in a way.
So what many people don't quite understand, and that includes also when we talk about packaging.
Like I made up a few videos on packaging, and I get like these comments like I wish this would be in Python, that would be done by Python, but there is no such thing as the Python.
There's just a bunch of core developers who are developing uh CPython, myself included, albeit not very active, and um they are not necessarily equipped to write a um production ready web server.
Right, web server, yeah, right.
They have no interest in solving uh packaging problems.
I mean, just because you're a Python core developer doesn't mean that you uh capable or interested in these kind of problems, right?
Like we can't we cannot just tell them to do that because nobody's paying them to do that.
So um this is just like um like a discussion where people just quite understand what they're asking for because it's it's not an abstract concept, right?
Uh it's not like Mr.
Python can uh reallocate developers to do something different, it's just a bunch of people with uh own interests.
Yeah, no, no, no, entirely.
I mean, I guess do you have any thoughts then on um, because this is the other solution, is let's just put everything on pip and then ask people to install it.
But the the the the the problem then is there's an awful lot of things on PyPI.
Um, and how do you how is a how's a newcomer to the community do you find the right packages, the right resource?
How am I ever gonna go and find at us short of being in the Python community and reading the blogs and hearing it come up?
How am I gonna discover it?
So many branches that we could take right now.
Uh obviously they should just listen to your podcast to learn about uh important things like that.
Yes.
Um so we've had this in basically this the same discussion at a course core uh not sprint but core what's the name?
Core summit.
At a core summit at Paco.
Language summit.
Language summit, yes.
Sorry.
Uh sorry again.
So we've had this topic at a language summit a few years ago when I was there too.
And there was the question, what what kind of things should we put into the standard library and why?
And my answer back then was already that we should put things there that uh make people write better Python that leads them to a better design.
For example, data classes, as much as I'm interested in using adders instead.
I understand that data classes are a game changer in making people write classes into just screwing around with dictionaries or God forbid name tuples.
And this is the things that uh should be in a standard library, things that lead people to good to writing good Python.
And uh something I don't want a web server in uh Python proper because I want there be competing ones.
I mean, there's this known new one called Grainion, which is written in Rust.
So there's no way that that could be in uh C Python.
And now we're talking about it.
Uh if we put a production ready web server into CPython, as the name says, it would be written in C, which is illegal as we all know now.
And um that's not great, right?
I don't want a um freshly written web server written in C in my standard library, so yeah.
It was yeah, I mean the the the the example came back to me this week.
I saw uh Julia Evans who does the web comics, the web scenes.
Um and uh she listed a um had a list of points about Go that she liked, and one of those points was that it's got a production-ready web server in it, and it came back to me.
I was like, oh yes, that was that was something I
but this is a perfect example because there is no Mr.
Python, but there is absolutely uh a Google.
And Google needs a production-ready web server, so they can just as well put it into the Go standard library and write it in whatever they want to write it, right?
So that's the big difference.
Right.
That's yeah, that's lovely.
Okay, brilliant.
So swing back to Atters.
Can you uh so I'm I use Atters all the time.
I really I really like it.
I think it's it's you know, as as you you say, it lets you define these little classes, and they they so then when you're using them, you're not indexing into dictionaries and you're not missing keys, you get type, you know, autocomplete, and you get all of the other things.
There's one thing that's you don't include, and you deliberately don't include it.
I wanted to talk to you about that, and that's validation, like data validation.
And you've got this line in the docs about how data validation should happen outside your could you just explain that tape?
Because you know, it's it's not there's there are other opinions, and I'd love to hear you talk on that.
Yeah, so this this harks back to ORMs, man.
Like we didn't we never finish the topic, so this is great.
No, we we we're getting to it.
We're getting to it.
No, no, no, this is perfect.
So um how do I start this?
So, yes, uh, adders does not do uh validation by itself.
I mean, it does have validators and these kind of things, but it is not primarily meant to do validation because I want my my um my data that I've worked with that I've write my business code with, I don't want that to do validation.
I don't want to have any information about how the code is coming from or whatever.
Like my core domain model is I want to have pure such that I can uh such it such that it is in a perfect shape for me to write that uh business code.
And there's been an amazing talk at the last um Pike Ascades called the rising sea, where the speaker shows how a problem is radically easier to solve if you have a proper data model around that.
If you take the first if you first take the step to transform the data that is coming in, and he's he's using I think some something from Code Advent, and transform it into something you can work with and then solve the problem, and that makes that the code that solves the problem radically, radically simpler and easier to understand and shorter and nicer than everything.
And this is where this comes in.
Like um, I don't mind the existence of Python or or whatever, like it's popular right now, but I think it is a problem when we use the same classes that we use for validation, also for our business logic.
And this is the same thing with ORMs, like because from these are two kinds of design pressure from two sides.
Like the ORM is basically your data model, it's the tables.
So your models in your code are gonna reflect that fact.
I mean, you can probably rename fields and do some uh joints and whatever, but in the end, the this is called uh table-driven design because the models that come out of your tables are defined by the table.
And in my experience, I mean, maybe I'm weird, but in my experience, you maybe have a say on how these tables look at the beginning, but over time you lose control over the tables one way or another.
Maybe another team is working on that, or someone else is really relying on it.
This all should not happen, this is all bad, but it happens all the time.
And like I mean, talk to anybody who's running production code if they can just change the uh data model.
They probably can't.
So that's from this is what's coming from database.
And uh validation is the same thing just from the other side.
Like you probably cannot just change your API, the JSON structures that are coming in once you've defined them, right?
So now you have this really you have these two types of design pressure that coming from two sides, and in the middle is your business code.
And your business code is supposed to be the pure one, the the important one.
It makes decisions if someone goes to jail or if someone gets uh married or whatever, right?
And uh this code, in my opinion, should not be affected by how the JSON is looking from the outside because someone decided five years ago it should be.
So this doesn't mean that, for example, Pydentic is bad.
Absolutely not.
I mean, people can use it and then transform into the data that uh they need for their business code.
But nobody does that.
Like I've been literally asked some from someone, like usually people ask me why should I use address if there's data classes.
So this time someone asks me why should I use data classes if there's pydentic?
And this to me is an attractive nuisance because you you do these two things, like you have this data coming uh from the outside and from the database, and it's extra work to shape them.
So you probably are not gonna transform it, and you end up with this weird code that is affected by how someone decided five years ago how to make the tables and how to make the API.
So this is the thought behind that, and uh this is why I prefer other ways to do validation.
And I also I feel like you need normal classes too at some point.
Like so it's just more layered.
Hi, this is Will again.
My website, learnjango.com, is now fully live and I'm having a Black Friday sale.
50% off all three of my books.
With the new site, you get lifetime access and free updates, meaning you'll always have l access to the latest version of everything.
I've spent almost a year working on it and managing all the user permissions, payments, and the rest.
Carlton and I will have a future episode taking a deep dive look into our respective projects this past year.
But if you want to support me in the show and get a killer discount, now is a good time to take advantage of the Black Friday sale, which is good through December 1st.
And the analogy with the Django projects is exactly right, is that you start off, you've got these nice models, they're really clean, they're tidy, they they they're expressive, they're quick, and then over time you've got oh, I've got an extra method here, I've got an extra field here, ah, and you it does get more complex.
And then what people end up doing is creating a kind of a mapping layer to move from the the table gateway, which is the the the the the Django model, into some more business logic type cleaner models or service layer or uh you know a a layer that's decoupled from that narrow RM.
I mean that's just the right way, right?
I I just don't I'm not sure if it's the the most common way.
Well, I think a lot of people uh there's that tension, right, between yeah, carrying on and struggling with or you know, and it's it's what's interesting is when do you make this this change or when at what point is it worth that extra sketching introduction?
That's what I said, right?
It's it's friction, like you have this these classes and they almost work.
So you either you can make this nice transformation that you just described, or you just make it somehow do right if statement here, if statement there, um you end up with terrible business code.
Uh but that's the that's the same with any growing software project, right?
It's not dependent on RMs or data mapping classes or you know, every
everything necessary.
Uh because uh you can control uh you uh the fringes of your code.
So so I think like my job is to deal with whatever is coming from the outside, and I consider the database also the the outside, and I can transform it into something I want to use.
And the same same goes the other way.
I I this is in my power, I can write some code.
And if I don't do that, that's on me, then I'm just being lazy and uh I deserve what I get.
But there is no tension if I do it from the beginning.
If from the beginning I say, okay, I'm using uh a validation library that uh just transforms into normal classes on the one side, and on the other side I use repositories or something like SQL C that uh spits out normal classes and then just control those.
Because then there's no friction to add it, right?
In the other case, you already have something that's almost working, and that adding a little bit of clutches and a little bit of uh dirty works here and there is the easier way than add a whole layer of transformations.
And it's also slower, it's like it's another indirection.
Yeah, no.
Yeah, it's that that that notion of an attractive nuisance, which I think is quite is the nice metaphor.
It's like it's it pulls you into doing the wrong thing in a way.
I want to ask you about UV and your YouTube channel, but before we switch, uh you have a number of open source packages.
Are there any in particular you want to talk about or call out?
I mean, there's too many for us to go through all of them, but are there any that you're particularly interested in or working on at the moment beyond adders?
Um if I have like my most underappreciated project, I think, is services.
Um for for many reasons.
For many reasons.
And I understand those reasons.
Uh okay, well, one reason
tell us.
Give us the pitch.
Give us the pitch.
That's the beginning.
It's it's so hard to pitch, like especially to Django people, because you're just used that everything is a global variable somewhere.
You're like animals.
That was not meant a Django.
I just have a friend that I have this constant discussion because he does the same thing in other frameworks too, because he's convinced that having singletons in a module is the better way.
And actually, he says if it works for if it's good enough for Django, it's good enough for me.
So fair enough.
Uh so the thing about services, it is kind of hard to explain in a general way if the person who's listening is not already cognizant of the problem it's trying to solve.
I mean, everybody heard of dependency injection at some point in their life, and they probably uh think it's something with Java and XML.
And one of my actually my second YouTube video was about how dependency injection is actually just passing arguments into callables and um by that way making things testable.
And um how you get those things that you pass into those callables, there's different ways to do that.
There's dependency injection frameworks that I personally find complicated, and then there's something that's called service location, which is what services is doing.
And the big picture is that you have a registry where you define how to create third certain objects as in types, and then you have per let's just use Django as an example.
Then you would per Django request you would have a container which you can use to create those resources, which can be database connections, it can be API clients, it can be anything.
And of course, the boon is that at the end, first the container cleans it up behind you, so you don't have to make sure that your API clients are being closed, or whatever resource I use you're using.
And uh you can easily replace those objects for testing.
It's you don't have to monkey patch to replace your database, which with pi with Django, it's a bit more complicated, but there is no an official Django services plugin where also James Bennett uh is part of, so I'm sure it's pretty good.
Okay.
Yeah, I mean, so the two things you mentioned there, one was your video uh where you introduced this.
Um with uh both you talk mainly about dependency injection, but then you mentioned services at the end, and you had a lovely line in there, which you just when you sweat said it, I was like, oh yes, it's that um all all all side effects, anything that can happen, like you write to a file, make a network request, make a database thing, all side effects happen through an object that's passed into the function.
And you say this is the key bit, right?
Is that instead of having the database connection card-coded in there and you do it, it's you you take it take that as the reference.
And then if you want to pass in a mock or a test, a test fake or whatever, you can do that very easily and test that logic separate to did it actually make the request or did it actually, you know, without having to do at mock, you know, at mock patch.
Yeah, yeah, yeah.
Um I just wanted to to quote Brendan Rhodes again where uh monkey patching is software bankruptcy, and he's right.
I want to say just one more thing.
You said it was underappreciated.
Yeah, uh, and it is like when I talk to people about services and describe what it does.
There's two reactions.
One reaction is complete blank stare, no idea what I'm talking about.
And the other is like this look of pain and recognition.
And and uh like my ex my at this point, my estimation is that the average company has like 3.5 implementations of something like services in their company.
Just half asked, because that's literally what they say.
Yeah, we've did the same thing, but it's half asked.
So you don't say.
And uh the problem with spreading is that this is very much a project that you or a package that you use when you start a project, the decision that you do when you start something new.
It is non-trivial to apply it on an existing project.
Like even I don't I don't use services in all my work projects.
Like I I just keep adding it when I get to work on my project or two or two or so every now and then.
But it is hard to sell because people only you need it maybe once per year or something, and uh yeah, the growth is very slow.
And again, you you have to have this intellectual curiosity to just deal with these kind of things, which most people don't have because they don't care, or I don't know.
Okay, James James, yes, James Bennett, who you mentioned there, uh, he wrote an essay um a long time ago against uh about you know the Django patterns, and he talks about he was arguing against using a service layer.
It's about no lean into the active record record notion.
Um and one of his arguments that he gave there was that you know people talk about needing a service layer in case you need to change the DB.
That's just never gonna happen in real life.
And it's you know, I'd be interested to get him on again and talk about you know how he feels and his development of um the Django services like um component there.
I mean, I'm not gonna tell uh Django people what to do, and I think Jams is on the money when he says that if you are working within a framework like Django, you should you should do as the Django notes, right?
Like it um it doesn't make a lot of sense to use a fully featured fully featured framework like Django and then start using it like Flask.
That's not gonna go great.
I mean I had to do that basically, that was what I had to do due to my constraints, and it was absolutely no fun.
I have replaced at this point zero databases in my applications over the past 20 years, so I don't think that is a good good argument at all.
But like a service layer can you use an RM2, I think.
Like it really depends how you exactly define it.
Uh for me, a service layer is more defined by the fact that it doesn't do any view things in that sense.
And what I'm personally leaning into nowadays is that uh I'm trying to do something kind of called like a uh functional DDD, like it is a lot of inspiration from functional programming where you do a lot of things with immutable data, and it's basically data in, data out, and then you apply the changes, which is even more even more um testable than everything else because it's just data, right?
I would not know how to use something like that with a heavy framework like Django.
So yeah, it's difficult to bring in these ideas.
You bring them in sort of slowly and you know think about them you know piece by piece rather than bolt, you know, you can't there is no um one step.
Oh, yeah, let's just switch to doing it in a totally different manner.
That doesn't exist.
But to me, like a service layer is mostly about controlling transactions and uh not doing anything with views, which already makes it more testable because you don't have to do view stuff to test things, right?
And it's like the separation of layers, it's um yeah, and I understand that repositories, for example, are also not very popular in uh Django for good reasons, but I could just drop words like CQRS or something like that, but that's and we would be here all day.
Yeah, okay.
So look, one more thing I want to talk to you about in this kind of software and gerine world, I guess.
It's just you've you've got this fantastic talk and essay combination on subclassing and subclassing in Python.
Um and perhaps you could give us your talk, and then perhaps we could talk, um, give us your overview of that, and then perhaps we could talk about you know a couple of examples from the Django world because Django uses subclassing very heavily, right?
You subclass models to create a model, you subclass the admins to make an atom, you subclass the forms to make forms.
So, you know, you probably have views on that.
It'd be interesting to just
Django is a child of its time, right?
So what what what are you gonna do?
Uh I've been very involved with the Twisted project, which is from a similar time.
I think it's slightly older, but it had the same problems.
Like everything is template subclassing, everything is subclassed.
And um, but at some point they realized that, or we.
I just have this memory of us sitting in Bristol at a sprint and eating Thai food and watching this uh seminal talk by Augie Feichler and Nathaniel Manista, and the name was like the end of object inheritance and the beginning of a new modularity.
And that was kind of the talk that in the Python community kicked off the the change of thought that maybe we don't have to subclass everything.
And um I'm not sure where I'm going with that.
Uh I'm just what I wanted to say is that Django just started at a certain time, so it has certain patterns ingrained.
And nobody's gonna rewrite the whole thing, right?
And it'd probably also be weird to just change the paradigm suddenly.
I mean, probably would be the right thing, but uh this is hard, I think.
This is really, really hard.
Okay, so let me let me phrase it.
Let me give you something like because I quite like the subclassing.
I think you know you subclass the model, and it's fine.
And I read your rest and I agree with all, you know, I kind of go through and I agree all the points.
I think, but you know, also, you know, if I just take the model and subclass to if I just take um Django's model-based class and then subclass it for my model, and I'm not doing crazy inheritance structures, I don't it it seems kind of okay.
You know, it's it's like I get all this behavior, I get all the things defined.
It's not like I'm losing myself in a million different call chains.
And it the what's so bad?
I mean, it you know, that's a rhetorical question.
Yes.
Uh I mean, obviously, again, it's all trade-offs, right?
Like in a case of models, the subclassing is mostly a mechanical thing to just decide to do it.
It could just as well be a class decorator.
And the difference between a class decorator and uh subclassed approach like this, which I demonstrated in the talk, is you just get everything inherited uh into your model.
So your models are very heavy, and it depends from when that's a problem and when it's not a problem, but you are on a slippery slope either way, and um but I don't find that kind of subclassing as bad.
And again, again, we are in Python, so some things only work when you do subclassing.
So uh but the point I was that I was making in the talk is that uh we have now two languages, Rust and Go, that have proven that you don't need subclassing as a concept to write great software.
You might need subclassing in Python to achieve third certain effects, like you have to subclass uh um exception or something like that, like for ontological reasons.
And there's different types of subclassing, which I also talk about.
Um, for example, specialization is fine.
Like Go has specialization, they are just cheating, they call it embedding.
It's specialization, it's nothing else.
It's um it just works in a very narrow way, and I use subclassing in the same narrow way in my code, but it is something you have to know, like it is an active decision, it's it's active design decision how you deal with those sharp edges.
While when you use composition, uh you have to take effort to break it, you know what I mean?
Like um subclass is basically broken by design, you get everything in, all the the borders are destroyed.
Um you you get a lot of things happening by accident and a lot of surprising behaviors, and if you go crazy, like you like you say, like if you have like this whole hierarchy, uh diamond-shaped, what I what do I know, right?
Then it gets extra hard.
So there's many points that you have to take into account.
And but of course, like just subclassing to effect to achieve a certain effect is not bad.
Nobody's going to jail for that.
But you have to be uh cognizant of the rules and you have to be cognizant on um how to deal with the complexity, and you don't have to do that with with um with with uh Jesus Christ.
With composition.
Yes, oh god, sorry.
That's okay.
No, so uh so it's probably I guess that the sort of take-home is it's it's like a code hygiene thing again, like it's another one of these method methods of making sure you don't fall into you know unmaintained or making your code harder to maintain in the future as you as you move forward.
And this is the thing when someone asks me for advice, I'm gonna tell them if you don't know, don't use subclassing.
Because it means that you cannot make this decision in an informed way.
So yeah,
and you give a good framework.
Yeah.
What once you have the knowledge, no, once you know what uh Madame Misk Liskov is uh telling you to do and these kind of things, sure, wield a tool, but but it's a much sharper tool than composition.
Okay.
There's one it's just one bit on the essay.
I agree with the essay as I read through it.
There's just one bit of the essay that I do disagree with.
It's right at the end, you you talk about uh, or you give this example of template method as being the pattern, the sort of the pattern you really hate.
But it's something that um because writing Django web views all the time, they're all nine, they're all the same, you know, they're all CRUD views.
90% of them are crowd views, and 90% of the operations of the CRUD views are identical.
And Django's class-based views are very much template methods, right?
You override the one little bit you need out of the template in order to make it make your your the particular view that you're working on in that time.
Why do we hate template methods so much?
Yeah, I hate them so much.
Because they're create this cyclical uh dependency and and this indirection through two classes and you're jumping up and down the hierarchy, so which is like the worst part about making things unreadable.
Uh I've just recently written an internal project where I also use template subclassing because the development experience of using the resulting object is much better.
So again, it's all a trade-off, right?
But I hope I wouldn't build a whole web framework based on template subclassing.
That I would
put it's the view, it's the views, right?
You've got an algorithm which has got some known steps, and those known steps sometimes you need to plug out a little well, that's template method, you know.
Yeah, but it's it probably it probably could have been solved with a strategy pattern or something like that, you know.
Like it just depends where you're putting in uh the code.
But it's hard to talk about this from thin air.
No, okay.
So we've got here in the Google that you get 300 million monthly downloads across all of your your software process.
Yeah, but it's more a function of Python getting incredibly comp uh incredibly popular suddenly.
Like, I mean, to get into the top 20 of PyPI donals, you need like more than 300 million donals.
Like this is it's insane.
Yeah, that is insane.
Okay, go on well.
Okay, so two things.
I we want to talk about uh UV and and I I want to ask about your YouTube channel, so maybe I'll start with that.
So your videos are fantastic.
I mean, I've look at them and I see you're doing a lot of production and you've got quite a few of them.
How how did you actually get started?
I guess would be the question.
Because lots of people think, oh yeah, a YouTube channel, but it's quite a lot of work and quite a lot of work to do it well, which I would say you're doing.
So what was that journey on you know starting your channel and then doing real production?
Because again, you're not just one straight take, you've got graphics and cuts and you're doing a really nice job with it.
Yeah, well, thanks for saying that.
Uh I mean I I disagree.
I think my videos are still terrible, but uh it's glad that my uh viewers are.
The ones I would do are would be way worse, right?
I mean, you know, but I I think until you've tried to make a video, just seeing even what you've done, I see hours of time in simple things that someone wouldn't if they hadn't done it themselves.
Yeah, you see absolutely correctly.
It is so much I mean, starting the channel was the usual programmer's hubris, uh just thinking that's how hard can it be.
Uh um I'm I'm gonna have to tell you the reasons too.
I started like reason number one, and that did not work at all, is that I hope to just get better at speaking English because I just get to talk more to the camera.
But uh that doesn't help if you're recording like one video per month because because I'm so slow at at the production, because as you said, production is really hard.
Uh sorry, I have to stop you.
I mean, your English is almost flawless.
I I don't really know how much more you would improve to uh thank you for saying that.
I mean it's true.
But the main reason was was really uh the implosion of conference invitations.
Like uh giving conference talks was my main creative outlet in my life because I cannot draw, I cannot clap on a beat or something like that, but I can give conference talks, and I've been told pretty good ones, and I derived meaning from that too.
Like I told you before, small company.
So if I want to have an impact to more than 20 people, I have to do something public.
And um the interaction with the community and the invitations to conferences around the world.
I mean, I've been to Japan, I've been to um to the US, I've been to Russia when it still wasn't that bad.
Um, it just stopped in 2019.
And um whenever I talk to conference organizers, everybody's just complaining how uh they are just out of money.
So basically, this whole bubble just burst for me.
And my other creative outlet, which is writing, has also kind of tanked because whatever Twitter and Google are doing there right now, like everybody who has the blog will tell you, unless you are not on Hacker News news uh front page, nobody reads the stuff anymore.
So I decided to go where the people are, and YouTube is apparently the second biggest search engine in the world nowadays, probably after TikTok, I'm guessing.
Yeah.
There is some logical synergies with conference speaking, right?
Because I'm doing like this um talking head video stuff, which is basically giving a talk.
Um, I had a photography side business many years ago until I realized that having a photography side business means looking into Photoshop after work, which I did not enjoy at all after a while.
Uh, but I kind of understand how lighting works.
So, and it still took me way too long to figure out how to record in my uh tiny room here.
Uh but yeah, I'm I'm getting somewhere now.
Right.
Well, it's all about the you know the key lamp and then the overhead and then the side one.
You've got the shadows working.
Like, I mean, I I can tell because I've like yeah, it's not so easy to do.
Yeah, basically I have uh one one big light on the right, top right, uh a small light to the left, which is basically just uh like a lamp.
I mean it's that is for webcams, but I just uh rotate it in into me talking to it.
Um I have a silver fill disk under my chin because I have deep set eyes.
And uh yeah, I record using my iPhone to an SSD.
Oh, do you use um cameo or what's it?
What's it called?
Is it cameo or they're the software for using your iPhone?
It's with a C.
No, no, no.
I I record directly on the phone.
Uh like the new iPhones, they have they have like this raw and log recording, so very high quality.
Oh but it's very big files, so I'm recording directly to SSDs on that.
So I connect the SSD to it.
There's like this crazy contraption that I connect to my phone, which is also like leading out an HDMI to my iPad so I see what I'm doing because it's right next to a wall because this room is tiny.
Uh, but I record the audio directly to my to my computer.
And this shouldn't have taken more than one afternoon, but it took me weeks of experimentation, and Lukash Langer was helping me a lot, especially with with the audio and everything.
And um, yeah, it's it's it's a process, yeah.
Um but learning about it.
I'm sorry to interrupt, I was gonna ask what um video editing software are you using?
Uh so I started using uh DaVinci Resolve, which is free, but I did not quite warm up to it.
So I switched to Final Cut Pro, which is like this Apple software.
And um, I'm really good with Keynote, which by the way I'm using for the slidey stuff in my in my um videos.
So it I felt more at home because DaVinci Resolve really feels like a Linux application, which it is, and um like in the best and worst possible way, it's like super powerful, but I I just couldn't get really happy with that.
Like, so I switched to Final Cut Pro after like I think up with my third video or something like that.
So, yeah, this is also very expensive, by the way.
So um, like um Final Cut Pro 300 bucks, then you spent even more money on the on the plugins and everything, right?
So, yeah, I couldn't be doing this without my GitHub sponsors.
So, and since I'm talking to one right now, so thanks for your support.
Well, I I think you you know you're right about that.
Thank you for turning out all the great stuff.
Yeah, the the conference talks.
I mean, uh uh our friend uh Jeff Triplett has just launched Django TV to try to put more light on conference talks.
But conference talks, even though they're available on YouTube, you look at the viewing numbers, it's uh you know, a couple hundred, maybe a couple thousand max.
Whereas, I mean, your your video I
think my subclassing talk has five thousand.
Uh like the the I think this is my best best talk I've ever given, like in every way, the subclassing one, and it has five thousand views, and so it's just like nothing, right?
Yeah, I know.
Like the UV one had 20,000.
Right, yeah.
Well, okay, well, I know we're we're coming up on time.
I have to we have to ask about UV.
I separately I would ask you a million questions about about YouTube, but um uh so UV, right?
So you have I think two two main videos, right?
Like one asking, is it the answer?
and then a second one saying it is the answer.
But what I what I was really impressed by is you really go in depth.
I think they're both about 20 minutes.
And uh, you know, so give us the pitch, right?
But I I I was really impressed.
You could easily have done just a quick clickbait, you know, short thing, but you really dove into why it's so great, but also how it could be improved um going forward.
So do you stand by, I guess as a question, you stand by the most recent video, you're still on the UV train, and you think this is the packaging uh answer we've been looking for in Python?
Yeah, definitely.
It it is not like nothing before.
So yeah, I don't so for the foreseeable future, I think it is the future of Python packaging.
Um so you want to give me pitch for my videos or for UV?
Oh I wasn't quite sure what you were asking before.
Yeah, I guess I would I I do this sometimes.
I just Carlton you know knows I just talk.
I guess I was I was just saying people should watch the videos, but I was giving you a compliment that they're really in-depth, you know.
So people should go watch them.
It you can't sum them up neat neatly, but I was yeah, far more in depth than you the usual videos, right?
If I see something on a topic I'm interested on, usually it's someone just saying, This is interesting.
Um, but you yeah, I guess we don't have time to dive into all the nuance that you cover, but you you you do a good job with it.
Yeah, it's mostly a result of lived experience.
You know, I'm not talking about things that I've read up somewhere, but I've been part of all this shenanigans around Python packaging, and I've been around for a long time.
So I I remember when setup tools finally got freed and uh was taken over by the community, and when it was a big deal, I remember when wheels came to life and it stopped being a big deal to install a binary extension.
Yeah, so um so I've of course that that's a gift of age, I guess.
So I I have the perspective and the context to talk about these things in this way, and just uh knowing so many of those problems that they are actually solving.
And I have the humility to say that those they are not really solving my problems.
Like I was reasonably happy before, like PDM worked great for me.
UV is much faster, but I can see how uh how important UV is for the Python community because again, at a community-wide scope, I don't care about the speed at all.
So, which also like there's been this NPM blog post about why uh npm should not be rewritten in faster languages, and I think this is a red herring, like if UV would be just as slow as PDM or PIP, it still would be a huge deal because people complained about Python packaging being um unclear and janky, right?
People things were broken and it was unclear what tool to use when.
There was a bunch of 90% solutions, like and you have to know which 90 where your 100% are in that spectrum.
So to use the right tool.
These are the things that people are complaining about, not that pip is too slow.
That was never an issue, at least not on a serious thing, right?
And um I don't think this is possible to achieve in Python, like we've tried, and hatch came close, but only by using Rust helpers again, like uh Alfek wrote this Py app thing.
So some Rust or Zig or C is inevitable.
Um, but don't use C, C is illegal now, at least in the US.
We all know that.
But now like making like modularizing it would make things more complicated again, right?
And you cannot ship like a packaging tool that hasn't binary extension.
That would just make it that would that would make things worse, not better.
So um So
you think it's the one tool thing that's the real thing.
It is the one tool that doesn't break.
That is my main point.
And I think that also resonated most with the audience of my video when I make making this uh this point.
And I of course I don't want UV to be the only tool, right?
I want PDM to exist and um and do well.
I I'm happy to keep Pip the default installer and um so people can use it if they want to.
I don't and I don't want to convince anyone to use UV.
It's fine, just use whatever you want, right?
But I'm really glad that we've just solved the biggest Python problem we've had, rivaled only by the Gill.
But I think this is a bigger deal than the Gill because people run only months later to using Python into threading problems.
But the initial impression of installing Python, of using Python, of installing an application, of starting an application, that is something that you have on day one.
Um that's a huge that's a big huge difference.
And people are like looking for hair in the soup, uh, which as a check living in Germany, uh two very complaint heavy countries.
I fully understand.
And we also totally need to do risk management, like we need to plan for when Astral's VC overlords will pull the rug under us.
But I wish the conversation were more solution-oriented and not not uh and less like per clutchy, like yeah, there are there are bad aspects to this.
Like this is absolutely true, but also this is something wonderful that we've been waiting for.
We've been begging people to use money to solve this problem, and now someone did.
So now it's our job basically to make the best out of that.
I just have one question on on it, which I saw a post this week um from Nolan Lawson, but he was talking about the he was saying why he's skeptical of rewriting JavaScript tools in parser languages.
Yeah, that's the one I meant.
Yeah, so but the point I took from that I thought was quite good, was that um the thing with when a tool's written in Python, or in his case JavaScript, users can sort of debug it themselves.
When they hit a problem, they can get in, they can understand the the the what's going on.
But if it's written in a another language, a compiled language, suddenly that ability to dig into it is is gone.
And I wonder if well, I wonder if that's a loss.
So I'm I'm gonna say the word again.
Uh it's a trade-off.
Um it absolutely is a loss.
Um, there was this um real interesting mastodon thread started by Jacobian, like you're very Jacobian, uh, where he kind of shitposted, kinda lamented the fact that uh you used to have to learn C to contribute to Python, then you stopped having to learn to C to contribute to Python, and now you need to learn Rust.
And that absolutely is a loss.
But the question is, what are we getting for that?
And I have to say, I've I have not debugged PDM or PIP many times, so for me it this has not been a loss in a practical sense.
And if it helps all those people that have been so frustrated by Python in the past, and by the way, many of those are in the comments of my video.
It's not even funny.
Like how many people are just airing their grievances with Python packaging?
So there's so much pain that still needs to get out.
So those people are there, and this is helping them.
Like they are the ones where we say this is the one tool, use it, and at some point you can still look at other tools, but for now, with this one, you don't even know you don't even need to go to python.org, you just download this one thing.
Well, I think we're we're out of time.
We could still talk about so many things, but I feel like this is a that
wasn't even half of my notes, man.
I can't.
I know.
Well we'll have to have you on again if you'll if you'll come.
Well, yeah, and thanks so much for indul indulging all uh you know silly little questions about this and the other.
But it's good to be able to pick your brain and get your responses.
Yeah, thanks for listening for my to my uh rambling.
I'm used to script everything out because of course uh English is my third language, so it's very hard for me to just have coherent thoughts uh live.
Yeah.
So which is well, we have links.
It's so nice to lie to me all the time.
No, no, it's
I'm not sure I could have so I speak Spanish in Catalan, so live in Spain.
I'm not sure I could have an in-depth technical conversation about ad hoc about software engineering in Spanish or in Catalan.
I'd struggle.
So I think you do for normally.
Thank you.
And the funny thing is I have much harder time to speak about technical stuff in German or Czech because it's just not
because it's not the language of yeah.
Yeah, yeah.
So yeah.
Okay.
Well, we have links to all these things.
Definitely check out um YouTube channel, personal site.
And again, thank you so much for coming on.
We are at DjangoChat.com, and we'll see everyone next time.
Bye bye.
Bye.
Bye bye.
This show was brought to you by learnjango.com.
Free tutorials and premium courses to help you level up and stay informed of all the latest best practices building websites with Python and Django.