Transcript: Django 3.1 Preview - Mariusz Felisiak

I'm Carlton Gibson, joined as ever by Will Vincent.

Hello, Will.

Hi, Carlton.

And this week we've got back with us my fellow fellow, Marius Feliciak, who's here.

Hello, Marius.

What are we going to talk about?

We were going to talk about something.

So Marius, go on, you're the guest.

You tell us what's in the uh what are the highlights?

What are we looking for?

Uh and released candidates two weeks before it.

So it's currently the best time to uh to try it, to install it on your existing project and see if everything and see if everything works properly, especially with new features.

So everyone that used uh, for example, Jason Field from uh Django uh contrib package can now try to use this directly from Django models.

Uh that's probably the easiest way to uh contribute to Django to try the pre-release packages.

Um and I just we'll link to there's a very comprehensive uh release notes that go through all these things, but we're gonna spend this episode talking about them a little bit.

And and the JSON field was done by um Sage, who uh Google Summer of Code student from last summer.

I want to talk about the features, but I also I if the two of you could could take off your humble hats and just talk about what is it, what do you two have to do to prepare a new release?

Because I know there's a whole lot of steps, but just to let people know what you know what are all the things you have to do to get something ready to be pushed out.

So uh we we we just need to remember to do not omit any step.

Exactly.

Something's broken.

The big issue I think is the point releases, like so like 3.0 or 3.1 or you know, 3.2, like the major releases.

They're the they're the difficult ones because they're only once every nine months, and the steps are there's a lot more steps.

It's like what did we do last time?

Oh yeah, like trying to remember all those pieces can be you know can be tricky, and it's like, oh and why aren't the docs building with the you know the Chinese translation?

And uh exactly there is a lot of small steps that that should be done, like for example, open pull requests in read the docs, or uh add a new classifier to uh PyPy or add pre pre-release to Django project to uh really run Docs build, etc.

Which what

uh the second the the the second yeah was the second and uh honestly the first uh uh major release uh that we uh are making totally without Tim, I think.

a good spot for us.

Um okay, so let's talk about the highlights of the release.

So one of them is that uh it requires Python 3.6 and up, right?

It's not supporting lower versions of of Python.

Um Yes.

Oh it is

Um supported.

Um I mean it's in uh what do they call it extended support, so it's only security fixes now for Python 3.5.

But in order to move Django forward, like the the three point, so we've got the last two, so for three point oh no for 3.2 for the LTS we drop the version, and then that will happen again probably for 4.2, we'll probably go 3.7.

We are dropping Python version only after the uh LTS version.

Yeah, so uh Django 2.2 will end uh its extended support after uh dropping support for um Python 3.5.

Yeah, so they will not support Python 3.5 uh before uh Django 2.2 will uh end its extended uh support.

Python 3.6 will be supported a bit longer, so uh Django 3.2, for example, it's not uh enough uh to drop support for Python 3.6.

Like you know, and ah it it that gets harder and harder as you get, you know, when 1.11 was at the end and we were still supporting 2.7 on 1.11, and like like a number of fixes every month were causing issues only on 2.7, only on 1.11.

It's like ah, we forgot that.

We've got to go back and change it because of this.

And you know, it just gets harder and harder the lot for that long-term support as it gets older.

Um because it's really not that much fun, but it requires a lot of focus and expertise to do, so hence the fellows.

Um so 3.1, so what what's the headline?

I mean, I'll tell you what I think, and you guys can tell me what it actually is.

To me, when uh people ask me, it's you know, async uh views are in there, there's JSON fields.

I'm excited about the new path lib in the settings file.

So when you run start project now, um it will default to pathlib, which just makes setting your path libs a lot easier.

But there's a host of other features.

Um and maybe we could just start with async.

So what's what should people take away from 3.1 and async?

Is it you know the question is is it finally ready to use a Django project?

Or what are we waiting on?

He's an he's a he's an expert.

Oh yeah.

Um so the first thing is don't change anything.

Just keep running your Whiskey server, and then all of a sudden you want to um make some API requests to pull up from data from a couple of places.

And so you can now just make define without changing anything else, you can define an async def view.

Like so instead of a def view, which is a sync function, you can find an async def view, which is an async function.

And then you can make use HTTPX or you know, whatever your favorite networking library is to go and make the API calls you want to do concurrently, and Django will run that for you in a in a thread pool in a thread, it'll run that's that single view in a thread, and you'll get concurrency in fetching the API views, but you haven't had to change anything else.

So, for a lot of people and a lot of use cases, that's that's kind of what you want to do.

For years, for years now.

I've been spinning up Node.js projects on the side in order to make concurrent requests to the AP to an API in order to um pull together data for a client.

And then the last few years we've been able to do that with Python web frameworks instead of spinning up no, but now you can just define that view in your Django project, and that's super, and that's without doing anything else async related.

So someone who's updating, if they just update from like 2.2 to 3.1, do they need to uh manually add the ASGI.py file or right.

So you don't have to change anything to do this first level.

And that's the that's the that's why it's so beautiful because you you can just literally upgrade, and then you just think, oh, I'd like to try fetching two HTTP requests simultaneously, and then when they're both back, I'll combine them and return them to the client.

And you can do that without anything else.

It's like, ah, this is your really is dipping your toe in async.

You don't have to change your way of thinking very much, you don't have, you know, you just have to write this one async dev view.

So that's that's what's super, and then the step up, the second step, is if you wanted to get involved with um response types that are more interesting.

I don't know, let's say you want to um do uh server-side events where you open a uh a connection to the client and every so often you pump some more data down to it.

So I don't know, like a ticker tape application.

This would be the textbook example here.

You open a connection and the server can every so often feed you some more information.

These kind of more interesting things.

At that point, you'd need to switch to ASGI and you'd need to create an ASGI.py file, and then you'd either use Daphne or UvaCorn or HyperCorrent or one of the ASGI servers, so this is instead of Garnacorn as your application server to serve your ASGI pipeline.

And what's amazing, so and the other side of the amazingness is that if you switch to ASGI, all your sync views will continue to work exactly as they had because underneath the hood, Django is switch is determining is this view an async view or a sync view?

Now there's a certain performance overhead to that, in that there's a certain context switching.

Every time you context switch, there's a really small performance penalty.

So if you're really, really, really after throughput, then you need to make sure that you're running ASGI with only ASGI middleware, with only with only ASCII view at the end.

And if you if you've got ASGI all the way through, then you don't get any context switching performance benefits.

But for most use cases, that extra teeny bit of throughputs, that that's not what you need for most use cases.

Certainly not when you're beginning.

Does that sound about right, Maris?

Like that's kind of my kind of my aspect.

Uh and of course every everything works properly, but uh uh but still I think in it the the journey has begun in Python 3.6 and in 3.7 it is uh it it really works well.

is still evolving, and you know there's still debates about the API and how it looks and what the recommended um call paths are and all this kind of stuff.

And so if you're gonna be using async, then do yourself a favor and get on to 3.8 or the latest versions rather than trying to run it in 3.6 because uh you caught you're gonna create yourself lots of issues there.

I see uh see issues on channels or on Daphne where it's like, oh no, if I update it to 3.8, this issue went away.

You have to tune it up and

You have to send you car in for a service.

Welcome to Django Channel Fox.

Uh if perhaps one of the ways people start so one of the things people think with async is oh now I can maybe replace Redis or some of these other services that are Q-based things.

I mean, I mean that's a imperfect statement, but I think that's one that someone could think that.

Could you disabuse, you know, either of you of of that comment?

Like, oh yeah, async Django, so now I don't need you know Redis or Memcached.

I say this because I've had that thought and because I'm asked that by my readers.

So it is out there.

You're caching server, you're not gonna get rid of that.

And if you're using Redis in that same um ballpark, you're not gonna get rid of that.

Yeah, so

So in principle, we will be able to create hooks for lightweight background tasks that we'll just be able to spin off using an async.

I go and do this async task in the background.

Now, Starlet, Tom Christie's framework has got that kind of um ability already in it, and we're you know, we keep looking at that and thinking, oh, maybe you know, 3.2, 3.4.0, who knows?

Um, but it's not gonna replace a queue.

You know, it's not lightweight background tasks, yeah.

Replacing your full, your your year, your queue with years of um battle-tested coding behind it.

No, I don't think you're gonna replace that.

Keep keep Django Q, keep Celery if you're using Apple, Huey that came across the code.

But don't like don't you're not you're not gonna replace those instantly.

Like maybe over time, but this stuff is still fresh and young.

And and what's sort of oh yeah, we could do this doesn't mean it's you know the same as right wrapping a an existing task or function with a decorator and it just happening by itself because all of that's written.

I just wanted to tee that up so you could disabuse that statement.

And I mean the cool thing about Starlet too, just with async, is you know, there's this fast API framework that it uses Starlet under the hood.

And um I'm sure you know we in the Django community will take some inspiration from how that's going, but that seems to be I don't know if there's examples of it at scale, but it's very people are enjoying using it.

I've heard a number of developers who've built smaller things with it and really, really liked it.

So, you know, it's very popular.

And what's really great with ASGI is so cool, like because every every level of ASGI is a middleware.

It's it's a every app, every bit of every middleware and every view is an ASGI app, right?

It's uh it's so you can nest, yeah, you well, it's kind of like an onion, but it's kind of like a a tree as well, in that you can nest them inside each other, and you can you can have a middleware that would root between two ASGI apps.

So let's say you're using Django, and then alongside of it you want WebSockets where we've got channels, which is you know where you need still go for WebSockets even now.

And then, but let's say you've got something that's written in Fast API, you could put that next to it, and it would from the same server, and you could root it within your kind of ASGI framework, and and combine the best of both worlds.

And if there's a nice middle ASGI middleware out there that you want to use, well, you can wrap your Django app in that ASGI middleware, and you know it's a lovely position for us to be in.

It's kind of really um what's the word I want uh like um fertile?

It's a really fertile environment for us to be building web apps in now.

I think I'm quite excited about it.

So, testing, how does async testing work?

Or how will it work?

Because there's some there's some of that now, and then going forward, how should people think about that?

So there is currently there is a async client that is built in uh testing framework uh that can or or should be used for async stuff.

To be honest, for me it was um a big challenge to uh review uh uh the entire uh change creative facing views, middleware, and testing.

But I I I think that's that's something that uh probably uh allow us to to find at least few smaller regressionists or a person that's uh that is not an expert in async world uh try to try to uh work with it.

I think that's now I'm uh a bit better in that uh uh but but uh but still so for uh uh for async tests uh you can now use uh async client.

Uh you still can use the same methods for uh testing uh get post or anything else, it is just uh uh called with await, yeah.

So it need to be uh awaited and uh uh in in instead of that everything uh works similar.

So you can you can test the asynchronous functions using Django test.test case.

Yeah, that was pretty cool.

Which is nice.

And Marius, if I could have have you answer this, um can you describe to people what it's like mentoring a Google Summer of Code student?

Because we've had them over in the past.

We have several now.

They add major new features, but I don't know if people I know that people don't know what the mentoring role is like.

So could you talk about maybe specifically with Sage?

So we've had him on and you know he's a student, he added um cross DB JSON fields, so not just in Postgres.

What is that process like, you know, for you as a f a fellow to mentor someone?

It was

Uh maybe because uh Sage was really uh a hard worker.

So uh he made a lot of uh a lot of uh job on his own.

Yeah.

So he made a lot of uh investigation about how it is supported in different uh dbs because it's not really uh a straightforward uh thing.

Uh because we have different implementation in different DBs.

We have PostgreSQL where there is a native field, yeah.

But uh in different in other databases we have, for example, text field that is just looks like JSON field.

It's for example text field with some extra constraints that are built in a database.

So it's it's it's uh extremely complicated.

Uh but yeah, it was really uh nice experience to be uh to be honest.

The hardest part was to to to uh to to polish it at the end because it contains a lot of small pieces uh in different parts of Django.

So we have changes in schema, we have changes in introspection, in uh different lookups implementation, decoders, encoders.

We need to deprecate old fields, uh we added some extra system checks, etc.

So a lot of different pieces uh that uh around the different layers in in Django.

Uh I can say that it was the biggest pull request I think that I ever merged uh to Django.

It's touched it uh touches around 3,000 lines.

Yeah, it's uh it uh has uh more than 400 comments.

So before before uh clicking merge, you need to check 400 comments and see if uh all of them uh are addressed properly.

I mean it it speaks to sometimes people you know want to know what database should I use with Django.

And it is true that Postgres has more features, just like search, um, and just because people have built them, um, so it's it's nice to have more uniformity, but it takes a tremendous amount of work, obviously.

And you know, at the end, there's there's this the the way that key parts are determined.

If you try and do a lookup on a nested JSON field, you know, like so you've got um my data with pets, with you know, dog rex.

I want the the dog who's got the name, you know, and you try and fetch that value.

That's really complicated.

Um, it's just mind-bogglingly complicated.

And fortunately, Marius finds it hard as well,

so it's too upset.

I think that was super.

Is it gonna be easy for me or is it gonna be difficult?

Uh you just shut uh you you just shut uh should change imports from Django country PostgreSQL fields, JSON field to Django uh dbmodels JSON field, and uh honestly that's it.

You also need to uh run um migrations to to to create new migrations, uh but uh they will they they should not impact your uh database at all.

Yeah, and uh even if the old fields are deprecated, then we still uh we'll keep the uh stuff fields so you don't need to uh change historical migrations that use the old fields.

So it should be quite easy.

That's well, you put me at rest.

So just wanted to highlight that.

And maybe so there's currently two Google Summer of Code students.

Um what's how's that going for both of you, right?

I mean, there's I know there's a uh online you know, progress with things they have to hit each time, but how much how much work is is that for the two of you on top of everything else you do?

It's like um you know, reviewing an extra PR, and then you know, every week or a couple of weeks I I spend a teeny bit of time making sure I've caught up with the students and that you know they're doing their thing.

So um Casper is working on the Django Um stubs project, which is like the type stubs for Django.

Um and he's working on the MyPi plugin for that, so that you know that will be hopefully be improved over the summer.

And he's we've just done the first assessment and he's merged a couple of small PRs and helped a couple of people, and he's going really well.

And then Ahmad, who's working on Django itself, is um helping to make the parallel the test runner work in parallel for Windows and also now but on modern versions of macOS.

So um recently macOS changed the default um process spawning method to or forking method to process start method to um spawn rather than fork, which is a low-level thing.

But the the difference is um a spawn is how Unix does it and it inherits the parent process's memory space basically, whereas a spawn creates a whole new brand brand new process entirely.

Um and from Python 3.8 that's the default um process start method on macOS because it will forking will stop working in a future version.

And so from Python 3.8, the the Django test suite doesn't run in parallel on Mac OS, which is a real pain.

It's like wow, because it goes from like six minutes to about 20.

It's this is not a good change.

So, you know, we're hoping to um get back up to parallel running on macOS and on Windows, and he's hoping to add Oracle parallel running as well, because the the Oracle test suite currently takes over an hour or something.

It's like ah come on, Oracle.

But if we could and only because we can't run it on parallel, but if we could get that working, then you know that we could cut our CI times and we could run Oracle tests more often, because quite often we don't run them because they take so long.

Um and so it's quite a good project, and that's really interesting.

And Tom Forbes and um Adam Johnson are helping with that, and uh David Smith, who's a new contributor, is helping to on the project management side as well.

So there's lots of people involved.

Well uh Google in particular has been very generous to Django, both with the Google Summer of Code, the sum uh season of docs, which um we've been Django's been accepted for.

They also uh donated five thousand dollars to Django security efforts earlier this year, so Django's been a big help.

Or Google has been a big help to Django, excuse me.

Um okay, so other features.

So there's so many I mean, there's so many features in 3.1.

I don't know.

Uh you probably can't keep them all straight in your head.

What is there to say about Pathlib?

I mean it's more elegant than the old the current way of doing it.

Uh but that's that's a a novice answer.

What what are what are the two of you excited about with Pathlib?

You know, why was it added?

Because that's it's a major change to swap that out.

Tell me tell me what it is, Mary.

What's what's this pathlib thing all about?

One basically all settings uh support uh pathlif instead of uh uh instead of the old way to do this, so use uh uh OS hooks to to create uh proper paths.

Nothing's gonna see if I've got a path, so like um the classic one is base deer in your settings, right?

And that would normally be what was it, os.abs dot path.absolute dear.

You know, blah blah blah dear name dot dear name, like this kind of horrible construction you do, and then now you can just create a path which is path of the file, and then you know it's it's an object.

I mean, and you and people have been you've been able to use it on your own before, but now that it's just baked in and when you run start project, boom, beginners will have it.

So I think it'll make it.

So there's two things.

Um, and there's there's one place inside the code where we have to cast back to a string that's in for SQLite, um, because it doesn't yet accept um a string in that a path object in that place.

But we cast back for you.

So everywhere that in the settings that you would have used a string, you can use a path.

That's the first bit.

And then the second bit is that we've updated the template so that when you create a new project, they automatically use these new path lib objects.

I mean, so just you know, thinking about things that people will see when they use 3.1.

So one is the admin, right?

There's there's a side sidebar has been improved, a little bit easier navigation on it, I believe.

Well it's kind of cool.

You can just choose a different model or a different change list or uh another different view that is on the level that you are currently using.

Um there's also uh an auth change.

Um I'm checking the notes here, but where the the hashing algorithms change, right?

It's SHA 256 instead of SHA-1.

Yeah, exactly.

So uh currently by default it is not used uh anywhere in in Django.

So cookies, uh station, uh Steiner, uh password uh reset mechanism, all of them are currently using uh SHA.6.

And uh still all uh support for old hashing algorithm remains until uh Django 4.0.

So uh even if we update to Django 3.1, then uh everything should work properly, even with old hashes.

Yeah, same origin.

So that what that means is you won't send cookies to, I don't know, third party or requests from third party sites, right?

And uh I'll put a link, it's in the Django forum.

Uh Carlton, you and Adam Johnson, because I think Adam Johnson made the change.

He's he's holding it.

He's got he's he's got Google Calendar reminders to check the spec to see you know when it's all updated.

The interesting thing with secure referral policy that people have hit is Chrome now um if you're embedded in uh if you if your site is embedded in an iframe, right?

So if you're like Discuss is a sample, then Chrome

And that wasn't that was a change that just came came in in 2.2.

And so people on 2.2 are having to create a middleware to work around that.

And okay, it's not a tiny thing, but from 3.1, you you can just set that in your settings file and that would just work.

There aren't many people who are embedding their frames in their sites in iframes, but those who are there will, you know, that's useful for them.

Um it's on the way.

Um Marius, what are your thoughts on what we can look forward to in 3.2, you know, nine months after 3.1 comes out in August?

So I would like to add fun uh functional indexes uh to Django in Django 3.2.

Uh so I think that currently is the biggest feature that I would like to include in the in the next release.

Well, yeah, so okay, so one thing that didn't quite make the cut for Django 3.1 was um proper reload um support for run server when you're running with ASGI.

So at the moment, the way the best way to get that is probably to install channels and use channels version of run server.

Um there is a you know there's another option out there, Django ASGI run server, I think it's called, um, which just which pulls in Daphne and will do the runs the the reload for you.

Um so it's that replaces the run server command.

We need to get that in because it's not a great situation where for 3.1 people will be run server, but they won't be getting reload um onto the ASGI environment.

So we'd like to I'd like to get that done for 3.2.

I'd like to get it in for 3.2.

Because I remember using Django before that was the case and you had to quit and start.

I mean, I know it was uh it was a while ago, but as a you know, as a user, I remember that was one of those things I was like, oh, this is so nice that I don't have to quit and restart my server all the time.

Maybe it was, I don't know, it was one eight, one nine, something like that, but makes me feel a bit like a dinosaur.

I'm like, I used to have to stop and start the server all the time.

It sort of had would detect Python changes, but not any other things.

Or I don't I can't remember exactly what, but I mean like you if you added a new tempate folder, that was never picked up.

I don't know if it is

these things.

Well, and part of it is because you, you know, if you make a change to your settings file, that's you know, needs to be those environment variables loaded in.

And so I think that was later on the reloading, you know, the HTML and CSS came earlier.

Anyways, a usability thing.

So any anything else?

This is you know, we wanted to dive a little bit deeper into the 3.1 and 3.2, shed some light on the role of fellows.

And it honestly makes me feel a little bit better about I've been grumping around about updating my books, but thinking about all the things that the two of you have to do for Django in terms of back support makes me feel a lot better.

So thank you for that.

I mean, async was the big thing, right?

Okay, so that's in now, and okay, we'll see what happens.

So, you know, that that needs to grow, it needs to mature, people need to start using it.

You know, who knows where it goes next.

There'll be loads of features to the ORM coming over the next nine months that you know it's constantly ticking over.

And wow, just the changes that are made over time are just super.

But 3.1 is just well, 3.0 point, you know, extra number, right?

It's just it's not a massive breaking change, it's not a massive jump, it's that evolutionary thing that Django does so well.

Um, and 3.2 will be more of the same, and you know, that's what's exciting about it, is because you can be on the latest version without worrying.

So if you will bump to 3.1, then you are safe.

We're recording it um two weeks before.

Um, people can still go and pip install dash dash pre-django and try out 3.1.

And you know, if you find any weird breaks, uh do let the Django community know, but it should be in pretty good shape.

Um, you know.

Yeah, the CI, yeah.

And tell us what what went wrong.

Then you need to wait for uh a month for a minor release to fix anything.

I know I've been you know looking at your pull requests and I have some sense of of the work that you and Carlton do, but not a full sense, but it's a lot to make it smooth.

And um, so thank you for all that work.

Thanks.

Carlton, do you want to take us out?

Where can people find more info on Django or the podcast?

Well on Django, you go to docs.jjango project.com and you know releases and ch read the 3.1 release notes.

So you go forward slash 3.1 and releases.

Oh, we'll link to how to upgrade too.

There's there's Django docs on that.

We're linking to how to upgrade your version properly.

And then we're obviously Django Chat, and we're Chat Django on Twitter, and there's a website, DjangoChat.com.

And um, yeah, join us next time.

Bye bye.

Thank you.

Bye.