Transcript: Django for Professionals - Book Release

I'm Carlton Gibson.

Hi, Will.

Hi, Carlton.

How are you doing?

So today's special episode week for me.

I just released my new book, Django for Professionals, which I've been working on for over a year.

And so I thought in this episode we're going to talk about what makes a website production ready in Django and what's that leap from the defaults you have when you spin up Django with SQLite and all these other nice things to like a real website you would deploy to thousands or millions of people.

Your first two books, Django for Beginners, Django for APIs.

This is like modern, up-to-date, very cutting-edge techniques that Django developers are using on the biggest sites in the world and sharing it with everyone.

And there's just no good resources out there.

And having spent a year banging my head on this, I kind of understand why that's a challenge.

Um, but we can talk through uh, you know, what those things are that make something production ready and kind of where I mean, because you can go up to certain things and say, okay, this everyone agrees that these are steps you should take, and then it goes into the realm of purely subjective and hyper complex.

And so in the book, I go right up to that point and then I discuss options, but I don't, you know, there are some things that are sort of philosophical debates around web design in general, and you know, Django itself can't solve those.

You might take one, you might take another.

It's not necessarily right.

I mean, you could have you could have a 500-page book on just you know deployment for one platform provider, let alone the dozens that are out there.

So I had to draw the lines in some places.

It's like, you know, they're big.

Um anyway.

But we deploy with Heroku, partly because I think it's one of the easiest ones out there.

But even Heroku, I mean, it took me quite a while to sort out the docs on containers.

We use Stripe, we use Sengrid, and you know, these third-party platforms, it's hard to figure out how to use them.

Uh, I don't quite understand why they can't have a basic guide for all the you know five, six major web frameworks out there, but for whatever reasons, they um

But is that because they're pushing forwards and like so the Docker deployments are some uh somewhat new, and so they just haven't had the time to backfill the documentation?

I mean, so let me let me pick on Stripe slightly because I love Stripe as a company.

I think it's the best payment processor out there.

But so Stripe is rolling out a brand new um, so we use checkout uh in the book.

They're rolling out a brand new API which is not fully documented yet, and I know this because um I have some friends who work on checkout there who I complain to saying, hey, I'm having trouble with this.

And I've written official Stripe tutorials for for Meteor and for other platforms.

So I mean I know how Stripe works, and I'm struggling with this.

And I I keep saying, like, you should just have a basic guide that's up to date.

Pick one engineer, like you know, one hour a week, and just go Rails, Django, Express, I don't know, PHP flavor.

And even if it's and just show how to do a basic implementation.

Um, and I think it's a combination of, yeah, they're pushing forward, they're constantly changing things.

I think they feel if they do an official guide, there's just too many layers of I think bureaucracy, honestly.

But it is one of the challenges of this book is that a modern Django stack has a lot more than Django in it.

And so I found myself repeatedly fighting with non-django things.

Um, Docker would be an example, and I want to use it and show how to do it, but it's not a book on these other things.

Um I think when people read the book, it will probably seem like, well, what's the big deal with Docker?

Because I, for example, like, or Stripe, like I make it as simple as can be and yet accurate.

Um, but it took me a ton of work to figure out how to do that because these are kind of untested things in a way.

I mean, Stripe is literally changing its API.

I mean, Docker, there's so many different ways that experts do it.

So ultimately I had to pick one and then really understand how it all fits together to try to make it simple because it's not a book on Docker, it's a book on Django.

It's not volumes, it's mounts, it's

Yeah, you know that yeah.

I mean it's not as bad as like React or something, but it's uh yeah, it's ever changing.

And I mean the big issue with Docker, I mean, so we use Docker in the book because you can have a containers, which I I speak about, so you can have an isolated uh operating environment.

And that is good for your development, that's good if you're working on Teams, that's good for pushing to production.

I think this is a new thing that will become more and more common where you the containers that you use locally, you push those up into production.

That actually makes it easier to switch platforms if need be.

You don't get locked into a certain platform provider, but it's all new.

Um, and specifically with Django, there's issues around um when you're working with software packages, if you use pip or pipemv.

So we use pipem in the book and you have lock files, and you basically have this tension between am I working locally or within Docker?

And anyways, in the book, I'm very pleased with how it turned out, but it was probably a solid four months journey for me to iron out the Docker parts.

Um, but it's important because that's what most uh developers are using these days.

They are using Docker, but they're doing it in um not everyone is doing it this way, many are, but it's not a there isn't a canonical guide to it.

So I sort of had to

You're you're right now.

There's just like so much code and so many decisions, and you know, I find it overwhelming.

So in this book, we just go step by step and we build up all the things that you need, and that's a lot of work to do.

And I think also a lot of people don't do that because most people don't have the luxury of starting a big project from scratch.

Um, maybe if you're at a consultancy, even then I think it's rare.

So most people aren't, you know, user registration, for example.

Like I spend a lot of time on this because I do build a lot of new projects, but most people don't.

You know, you airdrop into a project or you start something and barrel through, and then you kind of are in the mud and mire.

But if you had a clean slate, how would you do that?

Um, that's an example.

One of the chapters we talk about using custom user models, but we also talk about abstract user, abstract base user, um, customizing all the templates using Django Alloth.

Uh so the first half of the book is really standard steps that you would use in any new project.

So configuring static files, uh environment variables.

That's basically gonna be the same in any project.

And then the second half we focus on the bookstore, but again, we focus on things that are meant to be universal.

So we talk about permissions, we talk about search, payments, file image uploads, security, performance, testing.

These go beyond the tutorials, these go beyond you you gotta create these stuff yourself.

So that's kind of cool.

And you know, so in this book, I really didn't want to just throw my hands up and say it depends.

I really wanted to take an opinionated, uh make opinion opinionated choices, but back them up and say, you know, I feel confident saying this is something that this is a Django best practice.

And then if we get to a certain point and say, okay, now here's where there's a diversity of opinions, here's what they are, I'm gonna give you mine and show you that.

But trying to draw that line of, you know, where where is it more my opinion versus the community's opinion?

So yeah, so that was some of the challenges of this book.

I mean, the first couple books, the first two books didn't take this long to write, but um I'll probably write a blog post on things I've learned when I've had a couple weeks to settle on this.

But I think yeah, writing an advanced book is a lot, a lot harder because it's you know, more figuring out the technology and also you just more and more you're bumping against these um debates within the community on how to do things.

I mean, one of the the the I call it a meme.

I don't know if it's a meme, um, but one of these ideas that comes back again and again is that you need to use um uh you need to be somewhat conservative in your technology choices because otherwise you're spending too much time strashing about what's on you.

But deployment is still an unsettled project, and and all the the practices about how you get your static files in the right place and exactly how you serve them.

Um what did you choose there, by the way?

Yeah.

Okay.

But yeah, that is that's another example of we're just jumping all over the place, but that's fine.

Like when you so let's say you so you've used Docker, you set things up, then within Docker, you can create a separate production environment within Docker, which you need to if you go through Django has this fantastic deployment checklist.

So even if you're using a production database like Postgres, beyond debug, there's a whole list of things you need to fiddle with to be secure when you go online.

I mean, we've talked about this in some of the other podcasts.

We need to harden the admin.

We talked about one previous episode.

There's forcing SSL, clickjacking.

There's a whole list of things that if you haven't done before, you you really should do.

But you know, again, there's an 80-20 on everything, especially with deployments and security, secure cookies.

I mean, you can spend all your time on this stuff, and at some point you have diminishing returns.

And yet there is like some bare minimum things you absolutely should do.

You know, allowed hosts, for example.

So we cover the 80-20, and then I talk about, and here's where you can go nuts on it.

But uh there's always more to do.

And so I try to pass along at least what I think is good judgment and how you spend your time and how do you get as what the bare minimum and then be aware of here are the next steps to take.

So if you don't get over the baseline, it's not a question that of if your application is hacked, it's just when it's hacked.

So there is a minimum bar.

You have to get that far, and then you know, depending on your threat model, you might need to go further.

I mean, and I would say you gotta turn debug off, you gotta have SSL, you should change your admin uh URL.

That's the top three, if I would say.

But there's all there's a bunch more.

And you know, again, Django has this deployment checklist.

But I think the challenge is actually more configuring how you have you know two different local environments.

So this is where environment variables come in, which we use, because how before Docker, it was a lot harder to test your production environment, not in production.

Like there's jokes about like, yeah, I'll just toss it on production and do it, you know, do it live.

Docker makes that a lot easier where you can actually, within a container, have these production settings, run these things, check them.

You know, you could do it on a staging server, you probably should as well, but to be able to do it locally is just fantastic, actually, that you can use a Docker.

So we go through that in the book.

But it's a lot to keep in your head.

You know, you have local databases, production databases, you have local environments, production environments, and that goes across the board with all these settings.

You know, you you want different things locally because you want to be fast and you don't want to have all the security stuff, but in production you do want to switch things around.

I mean the server, right?

The default Django server, you you don't want to deploy with that.

You want to use unicorn or UWSGI.

You've choose green unicorn, gunicorn, whichever

Am I saying aroundicorn?

Some people green unicorn at EuroCorn, uh DjangoCorn Europe, and I was like, uh Green Unicorn.

I'm like, you know, Django Ponies and G Unicorn, and they're like, what are you doing?

So yeah.

So you choose Unicorn just because just because I well actually, no, I did I gave thought to this.

I've used it myself in projects, and also it is uh a little bit simpler to set up.

Um UWISGI has more configuration options.

So I think if you need it, you might want UWSGI, but as a default and certainly as a teaching tool, GUNicorn is the approach I chose.

I heard someone called it micro whiskey, the um uh Django Europe as well.

I never know these things.

Um but

I know classic classes.

I'm learning ancient Greek to so I can be an old man and read Plato in the original.

But uh that's a that's for another podcast.

Um what was I gonna say?

I was gonna say something really great there.

Oh yeah.

So in the uh UWSG docs, it says this is great if you're building a whole uh like a hosting platform or something, right?

It's deliberately more complicated because it's for you know, if you want to be able to provide a hosted environment for people to put their Django apps in, then you know, UWSG perfect.

But if you're deploying your own app, you don't need any of that.

So Gone and Con's a lot simpler, as you say.

And that, you know, that's just one example of many, which I think, again, is why because I sometimes think, why did it take me so long to write this book?

Because I'm pretty fast writer, I know what I'm doing, largely speaking, with Django.

Um, and I think it was because there's so many decisions just like that.

I mean, dozens and dozens of micro decisions where I have to think about it uh and then explain it.

And I, you know, I usually I pick one and I explain the differences because I want to be, you know, in some ways I was trying to cram everything into this book.

And it's a long book.

I mean, it's almost 400 pages, um, but it's still there's a lot I just took out eventually because I just couldn't couldn't fit it all in.

And um, you know, I'll I'll get to it in other ways.

I mean, one of the sections I'm most pleased with actually is the permissions area because permissions is I think the ORM and permissions are I would say the two things that advanced developers or professional developers spend a ton of time on, but there's not a especially permissions, there's not a lot of good educational resources, really any that I've seen out there.

And and yet the ability to create groups, to set permissions, you know, you on a on a form or on an action.

So in the in the book, we we set permissions for people who can see all the books, and you could change this to be individual books, and then when a payment goes through, we put in the view.

I was really pleased with how elegant I made this code.

As part of that action of the payment, you flip the permission on the user.

And so I was I spent a lot of time trying to figure out how to make that clean.

And I I haven't seen it done before.

I mean, people quite often there are um issues on the developer mailing list or on the track with about confusions around the permission system and you know it's quite popular.

I think again, yeah, people, you know, people get dropped in, dropped into these code bases, and you don't have the chance or the opportunity to learn or to build it up.

So we we build it up and we talk about when do groups make sense, you know, how you handle permissions.

Because at a minimum, on most sites, you're gonna have some at least three levels, probably.

You're gonna have, you know, your your admin, your superusers, you're gonna have, you know, in a newspaper example, the editor who can do more things, and then maybe a writer.

So at least three permissions.

You know, another example would be like if you had a university, you'd have teachers and students, they'd have different permissions.

But this crops up right away and is very, yeah, it's hard to hard to think about.

And if you are architecting and building this out and you get it wrong, it's a lot more work to backfill it than to get it right the first time.

When you're talking about anything security related, it's it's m it's very important to just take the extra time to measure twice and cut once because you can't afford a mistake.

And but how do you how do you learn that?

You know, hope uh this book I think will really help people.

Really, you learn it through pain and making the mistake on the job.

And that's um, you know, embeds the learning, but is not not not the best way to do it.

Um I'm trying

But to do it on a production environment is like, wow, hang on, that was my that was my business.

You know, I mean it's just that's how it is for anyone.

And in those environments, you need to, well, hopefully there's someone there who can tee up some small, you know, just how do you eat an elephant's like one bite at a time, right?

Like find some smaller things.

Like don't have some big integration, you know, testing thing that they just toss at you.

You know, find some small stuff.

Like it's gonna take a month at least to wrap your head around it.

And even then, you know, a lot of times on these big sites, you know, there's areas where you just kind of go like, ugh, like I hope I don't have to touch that.

You know, you're not gonna you're not gonna be responsible for everything.

That's you know, somebody wrote it, I did it.

Um that's why you have tests and you have confidence.

Um what else?

Oh, so search.

So I wanted to include search in the book, but figuring out how to how to do that.

Um, because search is infinitely complex, and most people just jump right to using Elastic or you know these uh solar, you know, these full throttled solutions with no buildup of how do you do it.

And so I'm really pleased with the chapter where we we add search because you want it in almost every site.

When we talk about forms, we talk about filtering and just layering on the complexity, then you can build in um Postgres, built-in support, then you can get to you know Elastic if you need it, you can do uh hosted solutions, but just doing a step-by-step solution because you know, search too, if you're using uh, you know, let's take Elastic locally, you have to spin up another server or container.

Like it's very non-trivial to get that going.

And most people just drop into that, they don't have a chance to build it up and think about like, well, how does it really work?

And really, it's a form and a filter, and you're passing the data back and forth.

And if you go step by step, you know, just for me, with my understanding, I've spent quite a bit of time with search on various sites, but having to go through it to teach it myself really helped me, and I think it'll help others understand what's going on there.

Because it's it gets complicated fast with like, you know, weights and uh you know all these things, but fundamentally it's you know user input and a filter of some kind.

To run an Elasticsearch cluster stably in production is actually difficult.

That you know, people have a full-time job doing just that.

Well, and and we had uh Simon Willison was on uh recently and on his personal site.

I mean, he's got a fairly basic implementation and it works just fine.

And I think that's another thing I want to get across is it's okay to have basic implementations if they work.

That's quite elegant.

I mean, uh Jacob Kaplan Moss gave a talk at PyCon this year on static um static files and talking about you don't need to jump right to a CDN right away.

You know, there is value in the basics and getting those right.

And um yeah, and even with, you know, just to go back to search though, I mean, I see I did a bunch of research and it's implemented a lot of different ways.

You know, you can you can do it as a post request, you can do it as a get request.

You know, you can do it with function-based views that are just 50 lines long.

You can do it with a class-based view, as I do in the book, where it's like three or four lines, you're just updating the get context data.

So um I always strive, just in my own code and certainly when I'm teaching, I want to explain it, but if there is an elegant, accurate way to do something, I will go for that every time.

But if you build it up and told the story, then that understanding is great.

And that's and that's partly, and that is true.

That is the danger of class-based views.

You're just, you know, turtles stacked on top of themselves.

And so you if you just drop in and change one thing, it does feel like magic.

Um but I do in the especially with the user user registration um chapters of the book, and actually throughout, I'll say, if I'm gonna do some magic, I'll say, hey, here's how I figure this out.

So for example, like updating templates with you know the email that new users get.

You know, I don't just jump to the the file, I say, okay, well, how do I know where this is?

How do I figure it out?

It's like, well, I literally I copy paste a part of it and I go into you know the Django source code and go, well, where is it?

It's like, oh, it's not there.

Oh, because it's part of Django Alloth, because I'm using Django Alloth.

Yeah, right.

Now I can see where it is, and now I can see, oh, there's a structure, and now I can see the other templates.

And now when I told you how to create the template directory structure in a certain way, because Django AllAuth goes, uses accounts versus Django uses registration.

So I actually spend quite a bit of time explaining how you figure this out because I want to like teach people how to fish, not just give it to them.

Because I agree, like that is it doesn't help if I just give it to you with no context.

And template loading is a co is a moderately complicated topic.

People get confused about that.

And then how do I override a template and how can I extend this template so that I just fill in this one block without having to you know re-implement the whole thing?

And uh like to go through that is actually uh you know something that people will learn a lot from, I think.

I hope so.

It took a lot of time to write.

Um, so what that's the thing with yeah.

Go on.

So you do anything there because you go to deployer, it's all right, you know, you're testing it locally, it works.

Have you done anything to when you put it up there, you find it your site crashes?

Why?

Because you've got I know slow clearies.

Have you talked about that?

We have a whole chapter on performance.

Um so there's two ways you can do it.

So one is you want to you can do things locally.

So again, with the benefit of Docker, you can test your production environments locally.

So bare minimum things, Django debug toolbar you need to add.

I talked through that.

There's actually a gotcha with using it in Docker that I walk through because you have to find the correct, what is it, the correct is it the port?

There's something a little tricky, but I I walk you through it.

But then, yeah, so you want to install Django debug toolbar, look at your queries, and then things like select related, prefetch related, these are go-to tools to basically uh they're complicated topics, but basically if you have a lot of queries, often if you can boil it down to one big one, that's more performant, but you don't really know until it's a production environment.

So I talk about things like you will probably want to implement these, but don't just do it and don't just do it locally.

Like put the site up live, you know, things will appear that need to be addressed.

Same with adding indexes to databases.

You may think, oh, I'll just index a whole ton of stuff, you know, the user ID, but you should hold off and the problems will present themselves, and there is a cost to you know, indexes, for example.

So all these things I I talk through, you know, compressing your front end assets, caching.

There's you know, Django has four different built-in layers per site, per view, template, and then low-level API.

So we I talk about how you would do it, but then also say, you know, don't necessarily just jump into doing it because that's you know, premature optimization is a problem.

Yeah, yeah, yeah.

Was it Donald Newth, I think.

Um but you know, because you're gonna get it wrong.

You you know, when you put your site out live, don't worry.

Like people are like, what am I gonna work on?

Like things will jump out at you like, oh, this is on fire, that's on fire.

That's actually the perfect time to optimize it.

Um, not just saying, oh, I'm you know, I'm gonna add Redis right off the bat.

I think you should wait until you need it.

They'll be somewhere totally different, and then you'll discover in production where those small number of performance bottlenecks are.

You optimize those and all of a sudden your thing is is fast, and you haven't actually had to do too much work.

That is another trap, is that it's when you wait for the problems to present themselves and put themselves into a hierarchy, it's way less time.

Um, but you know, again, how do you know what the problems are?

I mean, you you you you and I have a sense of these are the common things because we built a bunch of sites, we're familiar with Django.

If you haven't done it, it's a constant, just like, oh my god, where did that come from?

So the book sort of says, you know, here are the major things, here's how you would do it.

But you know, I do give the advice of wait.

Like that's some pro advice like there, wait if you can.

There are plenty of, you know, if you're worried about those things, go talk to users.

Don't prematurely optimize your site for a million users, go get a million users and then you can figure it out.

Get 10 and then figure it out.

Yeah.

It's engineering stuff is fun.

And even on deployment, I talk about how you know there there are deployment engineers for a reason, site site reliability engineers.

In fact, most engineers at a big company are working on deployment performance issues, and it's really fun to do because it's it feels like binary.

Like you can look and see, oh, page load time got better, or you know, our uptime increased.

It feels very satisfying.

But should that time be spent eking out those gains, or should you, you know, go talk to users and make, I don't know, like maybe pick on Quora, right?

Like, hey Quora, maybe I don't have to log in all the time to use it.

Would that be a better thing than you know being proud of how fast your site loads?

Probably.

So those are like engineering traps that you get into because you can document it and it feels good, but is that what you should work on?

I mean, generally you probably want to work on what you don't want to do in a startup setting.

I'm I'm answering emails, but I haven't got anything done at the end of the day because I spent all day in the inbox.

That's that's not productivity.

I spent all day improving my deploy,

I mean, that was something so I so I started.

Well, depends on

Depends on who you are.

So I was at Quizlet, which is now quite a big site very early on, and I give the founders a lot of credit that we about once a week, several people on the engineering team, as a small engineering team, would go into schools and visit and just see how it was being used.

And if you think about it, that's a huge cost, right?

Five days a week, that's a tenth of the workday.

And then really the whole rest of the day we're talking about things that we would learn, and that was a great way to see how things are actually being used, you know, things that we didn't think about would come up.

I mean, spending the time with your users is hard to do, but always the best thing to do.

Again, and it doesn't feel as good maybe as uh these known performance issues that you want to address.

Um, but there you are.

Like let's uh let's drop that.

Um how

Do you see these also these sort of 80-20s or these areas where engineers get really excited about working on stuff?

Maybe it's not the most relevant.

But in in the cost of that is extra API surface area for the entire user base of Django, and on balance it's probably not worth it for the entire user base of Django when what we could see is that this one line for that one person could just be like half a dozen lines overriding one function in a class-based view, and we'll you know, we'll say, no, look, we're not gonna add that extra service API service area over the whole API because there's this quick fix available, and then they get upset about that because they want it to be just so for them.

And it's uh and you're like, well, that you know, you could put that in your own project, or you could put that in a third-party app, or uh and that's right.

Well that's the difficulty is about

And we we include the hard batteries, you know, the things that are difficult for a user to implement themselves, we we'll we'll include, but we can't necessarily include absolutely everything for absolutely every use case because then it becomes unworkable.

Right.

o always gives you the option to roll up your sleeves and dive in and get crazy if you want which is fantastic because a bunch of some frameworks don't you know they're they're really great out of the box but they're hard to customize Django's a little bit more work but it's eminently customizable across the board uh which is its strength.

that wider environment and that kind of sounds really cool.

So um yeah so it's an exciting time.

And I should mention so the first so

Yes so you can go to the website jjangaforprofessionals.com which we'll link to in the show notes and you can read the first two chapters in the introduction there which will walk you through Docker and some projects and talk about what's going to come.

It's available as an ebook and and it will be on Amazon print and Kindle by the time this episode comes out.

If you have a choice I would recommend the ebook because you'll get updates.

I can't do that on print and Kindle but if you want a standalone copy you should go ahead and get those on Amazon.

So if you buy so each book is $39 uh dollars and if you buy two you get $10 off if you buy all three you get twenty dollars off there's links on um my site um wsvincent.com slash books which I'll put in there so yeah trying to make it easy for people to understand and and get to and um you know for me actually I've spent a lot of time all my time creating and now I'm switching my brain around to you know the marketing and displaying I'll probably do a video where I talk walk through the project and

show you know a couple minute video show hey here's what we've built this is actually something I'd like to add for my first two books is to walk through because the first book goes through five different projects and I think people kind of buy it based on trust I think like they read other tutorials I have but I can I can and should walk through and show a quick hey here's what we're gonna build kind of video.

Okay.

Even though a lot of it is the plumbing which is the interesting part.

I see amazing stuff out there so it's very intermittent.

But yeah I'll link to the newsletter that's the best way to be updated.

There's also an RSS feed um through that is available and is on the site I'll link to um and it shows up in the Django has a community feed right I'm not sure who's in charge of that but it's on the Django community feed.

Yeah so yeah so someone's submitted me uh which is very kind of them so it's what so when I publish things with the Django tag on my site it it pops up.

Yeah so it's out there um email me feedback um I'm really proud of it I'm I want to see how people think but it's been a full year of work and I think it's really going to help people out.

Bye bye.

Okay bye