Transcript: Deployments

In this episode, we're going to talk about deployment, the things you need to do to change your Django application and options for putting your site online.

As ever, I'm joined by Carlton Gibson.

Hi, Carlton.

How are you?

So let's this is one of the top questions that I get as a Django educator.

And it really is confusing the first time or several times you go through it.

So there's a long list of things that are different on local Django versus deployment Django.

As a Django fellow, perhaps you could talk about why Django makes some of the choices they use, such as SQLite and all the rest, to make local development easier.

Like, you know, SQL was a separate programming language back to the day, so surely it's called SQL rather than SQL, so we'll call it SQLite.

I, you know, I don't know.

Anyway, SQLite, why?

Because it's in comes embedded in um Python and you can just get up and running and you don't have to set up a different server.

And and so when you do start project and start app and you can run server, you're up and running with SQLite or SQLite, I just said SQLite.

Who knows what it's called?

You're up, yeah.

You're up with SQLite um without any extra work.

And and you know, when you're first starting, the last thing you need is oh, install Postgres.

And yeah, whereas SQLite is file-based, so it's just that little file right there, and boop, away you go.

So let's go through.

There's a list of things, and then and we'll go through maybe and talk about the choices uh Django has made.

But first we should mention there is a deployment checklist in the Fantastic Docs, which we'll link to, and you can run run a command um manage.py check dash dash deploy, which will in an automated way say, hey, you forgot some of these like must-haves.

Because there's a bunch of nice to haves depending on your uh project, but there's a bunch of must-haves of of which switching over from SQLite is in most cases one of them.

So you know that normally when you when you migrate the checks get run, but they that to make sure you have you haven't misconfigured an admin or misconfigured a model, but these extra deployment only checks, you have to actually enable the flag and run the command.

So let's go through the list.

So number one is probably debug.

So in the settings.py file by default debug is turned to true, which gives all sorts of nice debugging and other error collection.

Um, but you do not want to have that on for production.

You want to change that to false because that means that anyone out there in the great wide web can see things you don't want them to see about your project and it will be insecure.

So that is probably number one.

If you had to pick only one thing, is switch debug to false.

You know, it's a lot of different things.

It's everything.

I mean, you can do it with uh different settings files, you can do it with environment variables.

I'm curious, do you have a personal preference for how to do it?

Because it's not it's actually a little bit tricky configuring it so it automatically knows, oh, I'm in local, you know, change debug, and when I'm in production, change it to something else.

No, okay.

So most many people are probably aware of the two Scoops of Django book, which came out a couple years ago, which advocated multiple settings files.

So you'd have a base settings file and then a local settings file and deployment settings file, which would override, and so you would sort of chain them together and run a different set of settings files depending upon which environment you are in.

That's the standard pattern people are aware of.

There's nothing wrong with that.

I have been using Docker a lot recently, and with Docker, you can use environment variables, which is 12 factor auth type of deal.

And so there you can also override, but basically you can have a single settings file, and then you can um override your environment variables.

But it it is quite confusing.

I mean, I've just finished the the chapter on this in my um book, Django for Professionals, and I I've talked to quite a few experts about.

I mean, I could do a whole episode on just environment variables within Docker.

So it's um

It's it's how do you set environment variables sensibly and um think crazy things like you know, because of the way the the the the the heart of the Unix programming environment is setting uh environment variables arrive in your process as strings, even though they might be integers, you might want them as integers, so you need some kind of help to cast them to the right type, or you need to be aware that they're gonna come in as strings.

Environment variables are kind of complicated.

And they are I think at a first level um multiple setting files can actually be easy.

So you've got your base ones, which are the same in every environment, and then you've got your local ones where you've got, you know, your I don't know, your local debate database set, and then you've got your production settings, which have your production database settings plus this debug equals false.

So it's actually what it's two files instead of three, the way you would have with settings, but it's the same idea.

You want to automate it.

So this is a file that has key pairs between the environment variable name and environment variable value, and somehow you get that into your environment when you're launching

a process.

But if you're gonna have multiple environments, you're gonna end up with one of these m files for each of those.

And well, how is this any different for multiple settings files?

It's the you know, there's no there's no free there's no free lunch here, right?

Uh and and you can also just hard code your environment variables into your um Docker Compose file with Docker, though um I think generally speaking, a better practice is to create a.m file because then you can um uh git ignore that file so it's not in your source control, so you're not gonna be able to do that.

The whole the whole thing is otherwise you put things like your secret key or your database password and or whatever into your into your source control and you don't want to do that.

Um and so Yeah,

So you need some way to toggle between local and production.

Um it is complicated.

You can do the settings file approach, you can do the environment burial approach.

Pick one, try not to spend too much time on it, is my would be my advice, but um, it is tricky.

And that leads us to the second, probably the second thing that you would want to change or at least make truly secret when you're going to production, which is your secret key.

So this is uh what 50?

I think it's 50 character string that's something like that.

That's automatically generated.

Um when you do start project command, you've probably checked it into source control.

You can actually, I'll I'll do a link to it, you can use the internal Django to do on the command line to generate a new random secret key, which you might want to do and then link to that through an environment variable.

But that's something that you like it says in the settings files, but you like you really want your secret key to be secret because if some it's used for all the hashing of all this all the other secret stuff within Django.

No, but

So for a um production environment, um you have one which is outside of source control, and normally I would keep that in an in a.m file, and then if um I have uh soup in a supervisor or in um whatever process manager I'm using, load that into the environment when calling the the Django process.

Yeah.

And that works with Ansible as well, and you know, ideally you can store these things in some sort of vault.

Um so um HashiCorp have got one called Vault, and then um Microsoft Azure's got one, and Google have got one, and Amazon have got one as well.

I can never remember the names of all the services, but they're a nice place where you can put those and they're online, they're shareable with with a team, and then you can have your process access those on demand.

Because this is another key aspect.

If you're in a team setting, you might want everyone to have access to the code base but you don't want everyone to have access to your um stripe API key for example

ll link to that um number three I would say so debug secret key allowed hosts do you want to tell me what allowed hosts are

respond to those two but nothing else and so you put those into allowed hosts and then Django will re reject any settings any um requests for hosts which don't aren't match that's not you know that's not there are ways of spoofing the host header and other such things but that's a nice base level check it says no I'm not going to respond to requests for the wrong website

al host and 12708000 if debug is true what's that

You remember earlier on we talked about the check framework where you write manage.py check that's automatically run and that allowed host check is run every time and if deep if debug is false then it will check that you set allowed hosts correctly yeah and that that's independent of the extra checks which the deploy deploy flag def deploy

So let's see what would be number four I would say static file stuff.

So this would be images, CSS, JavaScript there's actually uh Jacob Kaplan Moss will link to this gave a fantastic talk at PyCon on um it's called assets in Django without losing your hair and basically you need to the uh in Django what are assets are called static for legacy reasons and then there's often media which refers to user uploaded content.

So you know uh so a static asset would be an image but you could also if it was coming from users you would put it into a media and his talk kind of goes into this but the reason why there's a distinction is because you don't trust your users you shouldn't so you want to put that information not on your server and then when you reach a certain size you probably want to take all those assets and not have them on your server either even the ones you've created and put them in a CDN um like S3 or Cloudfront something

like that.

But he uh maybe your experience is the same said you know you can go pretty far with not doing that.

If you've configured Nginx to serve your static files directly from the file system and you especially if you set uh the the right headers the right response headers so that those requests are cached because they're static they don't change they you can set long live headers actually right that instance that that really basic one server instance can serve a lot of traffic.

If you're part of a bigger team you yeah you need all of that you need that front end because that's what the front end team will give you but you need to bundle it up then you need to put it somewhere where your web server can find it be that on a CDN or be that on your single web server.

But you'd be amazed how much you can get a small instance to serve if you do the right things.

So one is so the database so we we mentioned you need to decide on your production database and generally speaking you don't want SQLite right now built in support for Postgres SQL, MySQL and Oracle though Oracle may not be there for that long.

no no no let's let's slag off Oracle

And so for four five years I was running MySQL all the time and then Postgres came along and particularly with um when Django bought in the uh contrib.postgres the extra fields like the JSON field the array field the h-door stuff you know there's range fields in there now there's case insensitive text fields there's all kind of extra stuff which Postgres um gave you as well as the Geo Django stuff which works kind of better on Postgres or Postgres um PostGIS ended up moving over to Post

gres and for a number of years now that's been my sort of go-to option um I'm a big fan of

But yeah Postgres is is perhaps the beginning place to go in the Django community at this time.

Adam Johnson I'm hope really hoping to get on he maintains a package called Django MySQL which um brings a lot of the features from that are similar to Contrib Postgres for MySQL and he'll come and tell us why you should use that or MariaDB which is the open source equivalent or the free software equivalent of MySQL because um Sun Microsoft um Sun or no Sun owned MySQL and Oracle bought Sun and so with it um MySQL is is owned by the Oracle brand so MariaD

B is the open is the free software um right offshoot of that.

so when Sun bought I'm sorry when Oracle bought Sun they got Java and MySQL as the for the package

top level URLs file and do this um just make it anything other than admin because if you were a malicious actor you could write a script that says hey whatever site slash admin would be a Django site and I will just try to brute force my way into the admin and cause all sorts of harm.

So most of those are WordPress URLs but also Django URLs appear in those and these are people running automated um penetration kits automated um scripts which um uh attack find a server attack a server and find if it's got weaknesses so if you by renaming your admin you can avoid most of those attacks for Django because they don't they it's not that your admin is any more secure it's that the automated scripts don't hit it.

So it happened it's a scary place out there in the uh internet, especially if you're rolling your own server stuff.

They are fully automatically.

Yeah, it's not personal.

And so so definitely change the admin URL.

Um there's actually um that Lacey Williams and Jeff Triplett have written uh an article um on hardening your admin, which we will link to.

So that's the basic stuff.

So it sounds like a lot.

Um in practice, you will write scripts or you'll get used to doing this for your Django sites, but the top ones are you gotta switch debug, keep your secret key secret, um, allowed hosts, you'll need to put in wherever it is in production.

Static files, you'll need to think about that.

You need to choose a production database, and you should do some basic steps around the admin.

And in addition to the check, there's also um pony checkup, I believe is still the site that's run by um Sasha Roman.

Sasha, yeah, that will you can put in the URL for your your site and it will run a bunch of automated tests to see if it's secure.

So that's a good like the I I always do that after I've gone through all these things just as like a triple check.

Um but there's also within Django, there's a couple commands um that maybe do you want to cover, uh Carlton.

So migrate, collect static.

Okay.

So for me, right, there's a one, two, three of deployment, which you've got to do every single time.

Um so the first one is upload your code.

You've got to get it onto the server somehow.

Now, you know, back in the day, that's r sync and a can of beer is the job about dip deploying PHP sites.

Um, but you can um oh what can you do?

I don't know.

You can um clone your repository on the server and you can git pull onto the server, which is not a bad way, as long as you don't mess around on there and get your your your um the your checkout messed up by you know getting so

that they can't pull from the other.

I think that's the standard approach.

Um you can uh extract some kind of zip archive, you can you can even set up a um a Pype i clone and uh pu set up tools, publish it to there, but uh that's you know

You you push the code.

So you you do like um git push organ, you do git push Heroku origin.

So that's so literally you the two-step is you push it to GitHub or or Bitbucket or wherever, and then separately you push it up to Heroku, and then it will automatically go through all the steps for each time you push new code to it.

And yeah, I think they can also pull on, you know, they can watch a GitHub branch as well.

Some of these services do that.

When you push to the GitHub branch, it automatically pulls on the on the server.

That's kind of cool.

What up with that?

Ah, yes.

And that's because it makes a ton of sense.

It's because it it just takes a while to understand that your local database is not your production database.

It shouldn't be.

Um you kind of get this drilled into you with practice, but that's why.

You know, your local and your production environments are different, but it is it is confusing and it takes a while for people to internalize that.

And I don't know if you have anything to add on that.

But I see people do that again and again and again.

And I certainly went through that process myself saying, like, oh, I just my site looks how I want it to look.

Why doesn't it look like that in production?

It's like, well,

So assuming you that your local IP address is allowing you or you can SSH tunnel into wherever your um production database is, you can set your local install to connect your to your production database if you want to.

When you're creating a small blog like this, this kind of works.

So locally, you're developing on the code, but you're still when you write a blog post locally, it still goes into your production database.

And importantly as well, your media files.

So you know you create a blog post and you upload an image, and that gets saved to your local media file, not to the one on your web server.

And it's like, ah, where's it gone?

And how do I sync these things?

Uh-huh.

It's uh one thing you can do as well is if it's read heavy, you can use SQLite and you can just upload.

You can file into place.

That would that that's kind of um poor man's sync to for one of that.

So, all right, so migrate.

Uh the second command is.

Yeah, yeah, okay.

And then the second one is you've got to migrate your database.

Um and so not if you've got a hosted, you know, you've run RDS and you've got a hosted Postgres instance, and um you've you've created some changes, you've tested them locally, you've put your code up there.

Now you've got to run migrate.

And so you need to do that on your server that's connecting to your production database.

Or again, you can connect locally to production database data.

And then the third one is collect static, where um you put all the static files from your project somewhere where your web server knows how to find them.

Um that's another thing that takes takes some time to internalize how static files work within Django.

Yeah, the

That's what I always find.

Like once I understand something, I go read the docs.

I'm like, oh, like they make so much sense.

Like I would have written them this way, but when you are learning it and when you are frustrated, you know, it's like these half truths, and you're like, ah.

So it's not the doc's fault, but that is the reality of Django beginner.

It's like um there's the docs, which are kind of like reference material versus the tutorial you need, the walkthrough to

This is my this is like my my meta understanding at this point in my career is that tutorials and and documentation are different things and should be and always will be, but often you want the tutorial when you're learning, not the docs.

So don't worry about it.

We'll cover we'll cover the bases.

Um actually for image stuff, I wrote a uh a recent tutorial on how to do image uploading um or file uploading within Django that covers some of the basics on static and media files that well we'll link to.

It gives you a

So yeah, where to deploy.

So so one way you were just saying is a virtual machine where Linode, DigitalOcean, I'm sure there's others we're forgetting, but yeah, it's

They all seem to offer what you net what I'd call a hosted runtime or a platform as a service where you just push your app code and it runs it for you and you don't have to worry about it.

They all seem to offer a V a virtual machine solution where you can provision a virtual machine.

It's like having Ubuntu server there, and you have to install the right software and run anything.

But again, you can get images like base images, which you provision from that one, and it's already got the right Python installed, and it's all and then you you just have to pip install um you know Django or your your project requirements and you uh create a virtual environment, pip install your requirements, and run it, and it works.

Um, because it with that base image has it has all the things you need to run a Django project on it.

And then they all now seem to offer a container runtime where you can put a Docker container.

Push a Docker container.

Yeah, and so you know it doesn't really matter which hosting provider I think.

I mean, I've I've only done it with a couple of them.

I understand in the abstract how it is done.

I could figure out how to do it in all the others, but real you know, realistically speaking, you you know, just sort of pick one.

Um if if if a beginner asked me, I would say do not go anywhere near a virtual machine.

Um you should do it as a learning exercise, but just use a platform as a service, use Docker Azure, Python anywhere, just get it up there.

The trap of virtual machines I found is that it feels like real engineering because it because it is, you know, DevOps is like a whole separate profession and like it's like oh getting gnarly and you get in there and you fix things, but it's just endless.

And your end users do not care where your site was hosted, they just want it to work and be up.

And so, unless you are at significant scale, in which case you shouldn't be the only developer, I would say don't go anywhere near this stuff, except for a learning exercise that's worth doing.

Or if you know what you're doing, you've automated scripts, which I suspect is kind of where you're at, Carlton.

And then you make sure that your deployment to that is scriptable.

So tools here like Fabric or Ansible, and you know, Ansible's the big sort of name, and that's what that's built in Python, you know.

Yeah.

Um but you so you've got a deployment script.

So when you run, you know, my project deploy uh Ansible playbook deploy, blah, it it uploads your code.

It you know, does the git pull or whatever on the server in the right place.

It migrates the database, it collects static, it runs, you know, the deployment checks that you want to do, it reboots your process manager so that re restarts your Django application.

The all these kind of things, you know, it it refreshes your um SSH certificate from Let's Encrypt.

You know, script all of these bits, and then you can start to think, oh, you know, I've got this running on the app engine, I could easily get this going on, you know, this app service over here, or this bean elastic beanstalk over here by just changing a few bits of the script.

And then all of a sudden, instead of being able to do it in one, you can do it in three, and then you think, oh, I could do this with a virtual machine, and yeah, I can provision a virtual machine, and and you slowly increase the

That's it.

So um Tom Dyson said on the Wagtail episode that you know we were talking about the price, or did he say it, or was it in the after chat?

I don't know, but he he was saying that you know that actually the platforms of the service, they're really quite affordable, quite, you know, until you scale up quite high.

Because the time you take to configuring a production machine or whatever, is is it's like, oh, it'd be much cheaper if I did it on this micro instance that only costs X dollars a month.

Yeah, but you're not counting your hourly rate that it took to get that micro instance in a position where it could serve those requests.

I mean, I've I've wasted weeks and weeks and weeks of my time on solo projects or you know, early stage startups fiddling around with virtual machines when you know you know, thinking, yeah, exactly that, thinking, oh well, I'm saving $50 a month versus this other this platform as a service.

Meanwhile, two weeks of my life have gone away.

Unless you're in a big company and then you know you get paid for Facebook.

Yeah.

Because there are different ways you can go.

You might be more interested in that back-end system admin devop stuff, in which case learn it because you ain't gonna get the roles that you want without without that experience.

But if it's not your specialty, if it's not your what they call the area of genius, then head somewhere else.

Oh, area of genius.

Uh she read some books, so I'll have to find out what the book was.

Oh, okay.

Yeah, well, and I so I I know that I am not a DevOps person.

I am a very product-focused engineer, if that's a term.

Um, I kind of veer more toward even though I spend all this time on Django, I veer much more towards the front end and I just want it to work and talk to users and get feedback.

Yeah, um where like at the top end you've got um, I don't know, doing the graphic design for the buttons on the in the UI and that kind of stuff.

And at the bottom end you've got the um configuring the network layer for your Kubernetes cluster.

Like Django's sort of up the front end side of it, even though it's a back-end framework.

The Django's actually front end.

Yeah, you are.

Django's front end.

And I think, you know, in terms of just a side note on career advice, because I get emailed for career advice all the time, which is kind of hilarious to me.

But I would say that it is probably much more marketable to have a strong back end knowledge, though it narrows down the number of companies you can work at because not that many companies are at scale.

However, you can spend five, ten years, you know, learning how to do all these things that are it's really hard to learn otherwise.

Whereas, you know, CSS or JavaScript, you can do that on your own a little bit.

I'm just saying maybe this is the bias that I see is that people respect ten years of back end experience more than ten years of front-end experience.

And maybe that's just making the wrong clients, but

I think it's totally misplaced because somebody who really knows CSS is absolutely invaluable and it's it's just as much a skill, and it's just you know, the great experiences can't be crafted without it.

Even the most hardcore, you know, lives in the command line engineer, when they see a site, they can say that's an ugly site, and they will implicit, you know, implicitly judge it in some way.

So, anyways, um, all right, I think we covered it.

Uh, it's a complicated topic.

Um, I've been spending months and months trying to figure out how to simplify this um for for my new book.

So I understand the pain of of all this, but the takeaway is you should check the deployment checklist, you should be aware of the check flag and pick a platform as a service if you're just starting out, get it running, and then be aware that there are all these other levels.

And it's uh it it is really fun and interesting to learn all this stuff, but it's deep.

But on the second hand, be conscious of hey, am I going down a rabbit hole that I don't really need to here?

You know, what what's my career, what's my project, what's uh you know, is this a benefit to me?

Yes.

I'll talk to you then.

All right, take care.

Bye bye.

Bye.