Transcript: API-First Architecture

My name is Will Vincent.

I'm joined as ever by Carlton Gibson.

Hello, Carlton.

Hello, Will.

Since this is a question I get all the time, you probably do as well.

And this will only be our second episode on Django Rest Framework, so it's worth more discussion.

Yeah, okay.

And also that was like that was about a year ago anyway.

So um we could refresh it.

So broadly speaking, so let's just so just to tee it up, so how do you structure Django Rest framework?

And this is, I mean, because I see teaching beginners, there's a big leap in understanding what's the difference, how does HTTP work.

So broadly speaking, so I'll I'll talk until I tap out because you're the you're the expert here.

But I would say there's two main ways you can structure Django Rest framework.

You can basically have so you have a front end and a back end, and you can put that all within Django, do a monolith approach, or you can separate it out into a single page architecture where you have your backend running normally on Heroku or wherever, and then your front end is somewhere else.

But before we get into it, um let's just tee up Django versus Django Rest framework, just for people out there.

Because again, how would you describe what's what's the difference between the two if someone comes to you and says, How are they different?

So its fundamental role is that core web problem of turning requests into responses, right?

You send a web request, you get a response back.

You ask for an HTML page, you get the HTML page back, and it pops up in your browser.

That's Django's job.

And then there's a special, and it does that great, it's got everything you need for all aspects.

And you can do everything that Django Rest framework offers with just Django.

You don't have to use Django Rest framework.

But if you're gonna build what we call an API, and that normally these days it means you you're serving JSON, which is JavaScript object notation, which is for a machine to consume.

So it's like a mobile app, it's gonna consume data from your website, it's gonna it's not gonna consume HTML, it's gonna consume like a schematic JSON document, and then turn that into something that it'll present on the screen for you.

Call those APIs and RESTful APIs.

Well, that we can talk about exactly what RESTful is meant to be, but Django REST framework makes those kind of endpoints easier.

So it it it handles um rendering to JSON, for instance, setting the right content types, um, I don't know, content negotiation.

So you might want you might be able to send out JSON, you might also be able to send out XML, you might also be able to send out HTML, you might also be able to send out, I don't know, YAML or who knows what.

But it it has passers which take incoming data that could be in JSON format or um multi-pipe form multi-part form post, like an HTML form, or any other format, XML data you might send across to your web server.

It will pass that data, it has it does content negotiation about which which types of request you want, uh which type of response you want to accept, and it will render that back for you.

So that's the kind of core bit that Django doesn't have that out of the box.

And I think the even baser way to say that is because I think sometimes people confuse URLs with endpoints, is at the end of the day we're dealing with the web, so it's it's sort of the same thing.

It's an endpoint if it's serving up uh JSON or or data, and it's a web page if it's serving up HTML, CSS, everything that you see.

These were so the it the the idea would rest, the the core ideas would rest were the it's an addressable resource that you can use these verbs on, more or less.

And I think part of it is that if you're dealing with a normal website, you don't have to really know about HTTP at all, really, or the verbs.

Um, but it is a little confusing because the verbs are in the header.

So if you go to look at a page, an API endpoint, you can't necessarily just see what the verbs are on it.

I mean, you can if you you know use Postman or if you use um you know Django framework comes with a web visualization tool, but sometimes it's a little bit harder to see what all is happening there.

So they only use two of the two of the verbs.

Um and it's only in a kind of API environment that you usually use put or delete as well.

And then there's head, which just gets the sort of um metadata for the for the for the endpoint and right other things, options and stuff.

That's basically it in terms of

It validates incoming data and it um it takes your whatever you want to send back and it serializes it.

It deserializes the incoming data, turns it into an odd Python object representation so that you Python can do stuff with it.

And then on the way out, it will serialize it back to whichever format your your renderer uses.

So JSON normal.

Yeah, I mean J it doesn't have to be, but JSON's the the sort of default.

Yeah, and then you have a you have a view, so REST framework has some great views.

It has so instead of the base generic view class that Python that Django gives you, which is just view and then detail view and list view, it has API view, yeah.

And then it'll have um

All these generic classes, class-based views, which they're really fantastic.

I I the other few weeks ago I was um cracking open a new project, uh you know, really excited, blank slate, uh cracked open REST framework generics, which is the the the module where all of these um generic API views live.

And I was just like, wow, I love this.

It's just so nice, it's so it's got everything you want.

It it makes it makes crafting APIs just just uh a delight.

And so I was I've just had a little geek moment.

Yeah.

If you want to see real examples, I have a whole book on it, Django for APIs.

And also if you're struggling with the uh DRF tutorial, I also will link to I have a beginner's guide to the tutorial where I hold your hand and you build the exact same amount of code, but it's very beginner-friendly.

It doesn't assume any sort of knowledge.

So those are two resources if you're hearing what we're saying but kind of want to do it yourself.

Um so the structure though, let's talk about the structure.

I mean, do you agree with this saying broadly there's sort of two options?

You can put the the back end and the front end within Django itself.

You basically you put it in the the static files, we can talk about that, or this SPA single page app where they live on different domains.

There's new, you know, we can we'll dive into those, but I mean I think the standard thing, if someone's talking about a single page app or they're talking about APIs and front ends, they're talking about something that's living, you know, the back end is living at app.example.com and it's serving up stuff to or excuse me, it's living at API API.example.com and it's serving stuff up to a dedicated front end, which is at you know app.example.com, and then you you know you deal with uh course headers, but you're doing cross-origin requests back and forth

So the way that's most common.

What would I say?

Yeah, I mean, and these it there seems to be this massive um I think because JavaScript or front end development has become become so specialized and so difficult, yeah, that there's this tendency to think, oh, it's we the back end is just an API.

Now there's there's a there's another way of looking at it.

The way I it doesn't matter whether you serve it at two different domains or not, you know, you can serve it at API, an app, or whatever.

You but you can s you can have your single page served at a single URL from your Django application and the API served from the you know other URLs on that same.

It doesn't have to be different domains, it's just you know, that's a pattern that some people that that's popular.

But the way I like to think of it is are you generating doing single page apps where you're literally it's you know you basically load a single HTML document which is almost empty and it just loads in the JavaScript, which then builds the application by fetching from the API.

So that's a single page application model where you know and you never leave that one URL.

I mean it might get updated, but you the you never refresh the page.

That's the point.

You never read, you never do a whole page page load, right?

Because they they tie into the browser history, so you can click the back buttons and that all works.

to SEO.

Yeah, that's a tricky.

And a lot of times that JavaScript will, you know, so say you've got a form, and you that form is rendered as a static HTML form, and it's fully functional as a static static HTML form.

And if you click submit, it would do a page refresh.

But then you load some JavaScript on top of that, which takes over that submit event and will submit it to the API directly without doing a page refresh.

So if you look at a I don't know, I won't pick a particular example because no doubt it would be false.

But you a lot of sites that I think have done very well, they they will be functional even if the JavaScript doesn't load.

So for a very small.

And it's a

It was, you know, you would sprinkle in the JavaScript, and jQuery was like the library that you would use to do that, but it was very much you sprinkle it in as opposed to nope, we're gonna toss it all over to dedicated front-end lands.

But there is a bit of a bit of a load time.

Rather than saying it's a monolith where it's deployed as a single application um versus um two applications to different subdomains.

It for me the distinction is are you doing SPA or are you doing largely service-side rendered or the at least the initial render is service side with then JavaScript laid on top of the.

I mean you you I'm trying to think there's one other step you need with that, I think, but maybe I forget off the top of my head.

Maybe you don't.

So you can you want to cache your static assets.

That maybe gets into the performance aspect we could talk about, um which actually I wanted to ask you.

That's a slight tangent, but since you maintain Django compressor, um Well,

You're involved with this, which I believe broadly speaking, third-party package, it will minify your front end and do some stuff on the images, right?

That's basically Yeah.

So I mean I'll just assume that you so let's assume you're loading your you don't have to do it this way anymore, but let it it still works and it's great.

You can but let's say you're loading all your um CSS at in in a single block at the top, and you might have three or four CSS files.

And it will what it will do is combine those into a single CSS file.

And if you're using pre-processors like SaaS or um post CSS or anything like that, you can run um your your CSS files through the pre-processor, then it will combine them, then it will minify them, and you then it will stick them in a known location so that next time it doesn't have to do all of that, all of those things.

And it will do the same for your JavaScript.

And you can either load you can load your JavaScript at the bottom, which you always used to do.

You used to put it right at the bottom, but you put it all together, but you want it in a single file on HTTP one.

It's not so simple on HTTP2.

You know, we're see we're all still ahead, we're still way ahead of the curve.

Just do the old stuff for now until you know the new stuff's proven exactly how.

You you can start stick it at the top and it will load parallel to your page rather than blocking it, which it used to do, which is why you put it at the bottom.

Um so my I I think the biggest performance thing around all this is caching and you know people not using a CDN.

So I guess my question to you was if you are are using a CDN, so Cloudflare or something in front, um, does it also make sense to use Django Compressor or is that a little bit of overkill?

So um the the way I run Django Compressor, this is this was gonna be a DRFF set.

So the way I use compressor is it's got a compress offline um setting where where you um as part of your build step, as part of your deployment step, you run the compressed management command and it generates all the um compressed files, and then you can put those wherever wherever you want.

And there's a manifest which it will quickly look up what the what the finished files were called and insert the right link element into the HTML as it's served.

But what's good is you get you get a single conjoined, compressed, minimized um CSS file.

And you know, if you're using other tools like Purge CSS, which eliminates your unnecess unneeded CSS, then you you know you get a nice small package and your page loads quickly.

Well, and and so to bring it back to our our our DRF discussion, part of uh I think part of why the you know the single page app structure is popular is also in part because now you do have it's rare to have truly full stack people, uh full stack teams.

You have such a divide, it's sort of nice to have separate code bases for back end and front end.

Yet if you're solo or small developer, you know, two or three, you know, maybe that's overkill, right?

I mean, if it's you or me building an API driven site, um, you know, I would probably start with serving it within Django, and then if I needed to, and I really needed to, I would go SPA, even though just because it's a little bit simpler.

You know, hard to say.

But the cultural thing is I think is an interesting take on why would you choose one over the other.

And you know, you don't want JavaScript engineers touching your back end and probably vice vice versa in a

large setting.

They want to build their app, they want to build it.

How dare they?

What I like about Django Compressor is it may I'm I'm I'm you know, I'm I'm happy to go all the way to you know Photoshop if I have to, but I'm a back-end fella when it comes right down to it.

So what I like about say using Django Compressor is it enables me to do get to a point where if I have to hand over, hand off to somebody who's a front-end profession, you know, a solid front-end professional, what I hand them isn't embarrassing.

You know, it's like, yeah, this is fine.

I'm you know, I'm I'm I'm able to use you know Webpack or Post CSS or whatever it's awesome.

You know, whatever your whatever your tool they like, I'm able to do that.

I'm saying, oh yeah, here's my build steps, and you know, I just run it via this compressor tool, which you know runs.

And they're like, okay.

And they can go off and build their thing without me feeling that I've you know I've got to hug my face.

But it is true.

I was gonna say, um, if you're curious if a site is truly an SPA or not, if you go look at the uh if you view source, if you see just a teeny bit of HTML and then just some long glom of JavaScript, that's an SPA.

If you actually see HTML and elements being laid out, that's probably more of this, you know, JavaScript sprinkled in.

But sometimes, you know, if you go and look at view source, because sometimes people say, Oh, can I I want to they want to build such and such a site, or I'm just curious.

If it's if you can't make any sense of it, it's an SPA.

It's just a whole glob of hopefully encrypted but compressed JavaScript.

Um or not not encrypted, what what do you do?

You um

Yeah, you minimize it.

Source maps is the thing.

So when you when it's totally compressed, all the um variables are renamed.

So like you know, I've got I've got a nice meaningful variable name and it gets renamed to A.

Yeah, yeah, exactly, exactly.

So if you if you I think Basecamp do this, I think they you know they some um I think uh David Hanmeyer Hansen was calling someone out for their bad, I don't know, age single page application or something, and and someone's saying, Well, but what about your JavaScript, which is all minimized and no one can make head nor tail to it of?

And he was like, Yeah, good point.

And they started shipping their source maps.

So you can, you know, view source on that, and you can actually then get into the JavaScript and it will be pretty printed with the the original source code rather than Oh,

I don't know if I've I've used that and I didn't realize browsers do that now.

Yeah, cool.

Well that and that's a great analogy of I mean, I've this happens to me all the time that I'm you know opinionating on something or other someone, and then that and I sort of either I think, oh, I should do that, or they you know ask me, like, oh, are you doing that here?

And I'm like, well, actually I'm not, but now that you mention it, I should I should.

Um I do want to mention um in hat tip to hang

You asked the question.

I didn't get to watch it.

Oh, I'm sorry.

Like, so I so I think that you know you are you asked something like um, what do I think people should do?

And I think, yeah, okay, I understand this whole drive towards the single-page application, but there are you know four billion people on the planet who have devices not capable of running a full React app on networks that aren't able to download them quickly.

And for me, server-side rendering is still the way forward, and that should be your first option, and progressive enhancement on top of that, which is harder to get right than it is to build a single-paid app.

That's that's the goal.

Um, that's what you should be aiming for.

And it's just better and richer and nicer and more in tune with everything you want.

And when you and your app will display on an Apple Watch and it will display on a Samsung smart television and it will display everywhere.

Whereas your your rich JavaScript application, you'll suddenly find that over the rubbish mobile network, the JavaScript page didn't load and they get a blank page.

No, it's very true.

I two things I wanted to add.

Or not even add, just digress from what you said.

Uh so the first is um there is another option for structuring things, which I have to give credit to Jeff Triplett for mentioning this, which is if you're using Docker, which I often do, you can run it all within um Docker.

So you can have a in your Docker Compose file, you can have a separate image for the web and then for the front end and run it all in one container.

So you can basically do an SPA within Docker.

Um I was looking for more code example or some guides on this, and I don't think I don't know of them.

If anyone does, let us know out in the show notes.

But you know, since Docker is just a box around everything, you can also do it within Docker, and but I don't know if that's necessarily that much easier because then you're in Docker land.

But I wanted to mention that.

But the other thing is when we were talking about Django Rest framework initially, part of why what trips people up I think is authentication is a lot more complicated in an API world.

Um, because uh basically because traditional web app uses session authentication, we have a cookie, so you user logs in, session object is created, and the ID is sent back to the user and stored in there, uh stored as a cookie, and then that's passed back and forth on every request.

So that basically one ID can be used to unlock you know 20, 40 attributes that are stored in the server.

So the server is the source of truth.

The ch problem with that, of course, is if you have multiple front ends, so you have a web app and a mobile app or two mobile apps at the same time, sessions kind of breaks down.

So then you go to tokens or JWTs, and you know, this all is confusing.

I have a whole DjangoCon talk on it, um, but I think that's another thing when people switch over to DRF land that they go, oh my god.

Because the official docs list four built-in ways to do it.

I actually just invited him.

I don't know if he listens to the podcast, but Dave, come on, let's talk about it.

It would be nice though.

Um so while they work well longer term, that would be nice.

They should be rolled into Django, just as two-factor auth should be rolled into Django itself at some point.

That's a

But you know, so that I guess I just wanted to mention in the context of what Okay,

So um there's there's so much capacity to maintain Django Rest framework and keep it going and keep it, you know, largely bug-free.

I mean, you know, there's a hundred tickets or whatever, but there's no, you know, it's it's got a regular update cadence and it it's you know evolves over time and it's you know, serious bugs are few and far between now.

Um most of the remaining tickets there are, you know, all there's a bit of uh vagueness about the contract in, you know, if you get in this really specific use case, it's not quite clear how it's going to behave with nulls and uh you know full of things.

But there's there what there isn't capacity to do is take every possible extension and bundle it inside Django Rest framework.

And as soon as that started would start to happen, Django Rest framework would become unmaintainable.

And so it's much better that there's a third-party package for IWAuth, and there's a third-party package for JWT, and there's a third-party package for, you know, if you need to map your um your serializer keys to camel case, because that's what your Angular JS client requires, then you know, there's much better there's there's renderers outside of Jack and Arist framework itself that you can use.

I mean, it's better in that it's not unmaintainable, but you also are shifting the burden of maintenance to third-party apps.

I mean, for example, JWTs, our friend Jose Padilla had the dominant one for quite a while, and then his life got complicated, and then there's a new one, simple JWTs.

And this is pretty core stuff that you know, and actually

But here's the thing if it is core, then it'll it'll end up being maintained.

I would say.

Oh because there's a demand.

So there's companies using it, so there's companies that'll stump up developer time to make sure that the package works.

And we should ask Jose a little bit about why he sort of I mean, I you know, I know he got busy and he wasn't monetizing it directly, but he had, you know, the dominant JWT package for quite a while, and thank goodness there's a simple JWT one, but you know, there's a number of issues on Jose's one that um cropped up.

Well, we're gonna at DjangoCon Europe, we're gonna I'm gonna be at the sprints.

We're gonna you know get more people on board and maybe improve the docs a little bit around that to make that easier.

I I did want to add too, um so I recently well, two things.

A lot a lot of two things for me today.

We haven't recorded in a while, so that's maybe that's why.

So one is You've got a backnote.

Um so there's some stuff on there, and I'm slowly porting things over, and and we'll do a whole episode on building that and things learned.

But I put a blog post out, um, actually again with Jeff Triplett, um on 10 most used Django packages based on just PyPI for the last 30 days.

And um we'll link to that.

But it was interesting to me in first in that so it goes Django, obviously, Django REST framework, um, Django Cores Headers, which our friend Adam Johnson maintains, because you you know, if you're using an SPA, you have to set the course headers.

Yeah.

Uh Django Redis is number five, which I was like, that's pretty high.

Um Django extensions, which makes sense.

PyTest Django, Django Storages, Debug Toolbar.

And then actually, I wanted to ask you, Django appconf is was number 10.

Yeah, yeah, yeah.

That's it, that's so that's

I mean, I I I haven't used it personally.

I mean, I looked at it, I was like, oh that makes sense.

But what it does, but it enables

So if you write a third-party reusable app and you want settings for your reusable app, Django appconf is a nice little wrapper around that.

So that's you who somebody big must be using that though, because it's used by Django Compressor.

I don't know who else

Yeah, maybe it's maybe it's wrapped into something else.

It must be.

It just seems way too high on the face of it.

Well.

It's used by X package and Y package and Z package, and they sum up to a lot of downloads.

Yeah, to unravel that.

Uh I since I'm in a publishing mood, I also published my super opinionated essential third-party packages, which was 18, 19, it was 17, and then people, of course, the floodgates opened with advice from people, and I think I added one or two, but um my personal highly biased takes on things I build everything with that um regardless of application.

So, anyways, those there's some plugs in there.

I remember a post it must when I was starting out with Django, it must have been on like, I don't know, Jacob Kaplan's Morse's blog log or something like that.

Like one some one of the early people in Django.

And I was like, you know, you're getting into Django and there was this post about, you know, these five top packages and it's

all awesome.

Well could um

But you know like I think Django debug toolbar was on there or something like that.

And I'm still using that today.

And you know everyone of course is like well here's five but then here's another ten.

So that kind of brought about because really five's not enough.

So that brought about decision for

Yeah and I mean

So they're sort of like top five essential and then um yeah it's a good question.

And I think it's helpful I always like to hear from people I respect in the Django community what they're mentioning.

I mean you know even um just just last week um you mentioned Django Sesame package by uh Imerick on magic URL opening that I didn't know about that's been around for for forever and is perfect for something I need to do.

So there's always more to always more to learn.

And um about five minutes before that you mentioned the importance of caching and of course that's where Django Redis comes in because everyone's using it for their um Django cache back end right?

30 minutes we got a couple minutes.

You know you so there's two type or I I think of it and you can dispel this but I there's when I think of caching so one is just uh you know use it you use a CDN so you throw Cloudflare in front to serve up your site if it's basically a static site.

So my learnjo.com site is essentially a static site so I have Heroku I mean I have uh Django hosted on Heroku and then I port Cloudflare in front of that and it basically loads you know instantaneously the the Redis stuff is when you're doing caching you know on your back end on your database basically um which is a more real world use case.

Yeah it'd be interesting I mean there is a whole discussion of you know Redis versus Memcache which I think we've talked about yeah we talked about the caching one where basically Redis is a little bit simpler these days and seems to have the mind share though memcache works fine as well.

But Redis

ediate people wouldn't use those um so I don't know it'd be nice to know what the full ecosystem looks like.

But yeah hard to say I mean who knows but ballpark probably accurate.

So all right what else I think that's that's it.

Hopefully that helps people we're very open to questions.

I think there's probably a lot more episodes we can do on Django and APIs and DRF.

Um so do let us know as ever on at Chat Django on Twitter.

You could also nobody's done this actually but if you want to send a voicemail and we could put it on air uh Djangochatpodcast at gmail.com you can send us a voice memo and um if it's a good question we'll play it and respond to it.

Anything else Carlton

Good.

Alright join us next time folks