Transcript: Migrating from Python 2 to 3 at EdX - David Ormsbee & Nimisha Asthagiri

Hello, and welcome to another episode of Django Chat, a weekly podcast on the Django Web Framework.

I'm Will Vincent, joined by Carlton Gibson. Hi, Carlton.

Hello, Will.

And this week, we have two guests from edX, David Ormsby and Namesha Astaghiri. Welcome,

both of you.

Thank you.

Thank you. Happy to be here.

Yeah, we're thrilled to have you. So we connected because we met at the Django Boston meetup,

And edX is both one of the largest educational sites in the world, and I believe has used Django from the beginning. So I wonder, we could talk about, do either of you know why Django originally? And then how has that been? Because you've been through a lot of these changes of Django, Python 2 to 3. And so hopefully we can dive into all of that.

The choice for Django, I think, primarily came out of the choice for Python, because this edX originally started as MITx, which, you know, it came out of MIT, and so Python was kind of the default choice, right?

The intro CS course teaches Python.

All the Europe interns that are going to work on it know Python.

And the fact that the first course was an electrical engineering course means that having NumPy, SciPy, that sort of set of tools was very convenient.

And so, by the way, these early decisions were made by Piotr Mitros, who created the original prototype for what became edX Platform and was the chief scientist for edX for, I don't know, five or six years, for a while.

What was the timing? Because I believe this goes back quite a ways, right? Before Python was standard at the undergrad level for a lot of places.

This would have been, the first prototype work would have started in, I want to say, October or November of 2011.

So sometime around Django 1.3, I guess.

And so, yeah, and so Peter also made the decision for Django.

But I mean, once you've gotten it down to Python, then, you know, Django was like one of the obvious choices.

and can i just ask because maybe our listeners don't know could you just give an overview of

what the edX platform oh yeah is and does good call so um let me show you on there oh sure so

yeah edX is um an online education platform it was founded by Harvard and MIT as Dave mentioned in

2011 we do specialize in higher education courses with courses and content from you know one of the

That's to worldwide partners from universities.

And what's the sort of scale that y'all are at right now?

Because it's in terms of, I guess, number of courses and users,

because it's in the millions, right?

Of users anyways?

Yeah, it's tens of millions of users, thousands of courses,

like probably over a million lines of code,

if you include all the Python code, if you include the different repos.

So yeah, it's at about that level.

And the other thing is that we are an open source platform as well. So one of the great things about edX is we have edX.org, which is our website, but then we provide our source code to the community. And the open edX community has, you know, thousands of instances of our source code where they are, you know, providing courses for regional content as well as even national platforms.

Wow. Yeah, we'll link to those in the show notes. Well, so one issue is since you started on 1.3 is migrating from Python 2 to 3. Is that something that has happened? Is it going to happen? How do you think about doing that at such a large site, either one of you?

Everyone's dealing with this right now, by the way.

Yeah, it's something that's still in progress.

We are hopefully weeks away from getting it.

So there have been a number of efforts that we've done to try to bridge that.

One is the, what is it called?

We have anchor tickets, which are basically a lot of,

Because so much of the Python 2 to 3 conversion is, you know, you can run the tooling for it and you just kind of have to sanity check that, you know, it's doing the right thing.

And like 90 some percent of the time it is.

But then, you know, so there was a very conscious effort

to be able to try to leverage our community better

and try to give them, like, really small incremental tasks

that don't require you to know the full stack in order to help.

And so that effort, Jeremy Bowman sort of really helped push that through.

And we did get a lot of contributions.

like and right now i think finial and company and some other folks are sort of driving the the last

bit like once once it's working enough so that the tests are are running on both environments like

it's been this uh weeks of just you know find the find the breaking thing watch 300 more tests

pass find the next breaking thing and then there's the and it's it's interesting because there's the

like there's the easy part relatively easy which is just well there's the strings right and then

there's the like code sniffs on previous stuff and then there's the real stuff yeah yeah there's

like rounding behavior is a little bit different and you know if we were to change how grade

rounding worked anywhere in the stack there there would be consequences and then even in terms of

backward compatibility if we you know chose to utf-8 and code the wrong way or whatnot it's

possible that our hash values are not the same as before and then we might have run into you know

backward compatibility issues once we actually launch this in production um yeah and also just

to give an idea like in our code base we do have a monolith with you know close to a million lines

of code depending on which you know which code you count um but then we also have a few like

satellite microservices that run you know around our monolith and so each

service needed to be upgraded and as Dave mentioned like there was a an

effort to try to do some of this initially with the community but you

know because we're running against the deadline we've actually taken in a lot

of this resourcing in-house to try to make this happen so and you know what

there are definitely some some things that we could do in terms of like regex

replacement, you know, in the strings. There's definitely, you know, just converting because

response.content, you know, now is returning, is being returned as bytes. You know, just,

there were a lot of code that just said assert in, in our test code and searching for, you know,

within response.content. So replacing that to be assert contains or assert not contains,

doing that in one bulk, you know, regex replacement in your ID and whatnot, that definitely helps,

you know, get us a long way, but there's a lot of these intricacies that we find as we are going

through. We're like, okay, what's the right way of making this change? And then are there any

principles that we can follow? For instance, doing the conversion from bytes or conversion from

strings to bytes, can that be closest to the perimeter, right? So, you know, throughout our

code base, as things are being passed, right, from one method to another and whatnot, you know,

pass them as strings as they were before, because that's, you know, that's what the business logic

is. But then only when you need to serialize it, let's say, put persisting in a CSV, or sending it

over the wire, or things like that, that's when you actually then do the conversion. Because one

of the concerns that I've always had with some of the Python three upgrade was, I felt there was so

much there's, I wanted to minimize the amount of code that has to do with strings, so that the

business logic continues to stream back at you, right?

You don't want, you don't want your code to be,

you know, you want your code to be as readable as possible.

You want the business logic to be the one

that's streaming back.

You don't want it to be, you don't want to get too caught up

on okay, what's happening in the details

with the string conversions.

So the more that we could try to keep that logic separated,

the better.

So that was, you know, that was one of the few principles

that I wanted to see, make sure that happened

doing this conversion. Well, it does seem that's the hierarchy of any conversion from two to three,

where you start off with strings, and those are pretty easy. And then you get to the asserts.

And that's relatively easy, but takes a lot of time. And yeah, and then it gets real with all

these other issues. Well, and on business logic, so this is actually, I wanted to highlight this.

We talked about this two weeks ago at Django Boston, because you had a really interesting

take on where to put the business logic. Because this is a question that Carlton and I get.

And for a lot of people, once you get to a decent-sized Django project, it's hard to

cram it into the models or the managers or the views.

Yes.

And David and I have, with other architects here at X, definitely have gone back and forth

on this.

And so there's, you know, to step back a little bit, like the reason why we're driving for

this, right, is we're trying to scale development of our platform.

And over time, it's definitely grown organically in terms of the code that's there and whatnot.

But now in order to allow our code base to be approachable by new developers as well as existing developers and for us to continue to maintain it, how can we do that in a way that, once again, it's easier to understand what the business logic is?

And so we've actually had a book club here with, you know, Dave was participating and others here as well for domain driven design and domain driven design, right, really talks about being very, very focused on what is the domain that you are building for and having those concepts and having the boundary, the bounded context and those boundaries being explicit.

So using that as a mindset, we were thinking, okay, well, given that we have a monolith, monoliths don't necessarily always have to be split into separate services.

Can you still have great abstractions in place while you're in the monolith?

And then if we choose to extract them as a secondary step for other reasons, that's definitely a thing that can still happen.

However, even within the monolith, what are those bounded contexts?

How do we make our code base such that it's understandable of what those interactions are?

Otherwise, there's a lot of tight coupling, and it's really hard to understand.

So, yeah, I can go.

So, Dave, you want to add anything to that?

Well, I guess one of the other things that I would add is that edX platform started as an application, right?

Like it started as an LMS and studio setup for a particular set of courses at edX.org.

And it was made into, you know, an open source project.

And it was intended to be that, but as a sort of, you know, the name is, I mean, the name is edX Platform, right?

The idea is people are supposed to be able to build extensions on top of it.

And so, which a lot of times you have a much stronger need for these, like, kind of internal APIs between the apps.

Because, you know, you might want to take a next platform, but you want to add your own thing that accesses enrollment data, for instance, or like because you have your particular special feature that you're kind of adding on to the side of it.

Yeah, so and having a sort of strong abstraction layer that you can like a sort of stable API interface within the application would be very, very helpful for that.

And we've tried to provide that in various places, but it's, you know, it's, it's a hard problem in a lot of senses. Like there, there are a lot of things that if you try to go into the sort of, if you try to move your logic outside of the, outside of the models, like you cut against the grain of Django in a bunch of, in a bunch of ways, right?

Like, you know, where are you going to do your validation if you want to have the same set of business logic that powers, like, you know, a REST framework like Django REST framework and your views there?

But you also want to have not, like, repeat yourself for, like, internal APIs.

Like, where does that logic go?

Like, your validation's in the serializer here, but, you know, are you going to call the serializer explicitly from your in-process APIs?

And, you know, you want to pass back query sets probably in case people want to modify them because, you know, it's often like, hey, get me these enrollments plus this data that I've keyed off of enrollments because I've added my own table here.

Like, how do you do that in a performant way that doesn't encourage everyone to do like N plus one queries?

So, you know, there's all these parts where you try to, yeah, it's been challenging for us to do that.

And I was going to say, and then to scale up to from a single application all the way up to like a million lines of code, there must have been teething points and difficult areas.

But then the goal, I guess, and I guess the question would be, do you think at a million lines of codes that the general structure is still comprehensible?

Right, no, so you would, this is why it's, at that scale of the code base, it's more

important, I believe, to be able to understand the historical traces, because unless if we

do put a lot of resources to make everything consistent, you will see a heterogeneous code

base, right, with some that haven't yet been updated to the latest direction and some that

are still a couple of years behind in mindset and so forth.

So we do have this concept of,

architectural decision records definitely you know this was also inspired by Michael Nygaard who

from you know had and ThoughtWorks had promoted this as well but architectural decision records

are very easy to very lightweight documentation it's immutable documentation records on a decision

that you've made and that goes into the docs itself it goes into the code base exactly in

a decisions a separate decisions directory and you know you have a context the decisions and

the consequences in each ADR and so for instance what the directory structure should be of our

monolith code base or you know that that could that could be an ADR and then this way people

know when exactly that was that was decided and then if they do see code that's not yet up to

date they can realize okay that was written earlier and if I'm now going to be working on

that part of the code base let me bring it up to the latest you know code standard so but yes so

in terms of maintenance of a large code base we need to you know develop processes such as this

to just sort of keep everyone up to date on because you're going to see broken windows everywhere so

which is which is the right window to follow and which ones are broken so i really like that idea

of the architectural decision record it's like leaving breadcrumbs so look here's the path yeah

and and one thing that we also found with going back to the where the business logic should be

kept, I mean, we were thinking about Django and initially when, so Dave had had, Dave was the

founding architect here. So Dave knows a lot about the initial war stories and how that happened. I

joined five, about five and a half years ago. So by then some of the architectural decisions were

already made and there was a lot of organic, organically made changes. So it was, it's actually

a great partnership I believe because I'm able to like have a little bit more fresher eyes and you

know Dave gives a lot more historical context but when I when I approached a code base I saw a lot

of divisions by technical concerns and this is a thing about Django's out of the box right where

you're thinking about views and models and urls.py and they are in my mind like they're very much

technical concerns of as an engineer like how to think about how to construct something but

if you want to figure out what are the business concerns what is it that this app is actually

trying to do you don't see that right away when you see these files everyone's you know just

models and urls and views or whatever so um so i think you know having a way to make that come

across much better is what we're looking towards and then one one great example of where this

applies is right now when we're now moving towards a different technical technology stack base when

we're thinking about moving our front end logic to be in react micro front ends right so now whatever

business logic happened to be within views.pi or even in a template or you know it shouldn't have

have been the template in the first place but anyway some of our code was so like some of those

things were in models.py and views.py and whatnot so like how do we if we had better abstractions

in place it would have been easier for us to do this replatforming effort and so I think from the

get-go if we could start our Django projects and apps in a way that it's clear where the business

logic is and let the web framework be a detail and this is what like i'm really talking you know

echoing uncle bob here but um you know the web is a detail whether it's exposing via rest api is a

detail whether it's being exposed via channels and um eventing models that's also a detail but

the business logic can stay more stable and more core to the app so that's how i've been thinking

about it. And I think Dave's bringing in the perspective also about the scalability and

performance impact, right? So like, if you do have these great, nice abstractions, Ninisha,

well, what are we going to do about those queries and, you know, the scalability of making sure that,

you know, when we don't, we're not always translating to APIs when, you know, like we

actually, if we do need to go directly to the models, we can in order to be performant.

I mean, I guess a couple of things I do want to add to that is just one, like Nimisha has been at edX for like five and a half, six years, the last couple of which she's been chief architect. So she's, she's got plenty of context. I do, I do have some of the like old stories.

But another thing is that, to be clear here, a lot of the early work on edX Platform, what became edX Platform, was not very intentional in a lot of cases.

You take a group of people, most of us had done Python work before.

Some of us hadn't.

You can actually tell parts of Studio that are written in kind of a Java accent.

But we had never done Django.

We had never done large Django.

And while Django docs are great for many things in starting out,

there was not as much guidance for,

Like, here's how you build this enormous application or the foundations of this enormous application or project, you know, that is going to grow to this size.

And even if there were, we probably wouldn't have had time to read it because we were in like frantic scramble mode.

Those entire buds are this hazy, sleep deprived thing that I just.

So so so certainly like things were put together very quickly.

And now we are trying to address some of those things, but as Nimisha said, it represents different generations of code and philosophy and expertise and understanding, frankly, of Django itself.

And so I can look at a piece of code completely out of context and tell you approximately what year it was written in, just because of the idioms that shift over time.

Right. Well, there's also been, I mean, Namesha, what you were saying about React and the logic from views to the front end. I mean, there's also been that shift in the last four or five years where, I mean, state in general and logic has moved to the front end. So I don't see how you could have architected for that five years ago because, I mean, was it 2013 when React came out or something like that?

I mean, that's also just the case of being flexible.

And I liked your idea of just having the Django piece, the architecture be as kind of simple as it is so that you can go back and forth as really as the web changes with how to do state and how to show things.

I mean, who knows if that's going to switch back.

Right. So for us, so what we started prototyping and assessing, and hopefully we'll solidify into a stronger best practice, but is to actually have a separate module. Right now we're calling it API.py, but essentially it's basically the domain logic.py, you know, where your business logic is.

And then the views.py file within a Django app would, you know, consume the functions or, you know, classes that are within those API.py files.

And that layer, let's just call it domain logic.py, that layer domain logic.py then is also an abstraction above your models.py.

And other apps that may exist in your Django project, they cannot go directly to your models.

They won't access your views or anything else directly, but they would go through this and interface right above your domain logic.py.

So that's how we're thinking about it.

And so this way, if we want to then have Django signals or Django channels integration or some other eventing, if we want to have a Kafka layer later.

And, you know, these are all ways of communicating out.

REST APIs are just one way of communicating out, right?

And so that's why views.py then becomes much thinner in that views.py would be more about it would it's it's it's responsibility would be more around the authentication layer.

Like other things that you might want to do, like response HTTP, converting things to proper HTTP response codes and formats and things like that.

so that's what that that it becomes it has this very separate separation of concern from where

the business logic is and models.py's separation you know concern would its responsibility would

be more about around the data um and so that's where i'm thinking we might we're going to try

some of our django apps and trying to implement it in that way and we're hoping that will then

allow us to evolve as the web evolves and as the you know all of our industry evolves so

Yes, but you're completely right that there are going to be things that we cannot anticipate.

So we'll always need to figure out how to refactor as we go.

But like currently, that's one way that we're thinking about it.

Carlton, how does that ring for you?

Because I was so struck by this conversation we had right before DjangoCon that at DjangoCon, basically the conversation I had with everyone was, where do you put the logic?

Where do you put the logic?

You know, and folks at really large sites and it's basically always somewhere else.

you know they call it something else but it's basically they yeah somewhat abstracted from

the traditional django hierarchy but carlton what what were you going to say right so so the sort of

um basic that you know to much simpler django example is let's say you um have your uh your

business logic your model validation in your save method right you've got just well fine okay so but

then what happens you've got a model form on top of that and then you your view validates the model

form it says yeah this is valid data and then so instead of returning 400 saying this is a bad

response this is a bad request you've got invalid data it says no this is okay let's go to save and

then you end up raising an error at the save point which turns into a 500 that's a server error

right so you don't want your validation logic in save i mean you might you might want to use it in

save but you also want it available to your form or to your serializer or wherever so the view can

use it so to to wear it earlier and say to the user hey what you've given me isn't correct can

you try again yeah and i think that's one thing that we also keep running into and honestly i'm

not completely happy with the sort of trade-offs even where we're landing it's just that it feels

like you're fighting the framework right in in a lot of ways like this is not like the like the

the primitive pieces that you're given don't plug in like don't connect to each other in quite the

way you want to to make this sort of abstraction work and you can do it but you know on either

you get kind of you can get kind of clunky code and like or like duplicated code and kind of

repeat that the validation call like somewhere else explicitly so yeah i i yeah i don't know but

one of the big issues that we find on uh passing models around for instance and sort of not having

that layer is that one, those things can change, right?

Like, you know, we can add stuff to it or whatever.

Two, it's like the model, passing around models is like this huge implicit interface, right?

That you're just kind of just throwing around everywhere because anyone can say, hey, I've

got this.

Now, let me like just grab that class, do a query and, you know, sort by this unindexed

field um you know on the field that on the table that has billions of items in it and like great

like you know and and even scarier it's like maybe that does that's fine on your machine

that's fine for your system because your system you've got you know two classes and 50 students

and hey like it works okay and then you know you bring it over and you try to merge it upstream

and it's like no like this is yeah yeah no if you could scale up you need to have more strict

rules and and sort of having an explicit layer where we say like these are the things you're

allowed to call and these are the things that we can make some kind of performance guarantees

around uh is is really it's really important for us and and also to clarify i mean there's

definitely a maturity model right like i don't i think what we are where we are arriving is

because of the scale that we are at exactly you know for a company that's starting up fresh and

new and with Django there's so many great things that Django comes out of the box and

actually its defaults are probably good enough for what you need at that time.

But I think depending then as you scale, perhaps by users, by developers or whatnot, then you'll

have different concerns and different requirements.

So for us, for instance, even this, what we talked about, even for edX at scale, we may

not apply this design pattern that we're talking about like domain logic dot pi whatever to the

entire code base right i mean this might be more for in domain driven design they have these terms

called core supporting and generic and core is the one that's more of your core value proposition it

has more of your domain concepts and so at that layer within your core you might want to have

these this way and this design pattern but the things that are perhaps more more volatile and

more in the periphery and you do want to change those more quickly and experiment you know you're

going to just use drf right off the back and use the serializers and models that apply directly i

mean fine you know that's quick you're trying to but then if after some time after a couple years

or or months you realize oh no this is a great core concept then you might want to figure out

how to stabilize it more so there's definitely a maturity model of the company of the code base

and then even of a feature but also the return on investment there's no point doing this kind

of super engineered high scale thing for a proof of concept.

You do a proof of concept, test the concept, does it work, then we'll

put the extra resources in and even for us like our monoliths the way that we're going about and

thinking about it is um are there parts of the monolith that are core to the business and core

to the platform and other things that are more extensions can we build them as plugins and one

thing that edX has developed and we'd love to contribute back to actually the Django community

once we extract it out of the monolith because right now it's still within it but there's a

something that we're calling Django app plugins. And it's built on Python's on Stevedore technology

that will allow one to basically, you know, import to their own extensions or plugins to a monolith.

The monolith provides some interface perhaps, but the plugin, its own view, its own urls.py and

even installing it all of those things can be automatically detected via stevedore so anyway

it's a it's a great technology we found it to be very valuable our our open source community

really appreciates it because they don't if they need to make a change or want to add something

they don't need to fork the entire monolith they can just create their own plugin and have it

automatically be detected by the monolith so um it very much goes with the solid design principles

with dependency inversion and and things like that so anyway that's that's where is also an

idea you know this concept of what's core and what's not core making that more explicit

um and the things that are not core core how can we incorporate them um with appropriate

bounded contexts and boundaries yeah and this goes back to what david was saying about

um giving people a set of apis they're allowed to call that you know are safe right well one thing

uh i wanted to go over is uh feature toggles right because i believe you you use django waffles now

because this is another uh again i was having this at django con with so many people when you're at

scale rolling out new features um you don't always want to just turn it on so i guess how um what did

you do before django waffle and how do you think about turning things on given the size of your

community um dave you want me to take this or uh yeah you you wrote that of i think so

i'm sorry the by the way of is is our sort of proposal process so that we have architecture

decision records that are specific to a given repo they're kind of local decisions by the team

on their particular like this this repo um but if you have something that affects that has

implications across all repos like for org wide engineering uh then we have the open mx proposal

process where we have sort of more general guidelines and uh namisha was the one who wrote

up uh the one on sort of feature toggles and such well it's like django has its depths i mean it's a

very similar uh and oeps were um so kale at x like he came up i think with that term but um

and this process where yes it is it was inspired by pep so we have you know so we have op1 op2

whatever and it's and so this is open edx proposals similar to python enhancement proposals

but yeah so there's one on feature toggles and definitely the reason we we decide we need to

have some good guidelines and best practices for it is because um we do when you're when you're

now we're talking about scaling um and even the deployment of features right and we don't want

We want to be able to allow teams to try features and whatnot in a way that it is more controllable.

We want to decouple deployment or enablement of a feature from deployment of the code.

So this way, teams have their own autonomy on when exactly they enable a feature toggle and how it is rolled out to the user base.

Perhaps they want to have it in beta testing and then eventually roll it out to a larger user base and so forth.

So anyway, we could we can share the OEP in the notes if you want of the podcast.

Yeah, it goes through a bunch of different like use cases.

So, you know, like I said, beta testing is one.

Some of the other ones might just be that we want to for operational reasons.

We want to just roll out gradually in case we are concerned there might be a performance issue or scalability issue or even functional issues and whatnot.

And so it allows a lot more control over that.

And the thing about this, though, is this, like feature toggles are great and it gives you that control, but then also increases the permutations in your code base.

Yeah, yeah, exactly.

You know, which set of permutations are your tests actually going to cover?

And so there's this also process of how do we then make sure that feature toggles that were created and all of those code branches are then deleted once they are no longer needed?

Yeah, exactly.

The cleanup is always the hard part.

Yes, yes.

And so making teams accountable and reminding them of that.

So this OAuth covers a little bit of that process as well and allowing us to have a tool and a reporting mechanism for understanding exactly when was the feature toggle created, when is the expiration time for it, what was the use case for it, and so forth.

So that allows us to then monitor those toggles.

And yes, as Will, as you said, we are using Django Waffles.

Django waffles are great. It allows us to, you know, specify whether they're on or off and then also which subset of users we want to turn it on for and things like that.

Yeah, it does seem to be the default that I can tell anecdotally that companies are using for this right now.

Yeah.

So another thing we've talked about I wanted to highlight.

So Django Celery usage.

And this is particularly relevant because at DjangoCon, there's a lot of talk around async starting to be rolled out.

And so both how you use Django Celery and I guess the huge question is if and when Django is fully async, do you see any use cases at edX or is it more of a side thing?

Because that's like the two-parter kind of for a lot of folks.

Like, do I actually need it when Celery and Qs work pretty well?

So as a lot of questions, take whatever you want to answer.

Yeah, I'll start with a few principles and design concepts that I have, and then Dave can talk a little more of the details. That sounds good.

Okay, so, I mean, for us, because we were thinking about running at an extent scale, right, we, and there's, you know, we want to keep the response time back to the users as, you know, within one to two seconds.

And if there are going to be some tasks or some requests that are going to take longer, either because we need to recompute your grades or we need to, you know, do some extra work in the background and whatnot,

it's very important for us to separate those into an asynchronous task and one

of the things that we found was that being intentional about perhaps even

separating our reads from our rights would be is very valuable and so when

someone is so basically don't do too many side effects and don't do too many

costly operations within a request especially if it's a user facing request

and the user wants to respond quickly.

And so you need to be able to put some tasks

into an asynchronous operation.

And so Django Celery was definitely one

that is a technology that we've used.

And initially we were doing it on RabbitMQ,

then we converted to Redis for scalability reasons

and other reasons.

But that definitely has been a way for us

to scale out our infrastructure.

I would love, though, to move towards another model

Because one of the downsides with Celery is that as the task that – basically, it's not a fully PubSub model.

So you do need to – if you want to be able to have the Celery task run on a separate service, you do need to know what – you know, like the – it's not very easy to have an API that is more like a – the subscriber needs to know what the publisher's API and vice versa.

right you know i mean like it's you're you're too tightly coupled well instead in celery you import

your entire django project right so you have to import the monolith into the celery instance to

run the queue and it's like it would be nice if if tasks could be sort of separate if they didn't

have to know exactly exactly and so um so anyway so that that's where i would like to lead to

eventually and i was thinking about more of an eventing architecture but but maybe there's

something else and i need to learn more about async and what it provides and there might be

some things out of the box now once we upgrade to django 2 in a couple of months but yeah well

django 3 is when it starts to get real and even then it's a 3.0 has ascii um server and then the

plan i believe carlton is 3.1 we'll start with views um and then orm thereafter more stuff

thereafter we tend to hug the long-term long-term support releases oh don't get carlton started on

this yeah yeah well well because um you know because most people do most people do yeah but

because also we need something to hopefully say well like when we put out a version of uh a release

of open edX for people to use because we do have sort of um like we we named them after trees but

But so, you know, we put out Ironwood and universities that use it are not going to upgrade until like maybe the next, you know, the following summer or something when school's not in session.

And so, yeah, we're not going to see async on the big repos.

So the smaller repos tend to go up faster because there's just less inertia to move.

And also fewer groups run the e-commerce service that we have as opposed to the, you know, everyone runs that X platform because it's where the LMS and the authoring environment is.

right um so yeah on in terms of the async though like there are features i would love to

to play with yeah like what um i mean channels is is is great i played around with version one

of channels before it became that i can't play around version two on the next platform because

you know well carlton's in charge of maintaining channels now okay that's no it's it's really cool

stuff uh and frankly there's a lot of stuff around like web sagas and such that we love to use

you know having async support for like service service calls that are not blocking can make you

know better use of better use of resources is great although I think edX platform is a bit

unusual for Django project in that a lot of our performance issues or performance challenges are

we're actually CPU and the CPU and cache related really yeah I know that's so there

so okay what are you processing like what's because okay because as we've been talking

i've imagined that you've got databases scaling issues you've got a lot of content you've got

videos you've got this course content you've got other loading sites uh loading pages but

where the calculation issues that's kind of interesting okay so this is um okay so so this

This is, yay, history stuff.

So when edX started, right, we were, like, again, like, imagine, so it started with a course, a single course that was being offered to both MIT students and to, and as a MOOC version that ran sort of two weeks behind the MIT cohort.

And, you know, like, okay, when Coursera, if, like, those early days, if, like, Coursera or Udacity, they can say, oh, you know, we're going to push back the start of this course by two weeks because, you know, whatever to help iron out.

And they're, you know, non-paying students are like, okay, that's cool, whatever.

You don't tell MIT you have to delay the semester for two weeks, right?

Like, that's just not going to happen.

So a lot of things like we those things were like really really like to

Sort of very quickly put together and one of the things that was put together was that the course format in those early days course

Teams is is XML. It's like a giant XML file. In fact, it was a giant XML file. That was a Mako template

so

And so the entire like definition of what the course is all the sequences all the problems were this big file that that Django read

on startup. And it's really easy to create a set of objects like this very quickly.

And that's fine for the prototype. But obviously, the prototype days are over and we are in the

world now where everything's in databases like you would expect. But a lot of the access patterns,

if you look in those early days, when you're trying to prototype something, you're basically

You're going for maximum power with minimum code.

And you can create, it is easier to create really powerful interfaces than it is to create performant data models, right?

As an example, if you look at the grading code, the original grading code for Hexplatform was basically, okay, here's my tree of content, everything in the course.

I'm going to crawl through and go through the whole tree

by checking my children.

And for each node that can be graded

as a gradable thing, like a problem,

I'm going to ask that pluggable interface,

hey, how many points are you worth?

And how many points did the student earn

based on their current state of your problem?

Because we have this notion of X modules back then,

and videos were X modules,

problems were X modules, everything.

So you had some common set of interfaces you could ask.

And that gave you a great deal of flexibility in terms of power.

But at the same time, how long does it take to show the progress page

when you have to ask this question of every node in the course?

And the answer is, you have no idea.

Because you've created this interface where it's like max score and get score, whatever.

And you've pushed off having to think about the data model.

But in return for that, you've lost all ability to reason about the performance of the system, right?

So we had X modules that started sandbox processes, like in Python, and did RPC calls to sort of get information because you don't want untrusted code to be running.

We had ones that would parse their internal XML, and depending on how many response types you had inside would change their score accordingly.

we had ones that would make HTTP calls to another system entirely to get back what the latest score for that person was

because the first version of pure grading happened on a different service.

And so if you wanted to make an X module that would return you a different max possible grade for username starting with C on Tuesday afternoons,

Like, there's nothing in the contract that explicitly forbids that.

I'm seeing a theme here with the architecture.

Yeah, exactly, right?

But once you sort of have that out there, like, you know, that's fine for the prototype and for an exercise.

That's not cool when you have millions of users and, you know, you're trying to run a course of scale.

And so you find yourself trying to claw that back in a bunch of different ways.

And so eventually what happens is you sort of have this sort of shift that goes from your courseware being these like smart objects that you can just make requests for and leave it to them to figure out how to implement it.

And you sort of shift that relationship to be, okay, I'm going to have this, I'm going to have a grading system.

The grading system has a data model.

And I know that the grading system, I can query it.

I can ask for things like what is the student's grade in the course and get a quick reply.

And then I'm going to make those, the X modules and X blocks, the course where like individual leaf nodes, they're going to push data into that data model so that I can query that efficiently.

Right. So, but you have this kind of like shift and that applies to a bunch of things.

We have to do that for grading. We're doing that for more scheduling related things. But it's hard to do that in a sort of backwards compatible way because we have a lot of course teams explaining a lot of effort into a lot of course content.

And so I think one of the things that Namesha was sort of pointing out was one of the ways we do this is to kind of shift it, like use async methods like CeleryTask, not async as in the Python async, but use CeleryTask in order to kind of shift the burden of processing.

So, for instance, right now what happens is that there is a grading system and it will hold your scores whenever they change.

But there's a set of very complicated permissions calculations that came about because, hey, we have this whole thing loaded in memory already, right?

Right, guys?

Right, yeah, yeah.

And so now what we do is when you change a score, all those crazy computations still happen, but they happen in a Celery task that runs, and then it puts your calculated score into this grading system that has a data model that you can actually make guarantees about.

And so that's been sort of our bridge in a lot of ways to take things from this smart object world to the discrete services world.

But yeah, that's the reason why we're CPU bound in so many things.

Can I just ask, if I was writing a new module, a new plugin, I'd just go straight to the new data store?

I wouldn't go via the old mechanism?

Yeah, I mean, probably.

I don't know.

Yeah, definitely.

I mean, there is a grade system now.

And if you want to ask questions about a student's grade, yeah, you would go to the new thing.

You won't want to mess up in that one.

Nimesha, you were going to say.

Yeah, I was just going to say that one of the design principles that, you know, when we're thinking about when we implemented the grading system and, you know, we're talking about persistent grades before grades were computed on the fly and now we're able to read them from the database.

But in order to make that happen, because we had so much flexibility with those X modules and X blocks and what they could do and giving us data in the runtime and all that stuff, we were inspired by, it's like the reverse ETL, you know, with ETL, you know, basically you're, with ETL, you're reading the data, then you're transforming it, and then you're writing into new form.

It's reversed in the sense that basically we allowed these plugins and these Xbox, right, to basically give us a lot of that content.

And they were able to then write and push that into a form that then they were able to then transform however they want to for read optimizations.

So basically, that's where that separation of reads versus writes come in, where like we want to have very fast read optimization views of this data and do it in a.

So we implemented this very simple interface where basically allowing anyone to collect the data and then allowing anyone to transform the data.

And then we automatically look through all the transformers that are registered in our system, run them through sort of like an ETL type of thing there, except it's LTE and that, you know, it's allowing people to write.

And then when it comes to responding to the user, we're able to do that very quickly because it's already transformed for read optimization.

And then the response times are now once again improved.

So it's tricks like this that have allowed us to figure out how can we take what we had legacy-wise.

They were a lot more generic interfaces.

We're going to be more intentional going forward with what our APIs and interfaces are, but we still needed to support the old.

so how do we do that in a way that's performant but you know now with optimizations in mind and

actually there's one other thing i want to in defense of the people who who who made the that

system originally like it is it is really powerful right like like there there are um if you are in

a situation where you don't know what the interface really is for like what what you should and

shouldn't allow for grading and also where there are bugs all over the place right we're building

the platform as the first course is running

on it, and, oh, this

bug, like, you know, you detect

this bug is

misscoring someone or, like, you know,

is interpreting the score wrong, then

if you store it in a persistent

state, then, okay, you have

tasks to, like, regrade them and

restore and modify their stores and

whatever. If it's

always dynamically computed on the fly, then

like, yeah, just reload the progress page.

Boom. Your

score is fixed.

And so it was one of those things where, like, you know, and even the sort of query anything, like, one of the sort of painful things about moving from, like, query the smart object to, like, separate systems is that you do lose some power, right?

Like, the very original version of the prototype that we had had A-B tests running in a totally hacky, crazy way, but they were running.

And it took us years to get that functionality back

in a sort of more predictable and performant way

just because like you, yeah, it's...

So there were trade-offs

and I definitely like,

I have suffered from the grading system more than most,

so has Namesha.

And so definitely I'm not going to say

that that's the way you should architect it,

but I guess I feel like it's important to note

that there are trade-offs,

especially for something that as young and quick

as we were doing back then.

When you're prototyping to just write a Python class and keep everything in memory is, you know, pickle it if you have to.

That's a great development environment.

And that's a great way of finding out what your requirements are when people can't specify them for you.

You can say, oh, well, you know, what does your grading system involve?

And it's like, well, A to F?

No, it's loads more than that.

What does it actually involve?

Well, we don't know.

Well, let's build something, let you use it.

And then when we've exposed what the actual requirements are, then we can build something

which scales up.

As we wrap up, one thing, if we can, I'd love to talk about the automated discovery of CSRF

issues, because this is something, Namesha, you and I briefly talked about Django Boston.

You'd mentioned just generally, there was a number of security things that edX had come

up with that to help Django, but weren't part of Django core.

Yes, yes.

So, I mean, security is definitely very important to us.

I think Django's definitely improved, at least in the last five years that I've seen, like, over time in terms of having more secure defaults.

I think for us, we do have a security working group within edX that triages issues that come in.

And so over time, there have been some issues that are more prevalent than others.

The automated discovery of CSRF issues, that was something that I had written a while back.

And sorry, I'm trying to just context switch what that was in exactly.

But I think what that was, was, you know, basically side effecting get requests, right?

Whenever someone makes a get request, we want to make sure that there are no modifications to a model or, you know, to any data, right, in your system.

On a get request?

Yeah, so in CourseWare, this actually, like, was the case for various student tracking things.

Okay.

Yes, and CourseWare, yes, might have a legitimate issue when it did that way.

But we found, so basically I had created this middleware.

It was a Django middleware, I believe.

And whenever there was a, it detected that there was a get request being made.

And then what it did was, actually, I think it used Django signals to track any post-save model changes.

And so it was able to detect if any Django model changed whenever a GET request was made, as opposed to POST request, right?

And then basically we reported this, and then you would be able to realize that this is a potential CSRF issue that needs to be fixed.

And I think there was other things, too, that I detected.

I may have detected if there were Django Celery tasks that was also being initiated whenever a

GET request was made. So anything that could have potentially had a side effect that could have

resulted. So that's something that we had created. I think what happened was we found a few CRSRF

issues within our system and so we had then been fixing them over time. And so at some point we

we wanted to make it a public application others could also use, but we haven't gotten around to

that yet. So we'd love to be able to contribute that as well. There's also like Robert's CSRF,

wasn't there like template level CSRF stuff that runs in our Jenkins builds now? Yeah,

those are XSS. Oh, I'm sorry. Robert has implemented an XSS linter as well. So he's

able to detect if there's any XSS issues in Django

templates and maybe other things as well. So Carlton, would Django be interested

in these things? Yeah, I mean, quite probably, yeah. I mean, if they're extracted nice and small

and easy to implement, yeah. I mean, just going back to the CSRF example, that's a

great use of signals, right? So people use signals as a way of decoupling their application

but they can get overused and they can lead to hard to understand and hard to follow

code. So the sort of maxim I always use is, do you know

at runtime who the receiver of this signal is going to be who's going to act on this signal

if you do don't use a signal but of course when you're monitoring is there are there any saves

across any models in this request you don't know who's going to be sending it so you don't

it's perfect case for signals it's it's a lovely example right no signals can be very powerful and

there are actually good use cases for it so this is definitely one another one i will say is that

you know when we're thinking about the monolith and the core and then we have these django app

plugins because there too we don't know who the recipient is and it's not core to our platform

we'd actually prefer to use django signals as a way of communicating that out but then to our

earlier point you know that has then becomes an intentional interface yeah and you have to

document that and provide guarantees and ported and all that stuff but django signals does allow

asynchronous it allows decoupling um you know of your components and so but yeah i agree with you

that's that's a good rule of thumb if you don't know the recipient that it definitely makes sense

i mean it's this is a possible interface that you could yeah yeah i mean i'm sure we could keep

talking i know we're kind of run out of time um thank you so much for sharing the realities of a

large scale uh code base because this is how it is for everyone there's always the trade-off between

prototype and and legacy and um you know i guess the thing when when beginners ask me sometimes

about these questions i say it's an honor to be where where you all are at you know as frustrating

as it is, it's like these are the problems you want to have, that you're at scale and you're

keeping up to date with Django. So it's fascinating to hear kind of under the hood

how it's all working and how you're thinking about it. And we'll link in the show notes to

a number of these, especially the OEP. That would be great. Yeah, really super interesting.

So again, thank you, David. Thank you, Namesha. Yes, thank you both.

And for those listening, we're at DjangoChat.com, ChatDjango on Twitter,

and we'll see you all next time. Bye-bye. Join us next time. Bye-bye.