Transcript: Migrating from Python 2 to 3 at EdX - David Ormsbee & Nimisha Asthagiri

I'm Will Vincent, joined by Carlton Gibson.

Hi, Carlton.

Hello, Will.

Uh welcome both of you.

Thank you.

Happy to be here.

Uh so we connected because uh we met at the Django Boston meetup, and edX is both one of the largest educational sites in the world, and I believe has used Django from the beginning.

So I wonder we could talk about do you know do either of you know why Django originally, and then how has that been?

Because you've been through a lot of these uh changes of Django, Python 2 to 3, and so hopefully we can dive into all of that.

Uh, because this uh edX originally started as uh uh MITX, which you know it came out of MIT.

And uh so Python was kind of the default choice, right?

Like it's uh the intro CS course teaches Python, all the uh you know Europe interns uh that are gonna work on it, know Python.

Um and the fact that the first course was uh an electrical engineering course means that you know having NumPy, SciPy, like that sort of set of tools was was very convenient.

Um and so like once you and so by the way, these early decisions were made by Piotr Mitros, uh, who was uh who created the original prototype for what became edX platform uh and was the chief scientist for edX for uh I don't know, five or six years for a while.

Because I believe this goes back quite a ways, right?

Before Python was standard at the undergrad level for a lot of places.

So sometime around Django 1.3, I guess.

Um and so, yeah, and and so uh Peter also made the decision for Django, but I mean once you've gotten it down to Python, then you know, let's um Django was like one of the uh one of the obvious choices uh for the

Good call.

So um sure.

It was founded by Harvard and MIT, as Dave mentioned in 2011.

Uh, we do specialize in higher education courses um with courses and content from you know one of the best worldwide partners from universities.

Because it's in terms of I guess number of courses and and users, because it's in the millions, right?

Of users, anyways.

Um so yeah, it's it's at about that level.

So one of the great things about edX is um we have edX.org, which is our website, but then we provide our source code um to the community, and the Open EdX community has you know thousands of instances of our source code where they are you know providing courses for regional um content as well as even national platforms.

Um, so one issue is since you started on on one.

Is migrating from Python 2 to 3.

Is that something that um has happened, is going to happen?

How do you think about doing that at such a large site, either one of you?

Uh everyone's dealing with this right now, by the way.

So I'm not it's

Uh we are hopefully weeks away from from getting it.

It's so um there have been a number of efforts that we've uh we've done to try to to bridge that up.

One is the um what is it called?

Uh we we have anchor tickets, which are basically um a lot of because so much of the Python 2 to 3 conversion is you know, you can you can run the tooling for it and you just kind of have to sanity check that you know it's doing the right thing, and like 90 some percent of the time it is, but then you know that so um so that was a very conscious effort to be able to um try to leverage our community better and try to give them like really small uh small incre

mental tasks that they that don't require you to know the full stack in order to to help.

Uh and so that that effort, um Jeremy Bowman uh sort of really helps push that through.

Um and we did get a lot of contributions, uh, but like and right now I think Fenil and company and some other folks are are sort of driving the the last bit.

Like once once it's working enough so that the tests are are are running on both environments, like it's been this uh weeks of just you know find the find the breaking thing, watch 300 more tests pass, find the next breaking thing, and then there's the and it's it's interesting because there's the like there's the easy part, well relatively easy, which is just

And then there's the like code sniffs on previous stuff, and then there's the real stuff.

Yeah, yeah.

Um, yeah, and also just to give an idea, like in our of our code base, we do have a a monolith with you know close to a million lines of code, depending on which you know which code you count.

Um, but then we also have a few like satellite uh microservices that run you know around our monolith, and um, so each service needed to be upgraded.

And um as as Dave mentioned, like there was a uh an effort to try to do some of this initially with the community, but you know, because we're running against the deadline, we've actually taken in a lot of this resourcing in-house to try to uh to make this happen.

So, and you know, there are definitely some some things that we could do in terms of like regex replacement, you know, in the strings.

There's definitely you know just converting because response.content, you know, now is returning is being returned as bytes, um, you know, just there were a lot of code that just said assert in in our test code and searching for you know within response.content.

So replacing that to be assert contains or assert not contains, doing that in one bulk, you know, regex replacement in your ID and whatnot, that definitely helps you know get us a long way.

But there's a lot of these um intricacies that we we find as we are going through, where like, okay, what's the right way of making this change?

And then are there any principles that we can follow?

For instance, doing the conversion from bytes or conversion from strings to bytes, can that be closest to the perimeter, right?

So, you know, throughout our code base as things are being passed, right, from one method to another and whatnot, you know, pass them as strings as they were before, because that's you know, that's what the business logic is.

Um, but then only when you need to serialize it, let's say put persisting in a CSV or sending it over the wire or things like that, that's when you actually then do the conversion.

Because one of the concerns that I've always had with some of the Python 3 upgrade was I felt there was so much uh there's I I wanted to minimize the amount of code that has to do with strings so that the business logic continues to stream back at you, right?

You don't want, you don't want our your code to be um, you know, you want your code to be as readable as possible.

You want the business logic to be the one that's screaming back.

You don't want it to be, you don't want to get too caught up on, okay, what's happening in the details with the string conversions.

So the more that we could try to keep that logic separated, the better.

Um so so that was, you know, that was one of the few principles that I wanted to see see make sure that happened during this conversion.

Well, and and on um business logic, so this is actually I wanted to highlight this.

We talked about this was last uh two weeks ago at Django Boston, because you had a really interesting take on how to where to put the business logic, because this is a question that Carlton and I get, and for a lot of people, once you get to a decent sized Django project, um it's hard to cram it into the models or the managers or the views.

And we so there's you know, to step back a little bit, like the reason why we're driving for this, right, is we're trying to scale development of our platform, and over time, it's definitely grown organically in terms of the code that's there and whatnot.

Um, but now, in order to allow our code base to be approachable by new developers as well as existing developers, and for us to continue to maintain it, um, how can we do that in a way that once again um it's easier to understand what the business logic is?

And so we actually had a book club here with uh, you know, Dave was participating in and others here as well for domain-driven design.

And domain-driven design, right, really talks about uh being very focused on what is the domain that you are building for, and um having those concepts and having the boundary, the bounded context and those boundaries being explicit.

So being so using that as a as a mindset, we were thinking, okay, well, given that we have a monolith, monoliths don't necessarily always have to be split into separate services.

Can you still have great abstractions in place while you're in the monolith?

And then if we choose to extract them as a secondary step for other reasons, that's that's definitely a thing that can still happen.

However, even within the monolith, what are those bounded contexts?

How do we how do we make our code base such that it's understandable of what those interactions are?

Otherwise, there's a lot of tight coupling and it's really hard to understand.

So um, so yeah, we could I can go.

So, Dave, you want to add anything to that?

Like like it started as uh an LMS and studio uh setup for a particular set of courses on edX.org.

Um, and it was made into you know an open source project.

And it was always intended to be that, but as a sort of, you know, the the name is top, I mean the name is edX platform, right?

The idea is people are supposed to be able to build extensions on top of it.

And so which a lot of times you you have uh a much stronger need for these like kind of uh internal APIs between the apps because you know you might want to take edX platform, but I you want to add your own thing that accesses enrollment data, for instance, or like because you have your your particular special feature that you're kind of adding onto the side of it.

Yeah, so so and and having a sort of strong abstraction layer that you can like uh a sort of stable API interface within the application uh was would be very, very helpful for that.

And we've tried to provide that in various places, but it's you know it's it's a hard problem in a lot of senses.

Like there are a lot of things that if you try to go into the sort of um if you try to move your logic outside of the outside of the models, like that you you cut against the grain of Django in a bunch of in a bunch of uh ways, right?

Like, you know, the where are you gonna do your validation if you want to have the same set of business logic that powers like you know a rest framework like Django Rest framework and and uh and your views there, but you also want to have not like repeat yourself or like internal APIs, like where does that logic go?

Like your validation's in the serializer here, but you know, are you gonna call the serializer explicitly from your in-process APIs?

And you know, do you want to pass back query sets?

Probably in case people want to modify them, because because you know it's often like, hey, get me all get me the you know, these enrollments plus this data that I've keyed off of enrollments because I've added my own table here.

Like, how do you do that in a performant way that doesn't encourage everyone to do like n plus one queries?

There's there's these all these parts where you try to yeah, it's it's it's been it's been challenging for us to do that.

Go on, I was gonna say, and then to scale up to uh from a single application all the way up to like a million lines of code, there must have been teething points and difficult areas.

No, so you you would this is why it's with with at that scale of the code base, it's it's more important, I believe, to be able to understand the historical traces.

Because um, unless if we do put a lot of resources to make everything consistent, you will see you will see a heterogeneous heterogeneous code base, right?

With some that haven't yet been updated to the latest direction and some that are still a couple of years behind in in the mindset and so forth.

Um, so you know, we do have this concept of architectural decision records.

Uh, definitely, you know, this was also inspired by Michael Nygaard, who from you know had and ThoughtWorks had promoted this as well.

But architecture decision records are very easy to very lightweight documentation, it's immutable documentation records on a decision that you've made.

Um, and you know, you have a context, the decisions, and the consequences in each ADR.

And um, so so for instance, what the directory structure should be of our monolith code base, or you know, that that could that could be an ADR.

And then this way people know when exactly that was that was decided, and then if they do see code that's not yet up to date, they can realize, okay, that was written earlier, and if I'm now going to be working on that part of the code base, let me bring it up to the latest, you know, code standard.

So um, but yeah, so in terms of maintain maintenance of a large code base, we need to you know develop processes such as this to just sort of keep everyone up to date on because you're gonna see broken windows everywhere.

So which is a which is the right window to follow and which ones are broken.

It's like leaving breadcrumbs.

And and one thing that we also found with going back to the where the business logic should be kept, I mean, we're thinking about Django and initially when so Dave had had Dave was a founding architect here, so Dave knows a lot about the initial war stories and how that happened.

I joined um five about five and a half years ago, so by then some of the architectural decisions were already made, and there was a lot of organic, uh, organically made changes.

So it was it's actually a great partnership, I believe, because um I'm able to like have a little bit more fresher eyes, and you know, Dave gives a lot brings a lot more historical context.

Um, but when I when I approached a code base, I saw a lot of divisions by technical concerns.

And this is a thing about Django's out of the box, right?

Where you're thinking about views and models and URLs.py, and and they are in my mind, like they're very much technical concerns of as an engineer, like how to think about how to construct something.

But if you want to figure out what are the business concerns, what is it that this app is actually trying to do, you don't see that right away when you see these files.

Everyone's you know, it's just models and URLs and views and whatever.

So um, so I think you know, having a way to make that come across much better is what we're looking towards.

And then one great example of where this applies is right now when we're now moving towards a different technical technology stack phase where we're thinking about moving our front-end logic to be in React micro front ends, right?

So now whatever business logic happened to be within views.py or even in a template, or you know, it shouldn't have been the template in the first place, but anyway, some of our code was.

So, like some of those things were in models.py and views.py and whatnot.

So, like, how do we, if we had better abstractions in place, it would have been easier for us to do this replatforming effort.

And so I think from the get-go, if we could start our Django projects and apps in a way that it's clear where the business logic is, and let the web framework be a detail.

And this is what, like, I'm I'm really talking, you know, echoing Uncle Bob here, but um, you know, the web is a detail, whether it's exposing via REST API is a detail, whether it's being exposed via channels and um eventing models, that's also a detail.

But the business logic can stay more stable and more core to the app.

So that's how I've been thinking about it.

And I think Dave's bringing in the perspective also about the scalability and performance impact, right?

So, like if you do have these great, nice abstractions, Nimisha, well, what are we gonna do about those queries and you know the scalability of making sure that um, you know, when we we we we don't we're not always translating to APIs when you know, like we actually, if we do need to go directly to the models, we can um in order to be performant.

So she's she's got plenty of context.

Um I I do I do have some of the like old stories.

Um but another thing is that um to to be clear here, a lot of the early work on edX platform, what became edX platform, was not uh very intentional in a lot of cases, right?

Like you take a group of people, we've we most of us had done Python work before.

Um, at least some of us had it.

You can actually tell like parts of Studio that are written in kind of a Java accent.

Uh but the but you know, we had never done Django.

Like we'd never done large Django.

And you know, while Django docs are are great for many things in starting out, like the the there was not as much guidance for like here's how you build this enormous application or the foundations of this enormous application or or project, you know, that um that's gonna grow to this size.

And even if there were, we probably wouldn't have had time to read it because we were in like frantic scramble mode.

Those entire muds are this hazy, sleep-deprived thing that I just so so so certainly like things were were put together uh very quickly, and I think um it's and and now you know we we are trying to address some of those things, but you can but as Numisha said, like it's it represents different generations of of code and philosophy and and expertise and understanding, frankly, of the platform of Django itself.

And so like you can I can look at a piece of code completely out of context and tell you approximately what year it was written in because just because like uh of the idioms that like shift over time.

Well, there's also been, I mean, Nemisha, what you were saying about React and the logic from from views to the the front end.

I mean, there there's also been that shift in the last four or five years where I mean state in general and logic has moved to the front end.

So I I don't see how you could have architected for that five years ago because I mean I don't was it 2013 when React came out, something like that.

I mean that's that's also just the case of being flexible.

And I liked your idea of just having the the Django piece, the architecture be as kind of simple as it is, so that you can go back and forth as really as the web changes with how to how to do state and how to show things.

I mean, who knows if that's gonna switch back?

Right.

Um right now we're calling it api.py, but um, but but essentially it's it's basically the domain logic.py, uh, you know, where your business logic is.

And then the views.py file within a Django app would you know consume the functions or you know, classes that are within those API.py files.

Um and and that that layer, that domain, let's just call it domainlogic.py.

That layer domainlogic.py then is also an abstraction above your models.py.

Um and other apps that may exist in your Django project, they cannot go directly to your models, they can't, they won't access your views or anything else directly, but they would go through this an interface right above your domain logic.py.

So that's how we're thinking about it.

And so this way, if we want to then have um Django signals or Django channels integration or some other eventing, if we want to have a Kafka layer later, and you know, these are all ways of communicating out.

REST APIs are just one way of communicating out, right?

And so that's why views.py then becomes a much thinner in that views.py would be more about it would it's it's its responsibility would be more around the authentication um layer, like other things that you might want to do, like response HTTP, converting things to proper HTTP response codes and formats and things like that.

So that's what that that it becomes, it has a very separate separation of concern from where the business logic is.

And models.py's separation, you know, concern would its responsibility would be more about around the data.

Um, and so that's where I'm thinking we might we're gonna try some of our Django apps and trying to implement it in that way, and we're hoping that will then allow us to evolve as the web evolves and as the you know all of our industry evolves.

So yes, but you're completely right that there's don't look there are gonna be things that we cannot anticipate, so we'll always need to figure out how to refactor as we go.

Um but like currently that's that's one way that we're thinking about it.

Because I was I was so struck by this conversation we had right before DjangoCon that at Django Con, basically the conversation I had with everyone was where do you put the logic?

Where do you put the logic?

You know, and folks at really large sites, and it's basically always somewhere else.

You know, they call it something else, but it's basically they sort of somewhat abstract it from the traditional Django hierarchy.

But Carlton, what what were you gonna say?

Right.

You've got just well, fine, okay.

So but then what happens you've got a model form on top of that, and then you your view validates the model form, it says, yeah, this is valid data.

And then so it does instead of returning a 400 saying, Nee, this is a bad response, this is a bad request, you've got invalid data, it says, no, this is okay, let's go to save, and then you end up raising an error at the save point, which turns into a 500.

That's a server error, right?

So you don't want your validation logic in save.

I mean, you might you might want to use it in save, but you also want it available to your form or to your serializer or wherever, so the view can use it so to to error earlier and say to the user, hey, what you've given me isn't correct.

Can you try again?

And honestly, I I'm not completely happy with uh the sort of trade-offs, even where where we're landing, it's just that it feels like you're fighting the framework, right?

In in a lot of ways.

Like this is not like the like the the the primitive pieces that you're given don't plug in, like don't connect to each other in quite the way you want to to make this sort of abstraction work.

Um, and you can do it, but you know, on the either you get kind of you can get kind of clunky code and like or like duplicated code and kind of repeat that the validation call like somewhere else explicitly.

So yeah, I I yeah, I don't know.

That I w one of the big issues that we find on uh how passing models around, for instance, and sort of not having that layer is that like one, uh the those things can change, right?

Like you know, we can add stuff to it or whatever.

Two, it's like the model passing around models is like this huge implicit interface, right?

That you're just kind of just throwing around everywhere.

Because anyone can say, hey, I've got this now.

Let me like just grab that class, do an do a query, and and and you know, sort by this unindexed field on you know, on the field that on a table that has billions of items in it.

And like, great, like you know, and and even scarier, it's like maybe that does that's fine on your machine.

Uh that's fine for your system because your system, you've got you know two classes and 50 students, and hey, like it works okay.

And then you know, you bring it over and you try to merge it upstream, and it's like no, like this is this is

No, if you could scale up, you need to have more strict rules.

Like I don't I think what we are where we are arriving is because of the scale that we are at.

Exactly.

You know, for a company that's starting up fresh and new, and with Django, there's so many great uh things that Django comes out of the box, and actually it's its defaults are are probably good enough for for what you need at that time.

Um but I think depending then as you scale, uh perhaps by users, by developers, or whatnot, then you'll have different concerns and different requirements.

So for us, for instance, like even this, what we talked about, even for edX at scale, um, we may not apply this design pattern that we're talking about, like domain logic.py, whatever, to the entire code base, right?

I mean, this might be more for in domain-driven design, they have these terms called core supporting and generic.

And core is the one that's more of your core value proposition, it has more of your domain concepts, and so at that layer within your core, you might want to have these this way and this design pattern.

But the things that are perhaps more um more volatile and more in the periphery, and you do want to change those more quickly and experiment, you know, you're gonna just use DRF right off the back and use the serializers on models.py directly.

I mean, fine, you know, that's quick.

You're trying to, but then if after some time, after a couple of years or or months, you realize, oh no, this is a great core concept, then you might want to figure out how to stabilize it more.

So there's definitely a maturity model of the company, of the code base, and then even of a feature.

There's no point doing this kind of super engineered um high-scale thing for a proof of concept.

You do it proof of concept, test the concept, does it work?

Then we'll put the extra resources in.

And one thing that edX has developed, and we'd love to contribute back to actually the Django community, once we extract it out of the monolith, because right now it's still within it, but there's something that we're calling Django app plugins, and it's built on Python's um Stevedor technology that will allow one to basically you know import their own extensions or plugins to a monolith.

The monolith provides some interface, perhaps, but the plugin its own view, its own URLs.py and even installing it, all of those things can be automatically detected via Stevedor.

So anyway, it's a great technology.

We found it to be very valuable.

Our open source community really appreciates it because they don't, if they need to make a change or want to add something, they don't need to fork the entire monolith.

They can just create their own plug-in and have it automatically be detected by the monolith.

So it very much goes with the solid design principles with dependency inversion and things like that.

So anyway, that's that's where is also an idea, you know, this concept of what's core and what's not core, making that more explicit, um, and the things that are not core core, how can we incorporate them um with appropriate bounded contexts and boundaries?

Because I believe you you use Django Waffles now, because this is another uh again, I was having this at DjangoCon with so many people.

When you're at scale rolling out new features, um you don't always want to just turn it on.

So I guess how um what did you do before Django Waffle and how do you think about turning things on given the the size of your community?

So I'm sorry, the by the way, O OF is is our sort of proposal process.

So that we have architecture decision records that are specific to a given repo.

They're kind of local decisions by the team on their particular, like this this repo.

Um but if you have something that affects the that has implications across all repos, like for org-wide engineering, uh then we have the uh Open Demix proposal process where we have sort of more general guidelines.

And uh Nemisha was the one who wrote up the um the one on sort of feature toggles and such.

I mean it's a very similar uh

So we have, you know, so we have OEP 1, OEP 2, whatever, and it's um, and so this is open edX proposals similar to Python enhancement proposals.

But yeah, so there's one on feature toggles, and definitely the reason we we decide we need to have some good guidelines and best practices for it is because um we do when you're when you're now we're talking about scaling um and even the deployment of features, right?

And we don't want we want to be able to allow teams to try features and whatnot in a way that it is more controllable.

We want to decouple deployment from um or or enablement of a feature from deployment of the code.

So this way teams have their own um autonomy on when exactly they enable a feature toggle and how it is rolled out to the user base.

Perhaps they want to have it in beta testing and then eventually um roll it out to a larger user base and so forth.

So um anyway, we could we can share the OAP in the uh notes if you want of the podcast.

But it goes through a bunch of different like use cases.

So um, you know, you like I said, beta testing is one.

Some of the other ones might just be that we want to, for operational reasons, we want to just roll out gradually in case we are concerned there might be a performance issue or a scalability issue or even functional issues and whatnot.

And so it allows a lot more control over that.

Um, and the the thing about this though is this like feature toggles are great and it gives you that control, but then also increases the permutations in your code base.

And um and so there's this also process of how do we then um make sure that feature toggles that were created and all of those code branches are then deleted once they are no longer needed.

How do you the cleanup is always the hard part?

So that allows us to then um you know monitor, monitor those toggles.

And and yes, as uh Will, as you said, we are using Django waffles.

Django waffles are great, it allows us to um you know specify whether they're on or off, and then also which subset of users we want to turn it on for, and things like that.

Yeah.

Um well and so another um another thing we we've talked about, I wanted to highlight uh so Django celery usage.

Uh and this is particularly relevant because at Django Con there's a lot of talk around async um starting to be rolled out, and so um both how you use Django Celery, and I guess the huge question is um if and when or when Django is fully async, do you see any use cases at edX, or is it more of a side thing?

Because that's like the two-parter kind of uh for a lot of folks, like do I actually need it when celery and task you know and cues work pretty well.

So that's a lot of questions.

Take whatever you want to answer.

That sounds good.

Okay, so I mean for us, because we're thinking about running edX in scale, right?

We and there's you know, we want to keep the response time back to the users as you know within within one to two seconds.

Um, and if there are going to be some tasks or some requests that are gonna take longer, either because we need to recompute your grades or we need to, you know, do some extra work in the background and whatnot, it's very important for us to separate those into an asynchronous task.

And one of the things that we found was that um being intentional about perhaps even separating our reads from our writes would be is is very valuable.

Um and so when someone is um so basically don't do too many side effects and don't do too many costly operations uh within a request, especially if it's a user-facing request and the user wants to respond quickly.

And so you need to be able to uh put some task into an asynchronous uh you know operation.

And so Django Celery was definitely one that um is a technology that we've used, and you know initially we were doing it on RabbitMQ, then we converted to Redis for scalability reasons and other reasons, but um, but but that's definitely has been a way for us to scale out our infrastructure.

I would love though to move to a towards another model because one of the one of the downsides with salary is that um as the per as the task that call basically it's not a it's not a fully pub sub model, so you do need to, if you want to be able to have the salary task run on a separate service, you do you do need to know what you know like the A it's not very easy to have an API that is more of like a the subscriber needs to know what the

publisher's API and vice versa, right?

You know what I mean?

Like it's it's you're you're too tightly coupled.

So you have to import the monolith into the Celery instance to run the queue.

And it's like I it would be nice if if tasks could be sort of separate, if they didn't have to know

And so um so so anyway, so that that that's where I would like to lead to eventually.

And I was thinking about more of an eventing architecture, but but maybe there's something else, and I need to learn more about async and what it provides, and there might be some things out of the box now once we upgrade to Django 2 in a couple of months, but yeah.

Yeah.

Most people do.

Like when we put out a version of uh a release of OpenEdicks for people to use because we do have sort of um like we we we named them after trees but uh so yeah we we put out ironwood and universities that use it are not going to upgrade until like maybe the the next you know the following summer or something when school's not in session and uh and so it's so yeah we're not gonna see uh async on the the big repos so the smaller repos like tend

to go up faster um because there's just less sort of inertia to to move and also uh fewer fewer groups um uh run like the the e-commerce service that we have uh as opposed to the you know everyone runs that x platform because it's where the LMS and the the authoring environment is right um so yeah on in terms of the async though like there are features I would love to to play with yeah like what um I mean channels is is is great I

I played around with uh version one of channels before I can't play around with version two on that X platform because you know

Uh and frankly there is a lot of stuff around like WebSockets and such that we love to use.

Having async support for like service uh service calls that are not blocking can make you know better use of um uh better use of resources is great although I think edX platform is a bit unusual for Django project in that a lot of our performance issues or performance challenges are were actually uh CPU and the CPU and cache

Like what's because okay because I as as we've been talking I've imagined that you've got database scaling issues you've got a lot of content you've got videos you've got this course content you've got other loading sites uh loading pages but where where are the calculation issues that's kind of interesting

started with a course a single course that was being offered to both MIT students and to um and as a MOOC version that ran sort of two weeks behind the MIT cohort and you know like okay when Coursera if in like those early days if like a Coursera or Udacity uh they can say oh you know we're we're going to push back the the start of this course by two weeks because you know you know whatever to telefire now and they're they're you know non-paying

students are like okay that's cool whatever um you don't you don't tell MIT you have to delay the semester for two weeks right like that that's just not gonna happen.

So a lot of things like we those things were like really really like two um uh sort of very quickly put together and one of the things that was put together was that the course format in those early days course teams it was just it was it was XML it was like a giant XML file.

In fact it was a giant XML file that was a Mako template.

And so the entire like definition of what the course is all the sequences all the problems were this big file that that Django read on that that uh the project read on startup and it's really easy to like create a a set of objects like this very quickly um and that's fine for the prototype but you know obviously like you know the the prototype days are over and we are in a world now where everything's in databases like like you would expect but a lot of the sort of access patterns if you look and

sort of those early days like when you're trying to prototype something you're basically you're going for you know maximum power with minimum code and you can create it is easier to create really powerful interfaces than it is to create performant data models.

Right so like as an example like the if you look at the grading code like the original grading code for for hex platform was basically okay here's my tree of content of everything in the course I'm gonna look I'm going to like crawl through do like check you know go through the whole tree by checking my children and for each node that can be graded as a gradable thing like a problem I'm going to ask that pluggable interface hey how many points are you worth and how many points did the the student earn based on

their like current state of your problem because we have this notion of like X modules back then and like videos were X modules problems were X modules or everything.

So you had some common set of interfaces you could ask.

And that gave you a great deal of flexibility in terms of power but at the same time you you know how long does it take to show the progress page and when you have to ask this question of every node in the course and the answer is you have no idea.

Because you you in sort of you created this interface where like it's like max score and get score whatever and you've pushed off having to think about the data model but in return for that you've lost all ability to like reason about the performance of the system.

So we had X modules that started sandbox processes like in Python and did RPC calls like you know to to sort of get information because you don't want untrusted code to be running.

We had ones that would like parse their internal XML and depending on how many response types you had inside would change their score accordingly we had ones that would make HTTP calls to another system entirely to get back like what the latest score for uh for you know that person was because the first version of peer grading happened on like a a a different service and so you know like yeah if you wanted to make an X module that would you know

return you a different max possible grade for username starting with C on Tuesday afternoons like there's nothing in the contract that explicitly forbids that.

cts that you can just make requests for and and leave it to them to figure out how to implement it.

And you sort of shift that relationship to be okay I'm gonna have this I'm gonna have a grading system.

The grading system has a data model like and the I know that the the grading system I can query it I can ask for things like what is this student's um grade in the course and get a quick reply and then I'm going to make those uh the X modules and Xbox the the courseware like individual leaf nodes they're going to push data into that data model so that I can query that efficiently right so you but and you have this kind of like shift and and that applies to a bunch of things you know we we have to

do that for grading we're doing that for more scheduling related things um but it's hard to do that in a sort of backwards compatible way because we have a lot of a lot of course teams expend a lot of effort into a lot of like course content um and so I think one of the things that Namisha was uh was sort of pointing at was one of the ways we do this is to kind of shift it like uh use async methods like celery tasks um not async as in the the Python

async but uh use celery tasks in order to kind of shift the burden of processing so for instance right now what happens is that there is a grading system and it will hold your it will hold your scores like whenever they change um but there's a set of very complicated permissions calculations that came about because hey we have this whole thing loaded in memory already right right right guys right yeah yeah um yeah and so uh now what

we do is when you change the score that all that those crazy computations still happen but they happen in a solar task that runs and then it puts your like calculated score into this uh grading system that has a a a a data model that you can actually make guarantees about and so that's been sort of our bridge in a lot of ways to to take things from this like the the smart object world to the like discrete services world um but but yeah the but that's that's the reason why we're CPU bound in so many th

ings.

I wouldn't go via the old mechanism.

Is that

And if you want to ask questions uh about a student's grade, yeah, you would you would go to the new thing.

You wouldn't you wouldn't want to mess in in that one.

But but in order to make that happen, because we had so much flexibility with those X modules and X blocks and what they could do and giving us data in the runtime and all that stuff, um, we were inspired by it's like the reverse ETL, you know, with ETL, you know, basically your your your with ETL you're you're reading the data, then you're transforming it, and then you're writing into new form.

It's reverse in the sense that basically we allowed these plugins and these Xbox, right, to basically give us a lot of that content.

And they were able to then write and push that into a form that then they were able to then transform however they want to for read optimizations.

So basically that's where that separation of reads versus writes come in, where like we want to have very fast read optimization views of this data and do it in a so we we implemented this very simple interface where basically allowing anyone to collect the data and then allowing anyone to transform the data.

Um, and then we automatically look through all of the transformers that are registered in our system, run them through, sort of like an ETL type of thing there, except it's LTE and that you know it's it's allowing people to write.

Um and then when it comes to responding to the user, we're able to do that very quickly because it's already transformed for read optimization, and there and then the response times are now once again improved.

So it's tricks like this that have allowed us to um you know figure out how we can take what we had legacy-wise, they were virtual, yes, a lot more generic generic interfaces.

We're gonna be more intentional going forward with what our APIs and interfaces are, but we still need it to support the old.

So, how do we do that in a way that's performant, but you know, now with um optimizations in mind.

Like, like there are um if you are in a situation where you don't know what the interface really is for like what what you should and shouldn't allow for grading, and also where there are bugs all over the place, right?

We're we're building the platform as the first course is running on it, and oh this this bug, uh like you know, you detect this bug as is misscoring someone or like you know, is interpreting the score wrong, then if you store it in a persistent state, then okay, you have what you we have tasks to like re-grade them and re-restore and modify their stores and whatever.

Uh if it's always dynamically computed on the fly, then like yeah, just reload the progress page, boom, where you're off your your score is fixed.

Um, and so it it was one of those things where like, you know, and even the sort of query anything, like one of the one of the sort of painful things about moving from like query the smart object to like separate systems is that you do lose some power, right?

Like the very original version of the prototype that we had had A-B tests running in a totally hacky crazy way, but they were running.

Uh, and it took us years to get that functionality back in a sort of more like predictable and performant way, just because like you yeah, it's it's so that there were trade-offs, and um I definitely like I I have suffered I have suffered from the from the grading system more than most, uh so is Namisha, but uh and so definitely I'm not going to say that that is the way you should architect it, but I guess it's I feel like it's important to note that there are trade-offs, especially for something that as young and quick as we were doing back then.

You can say, oh, what do you well, you know, what does your grading system involve?

And it's like, well, A to F?

No, it's loads more than that.

What does it actually involve?

Well, we don't know.

So well, let's build something, let you use it, and then when we've when we've exposed what the actual requirements are, then we can build something which scales up.

Um you'd mentioned just generally there was a number of security things that edX had come up with that um to help Django but weren't part of Django Core.

Yes, yes.

Um I think Django's definitely improved, at least from the last five years that I've seen, like over time, in terms of having more secure defaults.

Um I think uh for us, we we do have a security working group within edX that triages issues that come in, and so over time there have been some issues that are more prevalent than others.

Um, the automated discovery of CSRF issues, that was something that I had written a while back, and sorry, I'm trying to just context switch what that was in exactly, but I think what that was was um you know basically side effecting get requests, right?

Whenever someone makes a GET request, we want to make sure that there are no modifications to a model or um you know to to any data right in your system.

Yes, and so and courseware, yes, might have a legitimate issue that when it did that way, but we we found so basically I had created this middleware, it was a Django middleware, I believe, and um whenever there was a uh uh it detected that there was a GET request being made, and then what it did was actually I think it used Django signals to for up uh to track any post save model changes, and so it was able to detect if any Django model changed

whenever a GET request was made, as opposed to post request, right?

So um, and that then then basically we reported this, and um then you know you would be able to realize that this is a potential CSRF issue that needs to be fixed.

Um, and and I think there was other things too that I detected.

I may have detected if there were a Django celery task that was also being initiated whenever a GET request was made.

So anything that could have potentially had a side effecting, um, you know, a side effect that could have resulted.

Um so so that's that's something that we had created.

We I think what happened was we found a few CSRF issues within our system, and so we had then been fixing them over time.

And um uh so at some point we wanted to make it a public uh you know application others could also use, but um we we haven't gotten around to that yet.

So we'd love to be able to contribute that as well.

Yeah.

I mean, you know, if they're extracted nice and small and easy to implement, yeah.

I mean, just going back to the uh CSRF example, that's a great use of signals, right?

So people use signals as a way of decoupling their application, but they tend they can get overused and they can lead to hard to understand and hard to follow code.

So the sort of um maxim I always use is do you know at runtime who the receiver of this signal is gonna be, who's gonna act on this signal?

If you do, don't use a signal.

But of course, when you're monitoring, is there are there any saves across any models in this request?

You don't know who's gonna be sending it, so you don't, it's perfect case for signals.

It's it's a lovely example.

No, signals can be very powerful, and there are actually good use cases for it.

So this is definitely one.

Another one I will say is that you know, when we're thinking about the monolith and the core, and then we have these Django app plugins, because there too, we don't know who the recipient is, and it's not core to our platform, we'd actually prefer to use Django signals as a way of communicating that out.

But then to our earlier point, you know, that has then becomes an intentional interface.

Um, but Django signals does allow asynchronous, it allows decoupling um, you know, of your components.

And so, but yeah, I I agree with you.

That's that's a good rule of thumb.

If you don't know the recipient, then it definitely makes sense.

I mean, it's it's this is a possible interface that you could support.

I mean, I'm sure we could keep talking.

I know we're kind of run out of time.

Um, thank you so much for sharing the realities of a large-scale uh code base because this is how it is for everyone.

There's always the trade-off between prototype and and legacy.

And um, you know, I guess the thing when uh when beginners ask me sometimes about these questions, I say it's an honor to be where you all are at, you know.

As frustrating as it is, it's like these are the problems you want to have that you're at scale and you're keeping up to date with Django.

Um so it's fascinating to hear, you know, kind of under the hood how it's all working and how you're thinking about it.

And we'll link in the show notes to a number of these, um, especially the the the OAP.

That would be great.

Thank you, Namisha.

Bye bye.

Bye bye.