Transcript: Coverage.py - Ned Batchelder

I'm Will Vincent, joined by Carlton Gibson.

Hello, Carlton.

This week we're joined by Ned Batchelder.

So Ned, welcome to the show.

Um so we're gonna talk a lot about testing, but also your your career in Python and Django coverage and all sorts of things.

So um really happy you could come on.

You're someone we've wanted to have on for a long time now.

So thank you for making the time.

I think we've been I think we've been exchanging emails about this possibility for a while.

I mean the the short list I had uh when we started this was um you were on it.

So nice.

I'm honored to be early on the list and late on the recording.

It's very smooth.

Yes.

You run coverage.py, you've given lots of talks at PyCons here.

Um you've been most recently at edX.

Um I think everyone in most people in the Boston Python community know who you are, and many of the people um at PyCons know who you are.

But for those listening who perhaps don't, how do you describe yourself these days um in terms of your background in the community?

Um I I often say I'm deeply embedded in the Python community.

Um you touched on Boston Python, so I'm I'm here in Boston, Will, as you are, and um, or at least last I heard, who knows what happened has happened in in the 18 months of the lockdown.

People seem to s move all over the place.

But um Yeah, so I'm I've been organizing the Boston Python user group for over a decade, I guess, now.

Um you mentioned PyCon.

I I do I have done a lot of PyCon talks, but with the pandemic and a and a few uh missed PyCons, it's actually been a while since I've been to a PyCon.

And I haven't done any of the virtual Pycons.

So I I need to get back into that.

Um but I have been maintaining coverage.py for a very long time, um, and as well as some other much less interesting side projects.

And I work at edX as a day job, which is an open source project built on Django, and I work on the open source team at edX.

So I my whole life is open source Python pretty much.

And um you have this very cool mapping project I was looking at on your site, which we'll have a link to.

Um and so I'm I think I have a pretty good idea where you live, but I live in Brookline as well.

So I've been during the pandemic, my exercise of choice has been walking because it keeps me apart from people and I can do it from home.

And my son was at home with me and he needed to walk too.

And to keep it interesting, I've been trying to walk on new streets every time.

And I've done now I think 302 walks from the house to the house, and only two of them have missed getting any new streets.

So it's been a fun, nerdy way to get exercise.

And and we don't have time to get into all of the nerdy details that go through my head when planning walks and kicking myself for having missed a street I could have walked on and all of that stuff.

Uh okay.

And you put it on I saw an image go past this week, I think.

I I didn't know about this project, and then you've blogged about it on your site.

There have been two blogs, well, three blog posts technically about it so far.

But yeah, I when I completed my 300th walk, I did another kind of visualization about how hard it is to get to new streets on walks 201 through 300, because you have to walk over all of the old streets you've already walked on to get to the new streets and et cetera, et cetera.

So, or the two and a half mile radius, I guess.

Um, so yeah.

It's it's been a lot of fun.

Is that would you say that's part of a programmer um personality type thing there?

I don't mind the social distancing.

I'm I'm jealous.

I was stuck here with an infant the whole time.

I and I've walked on streets during these walks that are within 50 yards of my house that I never had a reason to visit before.

So it is eye-opening.

Um so Python and Django.

Um, maybe I could just ask.

So you have a long history with Django.

I think you said I've seen you say back to 2006.

Um, curious if you could talk about that and how you've how you've seen the project develop since that's pretty early.

That is early.

So in I think November of 2005, Boston Python as one of their events said, hey, let's do sort of a shootout of some web frameworks.

And different people volunteered for different frameworks.

I actually got Turbo Gears as the thing to try out, and someone else got this other thing, that new thing called Django to try out.

Um and we built some app.

We chose an app and we each built the same app and compared.

Um but shortly after that, in January of 2006, I actually got a new job working for a startup called Tableau, T-A-B-B-L-O, not the database.

Um, but that was all built in in Django and Python.

And the reason I got the job is that the founder of the company had asked around in Boston, like, how can I find some Python expertise?

And that eventually led him to me, and we had a coffee and you know, the rest is history.

And that was uh on Django 0.91 in 2006.

And and actually the first my first week on the job, um, I had given three weeks' notice at the old job instead of two weeks, and so uh through bad planning, my first week on the job was a week that my boss and the founder of the startup was going to be on the other side of the country for a business trip.

But you know, Monday morning I showed up in the office and he said, Can you put build some like ACL stuff?

We need some access controls on this app we're building.

See you later.

And I spent the week digging into Django and hacking around and building ACLs, and he showed up like Friday afternoon and I showed him what I was built, and you know, we were both happy with each other from then on.

So it was kind of a a trial by fire, but it worked out.

I mean, and it was the size the size of the startup was I think three engineers including me and the founder, and so there was just a lot of heads-down time just hacking on stuff.

There was not a lot of overhead or or difficulty um organizationally.

So that was great.

It was very early on.

Um, I remember we had one problem with our homepage that it was it was doing a 9,009 queries to build the homepage or something.

And the reason was that if you if you built, if you if you got a related object and then asked the related object for the parent, it would do a new query query to get the parent, even though the parent is how you got to the object in the first place.

And so I submitted a patch to Django that would just hold on to that object for later so that we only needed a hundred queries or something.

Like it was just one of those early on in the orm, you know, not there hadn't been that much optimization done.

So there was a lot of low-hanging fruit, and so early days you could you could dig into those kinds of things and make those kind of patches.

The great thing about Django now, in some ways, is there isn't that low-hanging fruit because it's so mature, but in those days there was still a lot of work that needed to be done to bring it up to speed.

You had to roll your own functions.

But we don't have to get into that.

I'm sure if I record.

So uh the we're gonna touch on this theme a bunch, I imagine, but I tend to start do a thing once and then keep doing that thing for a very long time and and not consider new things.

And the difference between function-based views and class-based views fits neatly into that into that narrative.

I know how to write a function view as a view, and class-based views to me still seem weird.

Yeah.

And I mean, on so since those early days of like writing spending a week writing access controls, I haven't done that much hardcore direct Django development.

Um, so in some ways it's you know, it's really moved on without me.

And I've it w that's one of the challenges, you know, you being a long time adopter of technology can sometimes mean that you're an expert in the way that technology was fifteen years ago and you're stuck because you've just been spending your time doing something else rather than keeping up with all the amazing work that Django's been putting out.

So like async don't even talk to me about async.

I will was asking me earlier on about something how how do I organize a I don't know where do I put my settings files or something and then I described a little file system layout to him and he's like oh old school I'm like yeah probably like

I would say these days you can use environment variables so you have one file and then you swap in with the environment variables.

But still works for Carlton so you know all the time

You know even if it's one file it's still

Like I know what I want to build I just want to build it.

You know it's it's sort of like installing Python.

If you like you know we have our everyone has a different way to do it unless you though actually I I did want to say though like with running Boston Python unusually you get a lot of exposure to a wide spectrum I mean both I mean because you have talks and there's pr um project nights you get total beginners and then you get you know postdocs at you know science places.

So um I want to just broadly ask you about that because you you you you probably see more of a spectrum than I do whereas most people are in their little lane with what what with uh you know work what's in front of them.

Like now we do it like this and and the old stuff doesn't matter right Carlton and I are are burdened with all that luggage um so we we will probably carry some of that around and and we may touch on another recent thing which was me redoing my website my personal website as an actual Django project which you know full disclosure I think I have seven or eight different settings files with different names of like base and that server name and et cetera et cetera and I know there's a better way to do it but I also know

that if I go and look for the better way to do it I'm gonna be presented you know in classic Python style with a dozen different options hacked together by a dozen different people each of whom thinks their way is the best and it's hard to sort it all out and et cetera et cetera again one of the one of the nice things about Django is the out of the box like Django will tell you how to do it kind of mentality which is a rare island in the chaos of the Python world where we love the fact that people can hack together

something and and get it out there and people will start using it but that makes for a lot of competition among all these solutions that can be really overwhelming for for beginners or not even beginners but people for whom it's not their their passion.

Like I don't I don't want to make settings files.

I want to make a website so please just tell me how to make my settings files so I can get on with the thing I'm trying to do.

But it's not JavaScript, right?

I mean because I feel that but then when I whenever I

JavaScript I'm always like wow compared to JavaScript Python is gloriously

So they haven't they haven't embraced the idea that there are many right ways to do things.

They've sort of embraced the idea that there is always a right way to do things but it's different every single year.

And and that's just as that's almost more difficult to keep up with because now it's not just like oh I chose A and you chose B.

It's I'm doing it right and you're doing it wrong because you're still doing it last year's way.

That's the way it feels to me at least maybe if I were more embedded in the JavaScript world it would it wouldn't feel quite so foreign to me.

But you you asked me about Boston Python and the beginners and it is true Boston Python has a wide range of people attending.

You mentioned talks and project nights we've actually slowed down quite a bit during the lockdown.

We haven't had a presentation night in a while one of the interesting things about running a a geographically based meetup during a pandemic is that all everything becomes virtual and then all sorts of people who live nowhere near you start showing up to your events and that's okay I guess I'm not I'm not sure like what's the word Boston mean in Boston Python if everything's on Zoom now?

I don't know.

I think that's right.

We get a lot of Python beginners and a lot of data beginners.

Yeah well there is a and I'm not sure there is a Django Django doesn't get as much true which I s isn't as large.

Which is a good I was like man I'm I don't know if I should go all the time and yet you found the time.

But it's a lot of I mean it's a lot of grad students or it's you know or people getting a graduate degree and whatever and realize they want to you know script something and so Python.

So yeah not much of a web focus.

And we've been doing the thing we've been doing in Boston Python during the lockdown, the new thing we've been doing is weekly office hours.

So Mondays at noon we just get on Zoom for an hour and it's that's one of those things that we couldn't have done in person um because who's gonna travel for an hour long thing at lunchtime.

But it's easy to do uh weekly and there's no prep for it you just show up and it's like people just start talking about whatever and maybe someone knows the answer or wants to talk about it and you talk.

So that's been very easy.

I feel maybe it's just because I get a lot of emails from people which I do respond to everyone eventually but I feel like I couldn't do that.

It would drive me nuts.

But I wish I could

I mean that's that's the open source dilemma right you can't you you no no one is owed your attention but you get into it because you want to give them attention and how do you how do you strike that balance and feel good about it and still make progress on the things you want to do and stuff like that.

So I mean

Do you have wisdom on that?

Like that's it is very easy when you put a project out there.

So as as Will mentioned in the introduction I'm the maintainer of coverage.py which is the coverage measurement tool for Python

See that's why

So I did it I did it then.

So yeah so coverage.py right so it's it's a big project in the sense that lots of project people use it because there really isn't any competition for coverage measurement in the Python world.

Most people don't even realize that the standard library has a coverage measurement module in it called trace dot they just come and get a third party module called coverage.py which I maintain um but as a result of that it's very there's a lot of issues that get written that I can't debug because they tend to be about esoteric execution environments like oh my tensor flow something or other didn't get coverage measured and I'm like I only understood half of those words and I'm not I don't have the time to go and figure that out.

So it sits there.

It's very easy to feel like well I put this out in the world so I, you know, I need to make it work.

And if it doesn't work, then that's me being a bad person, not fixing it.

And you have to get past that feeling.

You have to you you have to allow yourself to take time off or just turn away from it for a while.

There's there are months where I'm just not interested to work on coverage.py.

So I just don't.

Um and occasionally it'll feel like, well, maybe I'm never going to get back to it, and maybe I'm just done with it.

And then something comes up and an interesting bug or s or a release, and I was like, oh, coverage.py, that's a cool thing.

I'll do I'll do some stuff on it.

You know, it'll and and if it didn't come back, all right, then it wouldn't, you know, maybe there'll be a whole year where it never comes back to me, and that'll be it.

I don't know.

I mean, you two are both uniquely in this bucket.

I mean, because Carlton, aside from Django, has a number of projects that he maintains over many years as well.

It's exactly that.

Someone will post a bug and it'll be like you know, a an Nginx config and a few lines of some logging file, and uh, you know, and it's like I'm sorry, I don't even know where to begin to respond to this question.

And so I've been trying to click move to discussion for those.

Um because then it keeps the which it's it's a new thing.

And it's like, well, it's not it's not that I just want to click close, right?

But I can't answer this, it's not really an issue.

So maybe if I move it to discussion, it keeps the issues less interesting.

Because one one of the one of the sort of demotivators for me is when the the you know the issues are piling up and it's like I feel a pressure, and it's like, oh do you know what I haven't got the emotional space to look even look at the repo.

Whereas if I can if I can move those issues over to discussions, those unaddressable ones over to discussion, then maybe someone else finds it searching and they're like, Oh, I had this problem and I did this, and that does happen.

But then the issues can be like, oh, actually, there's an addressingable you know, code problem that's identifiable.

So the coverage.py repo I think has 200 open issues right now.

Okay.

Going back years, I'm okay with that.

Whatever.

Um, some people say you should close them if you're not gonna fix them.

The author can close it.

If they want to close it, they can close it.

Um, I mean I think the signals are pretty clear in that repo.

Your issue might not get looked at.

Um and so um one thing that has been great is that people maybe are getting better at making reproducible test cases.

A few times I've gotten a Docker image with the bug report.

Like, here is the Docker image showing it happening, and I'm like, great.

I know I can now I can see it.

Um because you know, too often you get people saying, Look, you know, it it failed like this.

Can you give me a reproducible case?

Well, here's a link to my Jenkins job that failed.

I don't how many times you know, how far in do I have to click before I can even maybe find the error message you're talking about?

You know, if you can't put work into this, why am I putting work into this?

So if you're listeners out there, when you write a bug, make it really, really reproducible.

Write, write the instructions for that art school friend of yours who doesn't know how to use computers, and and you are gonna be doing the maintainer a huge favor.

But you'd be doing myself

Oh, that's my code.

Oh, why is that?

You know, you've got a chance.

And I don't know anything about how TensorFlow works.

So we have to meet in the middle somehow.

Yeah.

I'll I'm I'm willing to dig into it.

It's interesting to dig into it.

There was a a bug report about pony or pony ORM, um which is one of these tools that they take Python code, I think it's the Pony ORM.

They they you write a query and in Python code and then they rewrite that Python code.

And so when your query is running, it's not actually your code running, it's a rewritten version of your code running.

So when you try to do coverage measurement, coverage doesn't think your code is running because your code isn't running.

It's been moved over into a parallel universe, and and that was very interesting to see.

I think that was the point.

I so I have a pull request against Pony RM to make that a little bit clearer to people, and they haven't bothered to merge it yet, even though it's really simple.

But there you go.

Turnaround is fair play, I guess, right?

Has there been any talk of so we have this in the Django community where there's third-party packages and the Django is kept deliberately quite small.

Um, you know, you only pull in things when you really need to.

Has there been any talk of pulling coverage into Python?

I mean, I know the PSF has fellows and stuff.

I mean, it's pretty much a standard tool for anyone I know who does Python professionally.

Um there hasn't been.

So there's two, there's two things you might have meant by that.

One is pull it into the standard library, and the other is move it under the PSF organization.

Uh just as a question.

Yeah, so there uh there hasn't been any discussion e discussion of either of them.

Um these days I think Python is doing the right thing by being very reluctant to put things in the standard library under the theory that third-party packages can evolve more frequently and will keep the bloat down on the Python releases, and it's just better to have things decoupled than to have everything piled into the box.

Um, that you know, we should all get better at installing and using third-party packages rather than hoping everything's gonna go into the standard library.

Um, and the PSF thing, I think the moving I so the two projects that I know off the top of my head that are under the PSF organization on GitHub are requests and black.

Um and I'm not quite sure, I don't know, that that one always struck me as an odd thing.

I'm not sure why the PSF, which doesn't even own the Python repo, why it would own third-party package repos.

I mean on requests it was because um the maintainer was stepping away and w you know there was some other stuff, so I think wanted to keep it going.

I'm not sure about Black.

Um but I know they have someone

So he's he's well in place there in the PSF and very involved.

It's not he's hardly walking away from it.

So I don't I don't know quite no know what it means when something is under the PSF organization.

Right, because it's not the PSF that's maintaining it.

I suppose it just means maybe a fun well yeah.

We always I mean we Django look to the PSF as sort of a big brother-sister in terms of size on these things because many of the same issues around um organizing.

So um we often look to the PSF for how to do things.

For instance, the PSF has PEPS, Python enhancement proposals, OpenEdX has OAPs, OpenEDX uh OpenEdX proposals, I guess we call them.

Um not quite the same acronym.

But so no no, the moving coverage.py hasn't been suggested, and maybe that's just because I I keep maintaining it enough.

I mean everything's fine, I guess.

Right.

I mean it's it's interesting maintaining coverage.py because um I get involved with new Python releases because coverage is intimately aware of how different Python releases run the trace function to tell to indicate what lines got run and what didn't.

Um and and 3.10, the reason I there's two reasons why I co coordinated the coverage six release with the 3.10 release.

Um uh Mark Hammond was doing a lot of work uh in 3.10 to fix, quote, fix how the trace function traced things.

There were lots of weird edge case cases, and he was doing a lot of work um to to fix that.

And when he fixed it, coverage.py would break because coverage.py was used to the old broken way that things were getting traced.

And uh so I think I wrote, I think, 10 different bug reports against Python 3.10 during its alpha phase saying it looks like this now, is this what you meant?

And about half of them were, yes, it's what I meant, and about half of them were like, oh no, that's not what I meant.

And so we were kind of going back 50-50.

What you know, does coverage have to change now or does Python have to change now?

Um and we finally got that all straightened out, I think, by the time of at least beta two, maybe even beta one.

Um so I was really happy to be able to push that out there.

But the other thing that coverage.py had to do for 3.10 is that 3.10 now has the match case syntax um for doing pattern matching.

And that you know is a change in execution and ironically a change in in syntax highlighting, which is something that coverage.py does along the way too.

So um there were a few, it was very involved release to get coverage.py synced up with 3.10.

That's I mean, um the the new syntax in 3.10 is quite cool.

In Django, we don't get to use it for years because obviously we've yes.

You know, we're still on 3.6

But quite a

There was a time when coverage.py would run on everything from Python 2.3 up to, I think, three point four at the time or something like that.

Um coverage six is mostly called six because I dropped Python 2 support.

Um so now I can use F strings, for instance.

So we're on 3.6 plus now as well.

Because so um Python 3.5 was end of life last um end of last year and 3.6 is gonna be end of life this December, I think.

And so what's your policy about dropping the support now with the new annual release cycle?

Um I know some people are like, oh, Python 2's out of support on January 1st, 2020, 2020.

I am pushing my new version that drops Python 2 on January 2nd.

And and I'm doing it, you know, for the good of everybody, because everybody should move up and etc.

etc.

And my feeling is look, I'm I'm building this tool so that people can use it.

And if keeping Python 2 support isn't a pain, then you know why not just keep it?

Um coverage six really what happened was that there was a different change in coverage that felt like it was going to need a major version bump because it changed the behavior.

Um coverage five and before would often accidentally measure third-party packages that you'd installed in your site packages.

Um and in coverage six we made a change that was it was cleverer about where the code was coming from and it would automatically exclude things that had been installed where third-party packages go.

And that's great, but some people said, you know, actually that kind of broke my coverage and I have to go in and change my configuration.

And so that felt like that should be a major version bump.

And since I was doing a major version bump, and it's been 18 months since Python 2 went away, I figured I might as well, you know, let's clean this up now.

Um, folks can still pin to the old version.

Um, yeah, that's always the consolation is you know, you're not by dropping support, you're not dropping anyone using your code because they can just keep using whatever code they were using.

So there was an article in the the news this week in the you know, the tech news about um uh what was it, that coverage itself isn't a a good marker of the the test space.

And what what they dug into was that it's the number of tests that help.

Like you know, coverage is handy.

And I I I remember asking you a question on Twitter about this a few months ago about is there a way of sort of marking tests, uh of marking which tests are meant to cover what?

Because I might have a few Selenium tests which are kind of big end-to-end ones, and just by running those, I get you know, I get quite a high coverage number.

And yeah, exactly.

I'm uh I guess I'd ask, well, what's your thought on that and how coverage works, you know, and how you build a quality suite?

And and and by the way, depending on what you do in your asserts, you might literally not be testing, you might be running lots of code and not testing any of it, right?

So um I didn't I didn't exactly read, I think I saw that headline go by about the coverage uh problem.

I didn't go and read the article.

Um, but your point is a good one that you can write a test which exercises lots of code but can't actually assess the results of all that code.

And if you had a way of marking a test saying that when I run this test, I don't want you to count the coverage or only count the coverage on this part of it.

Yeah.

Um and that there there have been a few ideas about that in the coverage um issue tracker, none of which we've we've implemented.

The the big I keep saying we, but it's really just me.

Um you and your alternative us.

Me, that's right.

The big thing in in coverage five was that you we it the package can tell you um for each line of code which tests ran to cover that line, which kind of gets at that same issue, right?

So you can take a look at the coverage report and you can see, oh, that line was actually only run by the integration test, for instance.

Was it could I could I somehow work this backwards such that I could say that I'm writing this test and it's for that function.

And one of the things that's that's that keeps me interested in coverage is those new kinds of ideas.

Like that's a whole other interesting question.

Like, what can I say about a test that could tell coverage to focus more accurately on what that test is doing?

Like some people will say, well, it should only measure the coverage of the immediate functions it's called, it calls, but it shouldn't measure the coverage of any function that those functions call.

But that doesn't feel right.

That's too crude a measure.

And I don't know if anyone's gonna want to put the work in to say, well, for this test, only measure the functions in that module, and then for that test, and you've got a thousand tests, you're not gonna go and decorate a thousand tests.

Like so what's the right what's the right balance there between the effort from the developer to say what they know and what they want about the coverage?

And then how do you say that in a way that coverage can make use of?

That's a very interesting thing to me.

And and that might be the next big thing, right?

So coverage five had a lot of big changes.

We switched to SQLite and we switched, we put in the context measurement so you could say which lines were measured by which tests.

Coverage six had a lot to do with fixing the tracing with Python 310.

What's what's next?

I don't know, what's the big feature for coverage seven?

Maybe it's something like that.

Um but I want to make sure that it's something that people will actually get benefit out of.

I don't yeah, you know, you we can we can invent esoteric strange features, and if no one uses them, then what was the point?

So it's it's good to be guided by the questions people ask in the issue tracker about doing these sorts of things.

I mean, so the flip flip side kind of question was that so say I'm doing I've got these nice unit tests which are targeted specifically just to their one function, and then I realize I need to refactor.

And what I actually then need is an integration test around the sort of the outside, and I kind of need to napalm all those nice unit tests.

I mean, what's your thought when you you face a challenge like that?

Like how do you address that that kind of tests?

Unit tests, unit tests by their design are tied to what the units do.

And so moving the units around is gonna require moving the unit tests in some way, either you know, getting rid of half of them or writing fifty percent more of them or changing what they all expect.

And it it's a lot of work.

And and you know, some people some people take the attitude like I'm not gonna bother with unit tests.

What matters is whether the application works as a whole, and I'm just gonna write a bunch of big integration tests, and what are the chances that it a big problem will get through those?

Maybe it's a good trade-off of benefit versus effort.

It's hard to know.

And by the way, so I should say though, for context, the with I uh coverage can tell you which tests ran which lines of code, but you can also do things like run your integration tests and say mark all of that coverage as integration and run the unit tests separately and mark all of that coverage as unit.

And then every line of code will either say integration or unit or both.

Right?

So you can decide sort of how coarse you want it to be.

The problem with marking every test on every line is if you've got a thousand tests and you're gonna have lines that are covered by 200 tests.

There's no point getting a list of the names of 200 tests for a line.

That's just that's too much information.

I went and looked at my own coverage, coverage's own coverage HTML reports, and they're a single file can have like a two-megabyte HTML file result because every line's got dozens of test names annotated onto it, and it's just it's not worth it.

Clump it up at a bigger level.

That's not a level that coverage can intuit by itself, but it gives you the controls where you can you can indicate those things when you run the tests so that you get the information you want.

Yeah, the pollen.

Right.

Eat less mostly plants.

I think that's a really interesting thing, and I wanted to just pull out some of your thoughts on that.

I mean, unless you you say more though, please.

So so it's a edX is OpenEDX, the code base, is a very large project.

I mean, the edX organization on GitHub has 300 repos.

There's probably uh at least a million and a half lines of code.

Um the main repo is called edX platform, and it's a giant Django project.

I think it's probably got about 400,000 lines of code right now.

And a lot of tests.

The tricky thing about me talking about the technology is that I've been mostly working on community issues at edX for a long time.

So my my hands on the bits of Django are pretty few and far between.

But edX does struggle with the bulk of tests.

We have pretty good coverage.

These aren't telling us enough.

Let's just get rid of these.

Which, to mean to be perfectly honest, I was I I felt a visceral shock at the idea of just delete tests.

Um but the fact is that if they aren't telling you anything, and they do take a long time, and especially these in particular, were the sort of front-end tests that tend to be kind of flaky, where not only are they not giving you any value, but they're taking up your time by by with false alarms.

Um so it was the right thing to do is to delete those tests.

And it's it's hard to strike the right balance because you know, developers, maybe like many people, but especially developers, tend to be very um susceptible to gamification.

You know, oh, there's a metric there, there's a needle that moves that way, I want to move it all the way that way.

And so, and you know, I'm I'm w one of the main purveyors of one of those needles, right?

Coverage measurement, what's your percentage?

You're not at 100% yet, you know, you gotta get to 100%.

Um, and that's a lot of work that may not provide much benefit.

So trying to strike a balance rather than just being the best or the last or the first or whatever with it is really hard.

And just I mean, Carlton, we both read it.

It's just so many things that if you're on a big code base, it's like, oh yeah, that that helps, that helps, that helps, that helps.

he's got a chapter on profiling that's worth it just for the chapter on profiling.

It's just amazing.

And we try to stay on the latest, you know, we try to stay on supported Django releases.

We're in the middle right now of moving to Django 3.2 from 2.2.

Um, but it's hard to move, I mean, um I mean aside from the question of what do we have to do to our code to make it work on Django 3.2, we have to go and look at 150 third-party packages and see are they running on 3.2?

And many of them are not because these packages, you know, they get stale.

And then we have to decide, can we get rid of that package?

Can we live with it the way it is?

Do we have to fork that package?

Um, and actually one of our one of our engineering leads uh here at edX, his name is uh Jeremy Bowman, he's doing a whole talk at the upcoming Django Con about um how we manage that and how we're trying to actually be proactive in the community to push out uh what we've learned about other people's packages and getting them onto Django 3.2 so that we can get onto Django 3.2.

So he's he's doing some interesting work uh you know pushing the envelope forward on how an entire community, not just a large million-line project, can move to Django 3.2, but how an entire community of third-party packages can move forward onto Django 3.2.

Yeah, that's uh Yeah.

Um and all the own the only uh bump was to add the Trove classifiers for 3.2 and 4.3.

Like right.

But it's still it's still like you know, do it, clean up the you know, clean up the last few issues, make sure you know update the package.

It's still, you know, it's a session to do and it needs doing.

And okay, for app conf, it's once every you know couple of years, do a release, it's fine.

But you know, other if your package is needs actual updates, then it's all the more.

And for us, it's it's hard to look at a third-party package and decide, oh, is this just a missing the missing metadata or is it actually not going to work on 3.2?

I mean, I think we're all engineers, we're all we all value it.

Um managers don't always.

Um you've worked in a lot of organizations.

What are you what is your advice for an engineer who is trying to sell taking the time to testing to a manager who may or may not be technical and may not see the value in it?

That's a good question.

Um that was the same that was the same boss who said that the way we're gonna test our code is we're gonna push it to production and people are gonna tweet at us.

Maybe it wasn't a good idea.

We're doing it

And and actually they that Bill O'Reilly clip of doing it live was was popular meme around the office.

Absolutely.

Um and I and and so what so one thing that happened there was that they were talking about, you know, we're gonna ship in a month, and I was looking at the software and I thought, this is not gonna be ready in a month, but how do I convince them of that?

And what I did is I brought in, I said we should do some usability testing.

And I got some friends to come in and be the subjects in the usability tests, and I ran the usability tests, and I didn't know much about running usability tests, but you know, I knew enough to sort of pull the wool over those guys' eyes um on the usability test.

And you know, I said, look, this guy's sitting down to use the app you you said we're gonna ship in a month.

What do you it does it are we ready?

And they're like, okay, I can see it, we're not ready.

Um so if the manager is not convinced that testing is worthwhile, um keep an eye on the production outages and try to convince them to do uh root cause analysis and have the analysis come up with what is probably the answer, which is if we'd known that you know X and Y and Z earlier, we could have prevented this.

Um the edX, the edX uh engineering culture is great that way.

We do RCAs all the time and it's a blameless culture.

And I don't know if I've ever heard someone say, you know, we let's just build the feature and get it out there, and then we can write the tests later.

You know, we could we just have to get the feature done in time.

You know, those sorts of ill-advised trade-offs that you can sometimes hear pointy haired bosses making.

Right.

I mean, because I I I came into technology through the business end, and I sometimes talk to MBAs who go and be product managers, and I try to tell them the story of, well, when you go and manage a team of engineers and you don't you're not an engineer, it's very easy if there's a new product that your boss says, you know, get it done in two weeks.

Your engineers say it'll take four weeks, and you crack the whip and find a way to motivate them to get it done in two weeks, and you think, wow, I just gotta crack the whip on these lazy engineers.

And the engineers think, well, this person doesn't know anything, doesn't value testing, the code smells, the tech that accumulates, and so both sides lose, even though the manager thinks they win.

Um having things like I my advice is generally like because the issue is you wanna when you're in charge of managing the team, like that's the onus is on you.

You want to have an out uh an output or an outlet for that.

So I would often recommend saying you need to have like a bug day every month, every couple of weeks, so that the engineers have to prioritize it, because every bug is important to an engineer.

But like, how do you, when you're not technical, figure out which bug really matters?

You say, okay, I'm you know, moving heaven and earth to give you the time and we'll celebrate it and get a gong or something, but you need to prioritize the bugs.

And then you're not just saying no all the time, you're saying, okay, yes, at this date, and then we'll and that sort of doesn't solve the problem, but that helps morale a lot, and it can also sometimes be effective versus you know, every bug that comes up isn't important, time often will help you with that.

Yeah, and and giving the engineers some measure of autonomy over, you know, some at some part of their effort is is a really good thing to do.

Um and by the way, this startup that I'm talking about, it he was not a pointy haired boss.

Um it I like to think of it as sort of a a healthy debate over the costs and benefits of alternate approaches.

Um and we we shipped a good good product, I think.

Like I said, we got acquired by Hewlett-Packard.

It's not around anymore, but it's yeah.

I mean it's a goal.

There's the the truth is it's somewhere south of that, but question is where.

Exactly.

Um Yeah, by the way, when we got hired by Hewlett-Packard, they they looked at us and they they they said, Oh, this this Django Python thing, how quickly can you port it to Java?

Luckily, those those people, those people went away and we never heard from them again.

But uh there was there was way more craziness after the acquisition by Hewlett-Packard than before.

So we're coming a little bit up on time.

Are there any topics that we haven't asked you about or things you want to mention um while you're talking to the Django community?

Um and uh it's interesting.

So that so the the cool thing about uh Django is it's got these templates, and templates aren't meant to have logic in them, but they can have a little bit of logic in them.

You've got if statements and loops and things.

Um and there's a plugin called the Django Coverage plugin that will tell you which aspects, which parts, which lines I guess of your templates have been used in your tests and which have not.

Um and that's an it's an interesting project in and of itself because uh I uh I when I first wrote it, I took the strategy of I don't need to stick to public interfaces.

I can use whatever I want that I find inside Django, and that's fine because it's on me and I'm taking responsibility for that.

Um and it one of the one of the problems with cut with the 6.0 release of coverage is I heard from at least two different people who said 6.0 broke my thing, and I looked at it and like, yeah, your thing was using private stuff that I didn't tell you to use.

Uh it's not my fault.

I don't have to do anything about that.

It was free.

You're welcome.

That's right.

And when it doesn't work anymore because Django has shifted, I update Django coverage plugin to do a different thing.

And I have a testing strategy where every every week on Sunday it runs with the latest tip of Django.

And if it breaks, I will hear about it and I can update it quickly before it becomes a problem.

Ironically, that test run this week only told me about how coverage six broke it and not how Django had broken it.

But you know, that's that's what tests are for to tell you that stuff broke.

The universe changed and your stuff doesn't work anymore.

And so maybe you'll have some work.

I understand that.

Well, we'll definitely link to that.

Anything anything else?

Your other projects aren't as interesting, but cog is an amazing little code generator that

And I wanted a way to generate the two from something, and the something was going to be Python, and I tried using Cheetah, if you remember that templating engine, but that was for text and not for code, so it was difficult.

And so instead I came up with this thing called COG, which is basically a way to have a text file in which you could embed bits of Python and it would run through the text file and execute the Python, and whatever the Python generated would go, would replace the Python in the output.

Um and it worked great for that, um, for the C making C and SQL.

Um and I actually use it to make my PyCon talks.

So I author my PyCon talks in a giant HTML file using a JavaScript-based slide slide package, and there's bits of Python in the presentation that generate parts of the slides.

So when I want you know a diagram or a table that's easier to generate with code than by hand, I put in the Python code and it generates that stuff.

So it's it's lives on it lives on in kind of a much different environment.

And everyone's and it it is it's one of those packages that I hardly ever do anything with, but about once a year I hear from someone who's using it for something or wants to make a change, and so it's got kind of this tiny but dedicated following.

So yeah, that's I guess I had forgotten about COG.

That's that may be my second most successful side project.

Um that's that's similar to its original purpose in life.

Yeah, exactly.

So I wanted to thank you for that.

I think that's like levels, like they say that you know developers want to do a blog so that you end up building your own static site generator, and this is like you doing that for talks.

So I respect that.

Exactly.

That's what you you know, because from that comes coverage.py and who knows what else, right?

If you lose that sense of play.

Yes, it's very it's very helpful to have a side project where either you can do it exactly right because you can't in your day job, or because your day job makes you do it exactly right in your side project, you can do it wrong.

Like just having that outlet for whichever thing.

You're frustrated.

That's right.

Testing, I don't need that.

Yeah, exactly.

Screw it.

My personal side.

And that was what what was the what was the old you're rewriting that as a straight Dango site now, right?

It uh so my my personal site started in 2002 as a bunch of Python code that generated HTML that I would FTP up to.

So you did

pre-pimp free freeze, right?

Was it flask a flask freeze, I think, right?

There's a flask freeze way you could do it.

I forgot.

So yeah.

It was all XSLT.

Uh-huh.

Okay, but don't laugh because it's still all XSLT.

But yeah, that's what I want

Um so it started as just like, let's use XSLT and generate a pile of HTML and FTP it up there.

And then I uh at some point I switched it to being a bunch of a Django site that I would use a static site generator with locally to generate a pile of FTP HTML that I would FTP up to the site.

And to do comments, it had PHP code um in there too.

And there was some interesting Django middleware that would execute PHP along the way for local testing or something like that.

Yeah, it got really wacky.

Um and then this summer my my hosting provider said uh we're gonna you're you we're kicking you off.

They were getting bought and they said they could transfer my site, and then they said they couldn't transfer my site.

So you gotta find a new host, and so I went looking for so I thought fine, I'll just do a real Django site.

I made one little stop on maybe I can just move the site as it is to a new host, but the old site was PHP 5 and the new host was PHP 7, and I definitely didn't want to invest any time in understanding how to upgrade PHP.

So I bit the bullet and I made it a real Django site that's really hosted on on uh uh digital uh Dreamhost, sorry, Dreamhost.

Um and so it's the god I had to re-implement the whole comment system, which was cool, and now I can do better things with the comments and etc, etc.

So but it's still very wacky.

So I still generate a bunch I import a lot of XML files into a SQLite database.

The SQLite databases are synced up to the server where the Django site will use XSLT to convert the XML into HTML and serve it.

So all sorts of wrong decisions, but it works.

It'll be 20 years in spring.

Um but now I can do cool things like if I have images, I can auto-convert them to WebP and serve them as WebP from then on.

So the first hit does the convert and then everyone else gets a better image, and I could just do JPEGs locally.

So it all you know, cool stuff.

Maybe it's the last question to take us out.

Um, so edX, I think it this is public, has just been so a nonprofit that's been acquired by a for-profit company.

I'm just curious what you can say about that and how do you envision that changing your role if at all?

Um but I mean honestly, there's I pretty much mostly I know what's public.

Yeah.

Um not much more than that.

Um you know, people so my role at edX has been sort of the face of open edX within edX, and then outside of edX, my role has been sort of the face of open edX from edX.

So I I think of myself sometimes some people think that I'm like the big brains behind Open EdX or something.

I I've called myself the sidelines mascot for Open EdX, like I'm the guy in the suit dancing around as hard as I can to get everyone excited.

Um so when the announcement was made that 2U, an ed tech company, was going to acquire um at edX, I think that was at the end of June.

Um people were getting in touch with me, like, like, you know, did you organize this?

And I was like, I I found out about it when you found out about it.

You know, it when the news went public, uh that's when I knew.

Um so yeah, to you is acquiring edX, but edX is a nonprofit, and a for-profit company cannot acquire a nonprofit company, or at least I I don't know the legality, but what's actually happening is that there is going to be a new nonprofit formed, which will get all of the proceeds from the sale because you can't you can't create a nonprofit to further education and then sell it to a profit company and then me keep the money.

Like I don't get any of the proceeds of the of the acquisition because it was a nonprofit, right?

The the money that went into the nonprofit has to continue the goal of the nonprofit.

So there's gonna be a new nonprofit who's maintain continues that goal of education, furthering education.

Um, and most of that X is gonna go to you.

Um there will be people who work for the new nonprofit.

We don't know who those people are, we don't know who's gonna run the nonprofit, we don't even know exactly when this is all gonna happen.

It's it's you know under legal review by, I guess, the Attorney General of Massachusetts.

Um so but it's gonna be an interesting time.

So one of the challenges of edX and Open EdX is that I mean, and this is a classic problem with any Django project, is that you s you set out to build a platform, but mostly what you're building is an application.

Um, right?

So like when like Django apps, oh I'm gonna have a blog app in my project.

And your blog app is never really just a blog app, right?

I mean it's not general enough to be used by other people, it's always got connections to the rest of the project, and you hard-coded the name of it in there because it was just too much trouble to make it a setting the right way, and you don't know what the API should be.

Yeah, yeah, yeah, okay, okay.

This is just me.

So way back in 2012, you know, edX was building a site to serve education and it was a Django site.

Um and the goal was always to open source it so other people could use it, but you know, mostly it was just edX that we offered to other people to use.

And over time we've gotten better about making it more of a platform and more generalizable.

But all along the way, we were we were deploying from master, and we deploy multiple times a day from master today to edX.org.

And that's you know the business that pays my salary, so it's very important that we keep it running.

Meanwhile, we also want to get contributions from people into that open source repo, but how do we manage that risk?

Right?

They're not running a site that has 30 million, 40 million, I don't know the exact count now.

50 million learners right now, so they're not dealing with the kinds of scale we're dealing with, they don't know what our roadmap is.

So up until now, the contribution process has been very tightly controlled by edX.

Every change has to be reviewed by edX, roughly.

We're opening it up a bit now, but for the most part.

Now that there's gonna be a separate nonprofit that's going to own OpenEX separate from the edX company that's running the edX.org website, what's the new contribution model going to be?

What's the flow gonna be?

How do how do we keep code moving at high velocity and keep the business stable under two separate legal entities, one of whom is gonna want for the most part to increase contribution, and one of whom for the most part is gonna want to keep the business stable.

So that's a whole new open source dynamic that we're gonna have to have to navigate.

I mean it really's a very interesting time.

It's big.

And you know, like I said, for the most part, edX engineers are focused on how can we make edX.org a little bit better today, not how can we make it easier for someone else to run another site that also does education.

Like no one's opposed to that goal.

It's just probably the fourth or fifth goal in their list, and they're not being measured on that goal.

So it's very easy for it to lag.

And so from the outside, it's like, well, Google or Facebook or Microsoft supports this open source package, and really it's probably a handful of people on the inside fighting uh, you know, they're getting paid by those companies, but you know, their boss isn't saying, nice job on React release.

You know, it's like, okay, when you're done with everything else, you can maybe do that if it helps us hire people.

In other words, you know, this the problems we have, are there other projects that have those problems?

How are they solving them?

And there's always differences between the projects that make it not quite a direct map, and so it's hard to find analogs.

But actually that the the example of React coming out of Facebook actually was a recent one that we discussed.

I don't know how we'd actually get to the bottom of that.

Like how do we get into the cubicles at Facebook to see what their actual you know goals and measurements are and how do they support it?

Maybe we should just go ask.

I don't know.

I don't know.

And there's sort of the classic way, you know, just do it like Red Hat does it.

Well, Red Hat doesn't deploy to master multiple times a day.

Well, just do it like WordPress does it.

Well, we're that's a little different.

So um, you know, it's it's interesting to work in open source and have to sort of rethink it from first principles to make it work for everyone.

Right?

edX loves open source as a way of expanding our engineering capacity, right?

Instead of having, I don't know what it is, 100, 150 engineers at edX, we could have 800 engineers among all of the people using open edX.

But we only get those 800 engineers if we can coordinate their contributions in a way and open up the channels so that the contributions can flow.

And that means we have to tell those 800 engineers how to get started, like Carlton said.

We have to tell them, well, what are we interested in?

Where are we headed?

What's the roadmap?

And you know, edX edX works like most companies.

It's you know, they're the edX employees and we all talk to each other, but how do we talk beyond ourselves so that we can get those 800 people?

That's that's the big challenge.

And we're entering a whole new phase of that with this split of the acquisition and the new nonprofit.

If you want, you know, someone was asking on Twitter just today about, you know, what are examples of big Django sites, well, Open edX Power means a really good repo.

There's a lot going on there.

There's a you know

And it's easy to find large projects, and it's easy to find good projects.

Finding good large projects is really hard because the large projects have been around for a long time, and so they've just acquired lots of first they've got the archaeol archaeological layers of how best to do it, right?

So way down at the bottom you've got the function views, and then maybe we've got some class-based views, and the settings files, like we said, have been through their evolution, but they've also just accumulated the tech debt and the cruft from having, you know, a hundred people work on it for eight years, and it's just hard to keep everything with one voice and best practices spread across a million lines of code.

And uh

Is that you still feel the same?

It's just that people ask you and you realize there's no solution, so you just have to pick one as a expert.

I that's uh that's how I feel.

Like when people ask me something like I don't know, but then I ask around and you know, Carlton doesn't know, Jeff doesn't know, a couple other people, Adam doesn't know.

It's like it's unknowable or there's no best practice because I just checked.

Well, and and sometimes we've got a joke among architects at edX that the answer we it's always the same answer.

The answer is it depends.

We uh we haven't said it yet.

We always we try really hard not to slip it in.

Right.

Well, it depends.

Are you doing scientific work?

Kanda.

Are you comfortable with the command line?

Well, you might like virtual envrapper.

But otherwise, why don't you just go into PyCharm and pick new project?

You know, it it depends.

Things things are complicated.

And that's the kind of thing that a newcomer they just want it to work.

And it and that's not just because they're a newcomer, it's because it's the part of the thing they're not interested in, right?

I'm not a newcomer, and I I would love to have not have to think about virtual envrapper and just have it be solved for me.

Seems like about once every six months I have to revisit how I get Python installed on my computer and how I get virtual env in those versions of Python and blah blah blah blah blah.

B, if you just need one version, you can just go grab the official installer.

That works fine.

And then if you need multiple versions like a regular person, you can use PyEnv and you know, I guess poetry if you want to get fancy.

But that third case, hopefully you it's not your first time installing Python if you need to have multiple versions to work on stuff.

Well, it depends.

It depends on what what they need.

I mean, I think I mean, so I've been thinking about this for my my Django books because I have a section on you know, how do you get to Django?

It's like, well, I got to slog through Python, and I have switched away from homebrew on Mac and just the Python.org installer actually I think works quite well.

And you know, yeah, there's all these things it doesn't do, like how do you switch Python versions?

It's like, well, there's a whole universe of thought on that, but we're exactly doing clean greenfield stuff, and so it's you know, Django 3.2 and Python 33310 soon, and don't have to worry about it.

Well, yep.

I know we've gone a little over, but I appreciate we had a lot of questions around testing and edX.

And again, you were one of the very first people we wanted to have on this podcast, so I appreciate you taking the time.

You can just you can just this is the the series finale.

Yeah, it's like

I always I always enjoyed doing these, and it's great to have a chance to have a longer discussion.

I know we tweeted each other and maybe even see each other in IRC or Discord sometimes, but uh having an actual discussion with paragraphs and replies and thoughtfulness is great.

Yeah, and I mean I sort of the goal is to lose people.

I was like, good, because I'm not trying to appease you.

That's right.

You're not the audience.

Yeah, that's right.

Bye bye.

Join us next time.

Bye bye.