Transcript: Django Testing and Debugging - Karen Tracey

I'm Will Vincent, joined as ever by Carlton Gibson.

Hi, Carlton.

Hello, Will.

Hello, Karen.

You've been one of the original people I wanted to have come on the show because you've been involved with Django for a long time.

Um and we've emailed about various things, so I'm very excited to have you on the show to share all your knowledge and wisdom with us.

And um, Django's not that old, but it's been around for a while, and there's been sort of these generational things of knowledge and changes around the framework.

So, anyways, we'd love to talk about how you got into Django, obviously what you're doing today.

Um, but maybe we could start with you know your background.

I know you have a formal background and how you got how you slid down the slope of computer science into Django.

Um I started, I mean, I chose computing as a as a focus in college, so I'm more of a traditional came into programming and computing via school.

Um that was a long time ago now.

Um I think I was fortunate in the timing that uh I was getting into choosing a career back when personal computers were just coming out, the internet wasn't a thing yet, uh, and there wasn't a lot of I it was at a time when it seemed like women were getting into all sorts of new things.

You know, women could be doctors and lawyers and all this other thing, and I didn't have any back pressure of, well, this is something women don't get involved in.

I think nowadays it's much more seen as a guy's thing.

I think I was fortunate in the timing that I didn't uh didn't run into that.

And I also came from a family that worked for IBM, both my mother before she started having kids, and my dad worked for IBM.

So it was sort of like uh this is something my family does.

Right.

So I think I was very fortunate there in how I got into it.

And um I went into I went to Notre Dame and that was where my dad had gone, and I went into the College of Engineering.

They didn't have computer engineering, but they had electrical engineering with a focus in computers, and um yeah, that's how I got into computing.

And then there was uh there came a sort of time where it it it switched.

I was wondering if you've if you've got a take on that.

Uh, you know, because it it seems, you know, now with Django Girls and the these initiatives we're trying to bring it back, but it's still very male dominated and I kind of wonder why.

Women were involved, I think.

Yeah, I don't know, I I wasn't I don't I'm not a historian, I don't know the the how it all happened, but if you look at my mother's experience, you know, she stopped working when she started having kids.

So it was more a societal thing of women didn't continue working.

Yeah, okay.

So there was that aspect to it.

And then when personal computers came out, I've read various articles that talk about how the games and the the marketing was all targeted at boys.

Right.

And it it became much more of a this is a a boy thing, and that women didn't girls didn't didn't get attracted to it, didn't get into it.

But I do think there's no reason I don't think there's any reason that the programming aspects I think women are an asset.

But I know that I mean there's like that book and movie Hidden Figures.

I mean, there's female mathematicians, I mean they called them computers, like because they were the computing before they had so there's a long obviously history of that.

Um I always thought maybe it was like once computers started paying well, all the men were just like, well, we'll take those jobs.

Whereas before it was seen as a little more menial, which I think is inaccurate, but it was very, you know, more detail focused.

Um who's I guess who's to say, but it's a sh it's a shame.

Um it doesn't make any sense to me from an objective point of view why that would be the case.

So what was I don't know what what were the languages that you were studying in school, and you know, how did you find your way to the web since you predated the web a little bit?

C is probably the language that I feel most like that's my core language still, even though I haven't used it in decades.

Because I uh I got involved in a research lab at Notre Dame, and that's what led me to to stay there and get a master's and a PhD, and we were doing um research into how mainframes and personal computers were gonna talk and interact with each other.

And we ended up uh that research lab focused on building an operating system, you know, it was microkernel-like, you know, a base set of services onto which you could build an operating system.

And all that was written in assembly and see.

So I had uh four good years of that was all I did was see pretty much.

So um

I the in the early 2000s I wasn't working and I was thinking about uh I should probably go back to getting a real job and having a decent income.

Uh and I should try and see, you know, interview for interviews, make sure you could, you know, solve problems or whatever.

So there's various sites you can go and see, you know, solve this problem, do that, whatever.

And I had programmed in Java and Python by that point.

But when I went to do these little problem solving problems, that like my natural language, my the language I gravitated to do things in, even though I hadn't used it in decades, was C, which I found interesting.

Um I did use

Did you have to do all that?

Well, I did interview a couple of times with Google and those were days of yeah, pretty much whiteboard coding.

Um and I'm very grateful now that I did not get a job with Google because I don't think I would have enjoyed it particularly.

Cactus didn't have that kind of interview.

Okay.

Because we we we we've had them on and

And I didn't fit, so I was in the co-working space.

Um I that was I joined in 2010.

I think they were a couple of years old at that point.

Yeah.

So and the reason I meant uh jumped to your book is because it's massively influential when I for me when I was picking up Django, because I needed I had the sort of introductory material, and then I had you know your testing and debugging book, which was like

No, I mean it was as

influential to you.

All the other

I don't need that anymore.

I've still got your book, and it it's like you know, it it's you know, um the unit testing framework, doc tests, um, the debugger, you know, you know, print logging.

Log logging, you know, is such a hard thing to pick up and such a hard thing to get right in Python.

And then your book was just like, you know, walked through in such a good way.

So, you know, that's one of the reasons why I'm super excited to have you because it's like, wow, Karen, she wrote that book.

Yeah, no, it's so hard to configure properly.

Yeah, uh, so that was I it was after I had finished the book that I started working for Cactus.

Um so I got out of school, I went to work for IBM for about a decade, and I took some time off, and then I got involved in crosswords, and I wrote my own set of utilities for building crossword puzzles.

And uh what got me to Django was I wanted a I wanted to be able to see my database of published puzzles from various um outlets, the New York Times, Los Angeles Times, I forget what all I had, and I wanted web pages where I could see the puzzle.

The the aspect, the the way I could see it in my crossword builder programs was little bits of it.

And I wanted to be able to see like this was the whole puzzle, this was the context in which this entry appeared, so that I could more easily evaluate is the would this work in this puzzle in this context that I was working on, sort of thing.

And I had a friend who suggested Python as a language to learn as was a cool language, and I knew very little about web protocols.

Um back in my IBM days, I knew like I knew we knew the I knew the details, the bits and bytes of the TCP protocol, IP protocol.

That was the level at which I were to the operating system, the middleware, um the web.

I didn't want to learn how to I I needed a framework.

So and I looked around what was there, pylons?

There was a set of them, and there was several to choose from, and I gravitated to Django because of the documentation.

Okay, interesting.

I didn't have to make a lot of decisions, like do I want I forget.

The other ones seemed like they were piece parts, like you had to make decisions, and I didn't want to have to make a lot of decisions about do I want to use this or that?

I just want to like do it for me and let me focus on what I need to do.

Yeah.

So that that's really cool.

Great docs and batteries included, and you know, 15 years into its lifetime, that's still what people say about, you know, we had um some folks on recently saying, you know, what you know, it's the docs.

It's the you know, it's the fact it got it's got everything with you.

So kudos.

I love that.

It's like why should we hire you for Django?

Like as a book author, like I love your chapter titles.

Like you have, you know, when you don't even know what to log using debuggers, like when problems hide, getting more information.

And then what was my favorite one is yeah, when the wheels fall off, like the Django debug page.

I love those things because you know it's it's hard to liven up a technical book.

Um, but anyways, no carlton you were gonna ask a question.

So um, you know, I was self-taught and it it was all print statements, you know.

And I, you know, I did I could reason my way through a program and I did I do print debugging the whole time.

And when I found your book, it was very much like ah bam, this is the tooling that goes that's that's a step beyond.

And so and to learn the debugger, and you know, that that's just like a whole new power tool that you've you've got and it really so it really did um you know, it really did change my programming.

I I wish there was a a modern definitive testing book.

I mean Adam Johnson has his, you know, he's gone a step beyond with you know, speed up your Django tests, but I think there's still a spot for fully testing a Django project.

Um if I did more Django books, I would do one on that.

Um, but it's a big yours is the only book on it.

It's a big topic, yeah.

It's like when it's a big commitment.

Um well, what was I doing then?

I was doing that.

I was I was writing crossword puzzles, I was helping out on Django users, I'd answer a lot of questions, you know, helping out with Django.

Um and I I use I volunteered at Cat Rescue and I have uh we track the cats in a Django database.

Um so I was I wasn't full-time on the book, but I was spending a lot of time writing the book.

Um and I just can't conceive of doing uh writing a book or even revising a book with a full-time job in the rescue.

Um yeah, there's just so much I would like to do that I don't have time for.

Well, I would so then okay, so you're using Django, you're you you you said you get involved in um you know helping on Django users, and then you got involved in helping with the running the project itself and the the board and thing, the DSF and things like that.

Can you talk about that um that part of it?

Uh it was

It was probably not very I I don't know what it's like now because I'm not on the board, but we dealt with um requests for aid.

I'm sure you still do a lot of that of people asking for grants for this, that, and the other thing.

Um, I think we did the around then was when we were working on getting the trademark and agreement finalized.

Um, and we had a lot of issues and back and forth with getting how do we affordably get legal advice.

Uh and accounting accounting help um and funding.

I think I s I susp I think Django is in a better position now than it was then with getting funding.

Um I mean I the servers, I mean how I guess how is how is it being used?

I mean, I guess sponsoring conferences and stuff or

Um we didn't have the fellows then, but there would there was a sense that something was needed to make Django sustainable, you know, a a self-sustaining organization.

Um like the fellows, because as an all-volunteer organization, not funded by a big company, it's like people's situations change, their interests change.

They you can't Django couldn't continue if it was gonna be the focus of you know, if just if the core group of people working on it forever, that just wasn't gonna uh work.

So if you're a volunteer open source contributor, you know, working on your patch, working on the old thing, doing a little bit of you know, new ticket comes in, but it's like this this fire hose of ticket ticket reports, and Maris and I are just spending the whole time, you know, paddling upstream against it, and that's fine because we're we're contracted to do that, but that's not sustainable for a volunteer contributor at all.

Answer questions on the mailing list, look at every ticket that comes in.

Is this valid, not valid, point them in the right direction, that kind of thing.

And yeah, I can imagine it's even worse now than it was then.

You know, why don't we move that to be more accessible, perhaps, because the track's a bit clunky.

But I think the ultimate reason is that the history there, you know, there'll be discussions featuring you and others that really explain why it is that the frameworks the way it is.

And if we just move to a different issue tracker, that knowledge is lost.

And um yeah, unless you can import it.

Git, I think.

Like anyone could create an issue or do something in track and move something along or or change the state or whatever, and we didn't have that capability in Git with GitHub issues, it didn't seem like

And so okay, it's difficult to learn the track, but it is very powerful, you know, and it yeah the triage pipeline.

Yeah, there is why things are the way they are.

Back when back in the day of working for IBM the testing we did was manual like there was a whole test organization.

You'd throw your code over the wall to test and test would come back with with issues.

We didn't do a whole lot of automated testing in my experience.

So my experience really is with Django and actually before I wrote the book I was not big into testing for my own project.

Whereas a lot of people I think come to testing thinking particularly when you're writing a feature and part of it is supposed to be tests like to prove that your your code works and that's not really the goal it's more to make your code maintainable meaning someone else can change it later and your tests will indicate to them whether or not what they've done is having some unexpected side effect of breaking previously held assumptions.

So my my learning of my my learning of Django and writing that book was a lot of learning about how to test or the value of testing.

I was wondering uh what you you know if you I come sometimes shed a tear for the loss of doc tests I wanna you think any way of rescuing those from the from the graveyard

ventually but I did I did find the unit test framework much more powerful I think um and just from a I think from a maintaining the tests point of view I think they're easier unit tests than doc tests.

I think I did get some feedback that I downplayed them a bit too much.

And I

I wonder if there's some sort of literate program in doc test style mashup where you can use the doc tests to ensure that your comments stayed up to date

Uh yeah doc tests I do kind of regret that they're entirely gone.

I mean I think they do have a place they could have a place they don't have a place anymore but there was a crusade to get rid of them at some point and they're all gone now.

Like I think they could have they could have continued I think there was a place for them but okay fair enough

I'm sure you have many to choose from so that's a that's a a big question to there's lots.

What what what's the life of a jobbing developer is Carlton would say

That's the phrase Carlton yeah yeah

Yeah.

in this huge huge wheel and you don't really understand why you're building what you're building or you hear from marketing they need these features and you're like that doesn't make any sense.

I don't understand that like um I really like the direct interaction with clients who are going to be work using um the projects over the years and we've I've worked on smaller bigger um there's one that's we still got that uh it's probably I think it's out on the website as a case study the the survey for schools school surveys that we've um it's

a tool that's used to administer surveys to

used at all depending on the time of year.

So over the summer like it's completely dead you could turn it off and no one would know whereas this time of year when schools are in session there may be surveys running for the state of Illinois, Chicago Public Schools other places um so it's it's that's an interesting project in the the periodicity of it and the having a set time where like you could make massive major changes and and get it all done and not

have to worry about migrating the data because you know the data for the last year gets purged and the new year you start completely fresh.

So it's kind of an interesting different sort of thing from a website where you're it's always up and you need to you need to maintain the data all the time and it's always got traffic and you need to be aware of that.

So there's all sorts of different projects some are small.

ker server for celery rabbit kind of stuff so there's at least I think five minimum servers that's set up with it's on AWS it's got an auto scaling group and it's set up to auto scale based on um CPU of the web servers.

When it is in act when we know it's going to be in active use and the traffic for this particular case of the entire state of Illinois starts surveys, you know, school starts at 8 a.m.

and the class is going to come in, you're gonna get a massive influence.

You're gonna get a massive influx at a particular time.

We did find that the auto-scaling could lag.

So we usually set it up so that instead of students, right?

Like a scheduled action at the beginning of the day to bring up at least six servers, which allows for enough time to get everything up if it needs 24.

Like it doesn't fall off a cliff before it adjusts to the fact that you've got that much servers.

So we do have some timed actions that bring up a minimum that increase the minimum capacity at the beginning of the day.

Yeah, okay.

And then let it then let the auto-scaling based on CPU handle it during the day.

I wondered about um whether your your original work on microkernels and client, you know, and the the the relations between mainframes and and personal computers, whether that ties back into this sort of containerization and Kubernetes things.

Microkernels is a is a hot phrase, right?

And that I wonder if you could have thoughts in that.

I'm sort of half being silly, but

My foc my my my desire to learn more is more on the front end uh that than on the there is a huge amount of Kubernetes interest at at uh cactus and um most of our current projects are deployed using Kubernetes.

This older project is not using Kubernetes.

Um and I don't know that we will ever move it to Kubernetes, but for the most part with Kubernetes, I'm like, okay, all that complexity, y'all figure that out and give me the give me the commands to run or or set up the C.

Okay, super.

So can I ask then which of the bits of front end that you you find most interesting, most exciting, you know, in the in the uh

Okay.

Um I'm excited about we're starting to experiment with HTMX.

We have to ask it in every episode now.

But we just we just yeah, the last couple episodes we spoke to some folks from Berkeley using HTMX and yeah.

So what what what are your experiences with it?

Um I it's interesting to me.

Like it's it's just replaced this part of a page sort of focus.

If you go back to that um that survey thing we wrote, the the survey taking part of it is all a single page.

And when you press next, it just replaces that part, you know, the the question part of the page, you know, submits the answers and replaces that.

It was a requirement of the the original specifications that it not say that you know you're on page, you know the URL not change as you progress through the survey.

You're not supposed to see the page number you're on in reflected in the URL.

So that was sort of dictated by the specifications, but it was it required writing, you know, a little JavaScript to do that.

And it's cool that HTMX would make that a whole lot easier to do.

I like it.

I there were several Django Khan talks about HTMX, it seemed like, or the JavaScript.

I I do like approaches where you can take advantage of all the stuff the Django has, forms, authentication, so that you don't have to redo everything in the front end, do it twice.

Um, so I'm excited about HTMX.

Don't have a whole lot of experience with it yet.

Have done like have added what is it?

Um, the endless scroll pagination where if you reveal the bottom of a list, it just pulls in more.

Infinite scroll.

Um it was very easy to do with HTMX.

One thing for me that it really lit up when I first gave it a go was I was able to be I was able to do things that I haven't sort of been a quote unquote allowed to do for you know, say five, six years.

I haven't been allowed to do it this way, you know, you can't do it that way.

And I was like, but I can do it that way, and it's really speedy and it's just using the basics.

And oh wow.

I kind of had a no wow.

I would like to you to learn more about uh using React for sophisticated user interfaces.

What I'm what I don't know, what I'd really like to figure out is if I can if we can figure out how to use Django forms and not like recreate all of the validation that you can put in a Django form and have to do it in both the front end and the back end.

I don't know if that's going to be possible.

The the tailwind people, the reason I got into it, the tailwind people said if you if you would be using jQuery, then we recommend Alpine.

You know, we you should definitely be so and it's sort of like that that for that level up from HTMX where you really want to keep the state on the on the client.

And one trick I was able to use was to send the data as form data rather than as JSON, because you know you just create a form object and add that you know in JavaScript and then send it and reuse then my form layer in Django, and that was that that turned out to work really well.

Because you mentioned it, but uh I mean you've had what over a hundred published in the New York Times.

I mean it's not just a hobby for you.

I would right.

I think I think I saw that you have over you have you have a lot in the New York Times.

Yeah.

So that was in the time when I didn't have a job.

I was doing crosswords, and I have not that did continue after I had a job.

I had one editor I worked with who had a curated list of uh constructors that he was working with for the I forget now where that was, but he so instead of doing freelance, it was it was scheduled, you have a puzzle due and it will get published kind of thing.

Um I have not been constructing crosswords now for a few years, so that's sort of also fallen by the wayside of life.

One of those things I'd like to get back to sometime.

Because I've seen like some documentaries on crossword puzzles and people make them differently.

Like how do you how do you get all to fit together?

I mean, it is a puzzle, right?

Like what what was your approach?

I focused on so there's two different there's main in American crossword puzzles, there's basically two different kinds.

One has a themed crossword where the long entries all share some theme, and then there's themeless crosswords where the long entries are basically there, there is no overriding theme to the puzzle, but the long entries are going to be interesting.

There's usually fewer words in them, and I focus more on the themeless crossword puzzles.

So there you would start with what are just some interesting words to have in a crossword puzzle.

And interesting usually means unusual letters like J's and Q's and X's and Z.

You know, how many of those weird oddball letters could you pack into a puzzle?

So you start by place you start by with a themeless, you'd start with one or two long entries, maybe that you want to pack into a puzzle and then figure out how to place black squares.

Like if you'd want to think about where you where you're particularly where your special letters are there may be some constraints on where you want want the words to end and start and end.

And then I would focus on the most constrained area of the grid and see what is possible to put in there and pick the best set and then move out to the left and by doing that you constrain you make another area of the grid more constrained and then focus on that.

Are you going to be able to fill that part of it?

The the program and other programs, my program and other programs would do auto fill like here's a here's a set.

This fills it

Yeah.

And it it's not like you can I in some cases you might say well just remove those entries from your word list and just never include them but then it's it's probably okay to have one or two of them in a grid and just having them available makes it more possible to to have something that is a great grid.

So it it's very hard to like you need to I ideally I guess maybe you could automate the whole thing if you could rank your word list or give assignments value to the to the entries but there's just I mean that's a huge task.

No your crosswords that's going to be your bestwords

So they appear in compilations of the New York Times crossword puzzles.

So we we talked about you know um your preference for front end and working with and creating you nice user UIs and that's something that you're interested in designing good UIs and then we we cut into HTMX and React and talking about those things but I wonder if do you think UI design is more of a people you know is it a software you know is it if it's more of a design thing right it's it's not oh if I do it in this right

there's a definitely a technical aspect to it as well um I think all of software is more people oriented than is generally understood.

Even just writing readable and maintainable software like the person writing it today is not going to be the is not necessarily if it's gonna be successful it's not going to be the person maintaining it tomorrow.

So just being aware that the code is not just for the computer ever is an important aspect of programming.

Even if it is you maintaining it.

But like I mean things change.

yeah well

Yeah having just built node so it's sort of a full circle like you know because to your point about you know why is the UI bad I'm most of my friends are non-technical so they'll be like oh this website's terrible why is it terrible and I usually say because they don't care about you you know it's a it's a government website or it's a corporation.

That's not the focus.

able for a user I don't know where that line is though.

Yeah especially

Well I even notice it myself when I'm using a website and it it doesn't do what I want I I don't even bother to read the error message make it go away.

usly were like anything we can do to break bring down the time and they brought it down from I think like 30 minutes to five minutes for these different sections and it was just just put it in front of people and just see where they get stuck and you know sit back and shut up.

And that's kind of what it what it takes right even at a startup like startups I've worked at you you need to spend that time to go and watch real world users using it because generally they'll get stuck on things you didn't think of right like we would always want to show a new feature at this education startup was that we'd want to like go to a classroom every week when in a classroom we'd want to show off this new feature and nine times out of ten something else broke or was confusing and we're like oh we need to focus on that which we never thought of but but it's you know putting in the time and it can feel like wasted time when you have

a long queue of features and bugs but ultimately of course you know sitting with users and making like senior people sit with users not just people who are told what to do make like the product manager make like the CEO sit there and watch a user struggle through generally kind of clicks whatever you've been someone's been saying about the UI.

Yeah.

Which I jokingly but also seriously say to people it's like you know right a consumer or a uh enterprise product you will hunt for that button because there's money involved the consumer product if it's not right there you won't so something that's free the UI is always going to be better than I don't know pick a company to rag on into it.

Like whenever there's a long chain of of things in between the end user and the person writing the code, I fear there's a there's a loss of context and understanding around what what is required, what is needed, what would make it good.

Yeah but no more

You know it's we got the database working and we got it served we got a form up the form just about takes the data the unit test passed oh can I actually make the form nice no no no move on the pro no and that kind of really bare bones development environment is quite frequent I think.

And if it can do that task then

it's done.

So it's like love like bingo between us to keep it interesting Carlton right

So I think that's why I'm like well yeah anyways well Karen thank you so much for taking the time to come on I'm really wanted to have you on dabble too much.

I mean the specifics and the details and the error message may be different but the I think it tried to focus on doing things in small steps and validating what you got and understanding when like I think one of them is breaking things on purpose like understand what it's gonna do when things don't work you know so that you can understand what it's saying when things don't work and you don't know why it doesn't work.

So I think there's still stuff in there that's valid.

No I don't know how

So that but the the underlying principles there still account but the rest of it is still kind of basically up to date.

And the same with your book is you know like you you you explain the um the debugging page like the you know the chat and that's still exactly the same.

That's you know it's hardly changed in

Like there's a there's a tendency to see the top and say like okay it's broken and hardly even read the error message and just go back to the code and I was like you can actually dig in here and see why it's broken and that can that can make things a lot better in fixing whatever the underlying problem is like you don't have to go back to the code and just like no it's broken somewhere you can see it's broken in a specific place because of these values and variables and stuff.

Yeah.

And it's like this it's just gold dust to be honest.

It's such a good book um and it's really stood the test of time so I I can I can keep um flushing about it forever.

I love it.

It will