Transcript: Django Co-Creator - Simon Willison (Ep 21 Replay)

I'm Carlton Gibson.

I'm joined as ever by Will Vincent.

Hi, Carlton.

Hi, Simon.

Simon, you well, you're you you're one of the three hands that created Django, right?

That's that's your sort of opening in the Django community.

That's your so for us, we're just like, wow, amazing, you you created.

Can you tell us something about let's start with tell us about the birth of Django?

Come on, that's a

So this is what nearly 15 years ago now.

Um this was um back when I was at I was at university and in England, and my university course had an option of a year in industry, so you can go out and spend a year working for someone and then go back and finish your degree.

And it means you can get student visas for things and so forth.

And um, I had a blog, I was writing about web development.

There weren't that many of us doing this back in what 2002.

Um Adrian Holivati also had a blog, and he was writing about web development for this little newspaper in Kansas, and he posted a job ad saying, Hey, I need somebody to come and help me build newspapers and cat newspaper websites, and it coincided with this opportunity for a year in industry.

So I thought, well, that could be kind of fun, you know.

I could I could go to Kansas and spend a year there and and work with this guy who I'd been been following online for a while.

And so I got in touch and said, Hey, would this work as essentially a sort of paid internship?

He said, Yeah, absolutely, we can do that.

And um I headed out there and I spent um I spent most of a year out there in Lawrence in the tiny little town of Lawrence, Kansas, working on what turned out to be Django, though at the time we just thought it was called the CMS.

Right.

It was like the the code we were writing for the newspaper websites.

What was that transition?

Um I mean it was 2002.

We'd both hit the limits of what we felt was productive to do with PHP.

This was back when PHP had only just got classes and things.

Yeah, very

Very early days.

And we'd both, we Adrian and I were both really into Python.

We'd been following the work of Mark Pilgrim, who'd been writing all sorts of great things about Python.

We wanted to use Python for websites, but the uh back then the answer for how do you use Python on the web was was quite flaky.

You know, there were a few very early web frameworks.

Zope was the sort of majority mind share for Python on the web, and it didn't really fit the way we thought about how we wanted to build websites.

You know, we were into things like really cleanly designed URLs and using CSS to separate your presentation from your content and all of these kinds of things.

Um we started poking around and figuring, okay, can we use Python to build these websites?

And actually the real inspiration for Django was that we were looking at using ModPython, the Apache module for building the for building things.

And we realized that um there weren't that many people using it, and we were a little bit nervous that Mod Python might turn out to be the wrong direction.

So we thought, okay, well, what we can do is if we build a very thin abstraction layer between our code and modpython, then if it turns out modpython was the wrong choice, we're we're not in, we we can we can switch to something else and use it.

And so that's where Django came from.

We started out with the request, the request object, the response object, the way URL routing works, those kinds of things.

Because essentially it was to it was an insurance policy in case mod in case modpython turned out not to work out.

Modpython worked out great for years and years and years, um, but that was that was really why we why we why our CMS had its own had its own thin abstraction layer.

Oh wow.

It's an indirection layer on ModPython originally.

And I should say a little bit more about the Lawrence Journal World back then.

So this was a tiny little newspaper in a tiny little town, but it was very well resourced because the um the family who owned the newspaper had become independently wealthy off, I believe they were laying fiber optic cable around the Midwest back in the 70s when everyone thought that was a crazy thing to do, and then they sold their network to Comcast or somebody.

So they had a lot of money, they were very heavily invested in the local community, and they were running essentially this sort of media empire that spanned about a hundred miles.

So they had the local newspaper, they had, I think they did they have radio stations, they had the local TV station because they'd run, you know, they were they were running a broadband provider for this little town.

So Lawrence, Kansas, back in 2002, had fantastic broadband across the whole town, which meant that their newspaper website could experiment with things like online video.

No other local newspaper was doing online video because who's going to be able to watch it?

Um, and so that was really exciting.

And um basically they were trying to, you know, because they had all of this investment, they had this really well-staffed newsroom, they were really trying to push the edges on what you could do with web stuff as well.

And the chap running the online department was uh a guy called Rob Curley, and he was very much of the opinion that you should go all out on everything that you were doing.

So a lot of the fun stuff we did with Django was sort of inspired by Rob coming up with ridiculous ideas for things to do with the website.

My favorite example has always been that um it was um it's kids' little league season, um so it's when kids play softball.

And you know, in England this is not a big deal.

In in small town America, this is absolutely the most important thing.

So um, so um Rob said to us he wanted us to take these little kids' softball teams and treat them like they're the New York Yankees, right?

So we'd have a website with player profiles and team alerts and schedules for the league.

And um, he sent two of the interns out to take 360-degree videos using what QuickTime VR back then of every pitch, every every baseball pitch in the whole city, so that you'd have those on the website and you could get like a virtual view of what this these kids' softball.

Um, and we did SMS alerts so that parents could sign up and get an SMS about what their kids were up to and all of this kind of stuff.

And um, I've got a lot of this went into the newspaper, you know, because it turns out if you print a supplement in the paper with a bunch of photos of kids playing softball, everyone in town who knows any of those children buys a copy.

Yeah, and all of the local businesses want to be seen sponsoring the kids' softball and stuff.

So it's a big commercial venture.

Um, but we so we we had and we basically had two weeks to build this website.

So this was sort of the proving ground for Django.

This was what it had all been working up for.

Like we had the we we had, well, it wasn't the ORM back then, but we had code generation for our SQL bits, and it allowed us to get this thing out.

And that's sort of that's when we realized that what we were building was really working, you know, that we could take on these ridiculously ambitious projects with a very small team in a very small amount of time.

And yeah, it's I mean, if thinking back at similar things I was doing with P or trying to do with PHP, it was just crazy.

It was like, you know, the same sort of time scale, same sort of epoch.

It was it was difficult.

You were building a lot of things from scratch and there weren't the frameworks to support you, and it was tricky.

They built Basecamp and then they pulled Ruby on Rails out and made that open source.

Django was very was slightly different in that we already had this website called Lawrence.com, which was the local entertainment's website for the town of Lawrence, Kansas.

So it had all of the bands that were playing, it had articles, it had restaurant reviews, all of that kind of stuff.

And this is the Mammoth website that Adrian had been building in PHP, and it was an extraordinary thing, absolutely amazing website.

And the goal with Django was always to grow the framework to the point where it could run Lawrence.com.

So Rails was extracted from Basecamp.

Django was sort of evolved to fit a site that we already existed but that we wanted to replace.

And we did manage to replace that um, I think just before we we launched Lawrence.com on on what became Django just just but just shortly before I ended my um my paid internship there.

I'm I I can't remember the exact details, but I believe, yeah, the sites framework was in there right from the very start because we knew we were gonna have one CMS that was supporting multiple different websites on the front.

Right, okay.

So, yeah, there's a there's a lot of stuff in Django where knowing its history of of how it came out of this this tiny little newspaper group that was that was punching well above its weight sort of helps helps helps explain some of those those different features.

So then you it's a pretty productive uh work placement, I would say, in the history of work placements.

Um but then you went back and you had what another year or a year of the yeah, I

And actually, um I one of I think the most impactful things I did at the newspaper is I helped to hire Jacob Kaplan Moss as essentially my replacement.

So we only overlapped for about a month at the end of my my time there.

Oh, okay.

Um but but but you know, Jacob, Jacob and Adrian, once I'd left, absolutely pushed forward and and and got the thing open sourced.

And the open sourcing was um, they had to make the argument to this very sort of traditional newspaper business that they should release this code into the wild.

And they went in there with a bunch of arguments.

I believe one of the biggest arguments was look at what happened with Ruby on Rails, right?

Ruby on Rails and Django were developed privately at about the same time.

Ruby on Rails was open sourced about the type, just after I'd finished at the newspaper, and it took off like a like a rocket.

Um, and so they went to the newspaper management and said, look at this thing that has happened with this.

We've got something that could also be really successful, that make it easier to hire people, we'd have other people doing development.

But um impressively, the the argument that really convinced the newspaper was we have been using open source software for years to build this business, right?

Everything like mod Python is open source, we use Linux servers, all of that kind of stuff.

This is our opportunity to give back.

And that was the argument that that swayed them.

Um from what I've heard, that was uh it was a no, this is absolutely makes sense that we'd we'd we'd we'd

That's yeah, that's fantastic.

I mean, I was just at PyCon for the first time.

I I saw you were at PyCon and I had flying back for a number of people on the plane with me, and a bunch of them were they were kind of big uh I don't know, industrial companies, and they were saying they don't really know Python, but they were saying, Yeah, we're all switching to Python because it's the only way to hire people, because they're switching over from Java solely because they can't hire any anyone under 40 to work in Java.

So they're kind of like dragging and screaming a little bit in.

So it's interesting that the recruiting thing wasn't the one that hit it, it was the more altruistic, like and again.

So um, but um, which I guess maybe that, but but um they actually the newspaper span off a separate company for several years that was commercializing a CMS built on top of Django, and so for quite a while this was doing very well amongst lots of other newspapers, um, because they were uh and and at that point, you know, by this commercial CMS, you can hire developers because it's built on top of Django, and so it's open source, and so there's a there's a marketplace for that.

So a lot of those commercial factors did pay off really well, but it was the um it was from what I've heard, again, this was after I'd left the newspaper, it was the altruistic argument that was the convincer for getting it open sourced in the first place.

I mean, what what kind of what was that like stepping back into a formal academic setting after having this such a prolific uh work experience?

I mean, one of the things I learned working on Django is um, you know, people who have computer science degrees often say, you know, I don't use that stuff very often.

The Django template system comes from me having done compilers 101 at university.

I had not yet done compilers 102, so so it was very much a sort of loose sort of lexer and interpreter kind of thing that was going on.

But um, you know, um I I I think, and and it was interesting as well having bit sort of being able to step back.

I actually wrote a project in Ruby on Rails um when the moment it came out because I was like, this is the thing I had at the newspaper, right?

This is brilliant.

Somebody's open sourced a productive framework wallet that I can get on with.

Because I, you know, I I didn't know if the um if if what became Django would would be open sourced after that.

So I I I did dabble around with Ruby on Rails, and it was also just interesting seeing that explosion of interest in Django and all of the like um contributions and the creativity that came out of that.

Um so I don't feel I didn't feel yeah, it's interesting.

I've never really thought about the um that transition back to academia.

Again, it was so long ago, and I was I was so green in my career.

You know, I'd essentially had I had a bit of a programming job before university, which was working in the first dot-com boom in like 1999 to 2000.

I worked for a um uh an online gaming company down in London building uh building their file downloads website.

Um that was super fun, but uh then everything completely collapsed.

The company that I believe they lost like 30 million pounds or something.

So it was uh proper.com.

Yeah, oh absolutely, it was very exciting.

Um but yeah, so I I was quite happy to be safe in academia for a couple of years waiting for the um waiting for the dot-com um explosion to to boil over again.

Is this the same uh SimonWillison.net that you have today?

It's been through a couple of different um web addresses, but but yeah, that's my blog is

I love it.

People should take a look at it.

I love it.

Yeah, I um I did a big upgrade to it a couple of years ago.

I moved it from being 800 pixels wide by default to 1024 pixels wide because it was looking a little bit narrow on modern screens.

Um but yeah, basically that blog's been going since 2002.

Because you you later worked with him, I think, right, in a capacity.

Yeah.

I mean, no, I so I started the blog when I was 21?

21, yeah.

Yeah, I worked, um I contracted with Torchbox for for about six months at one point.

Um and that's actually, I think that's where I met Andrew Godwin.

He was at Torchbox at the time.

Um Andrew and I, back at for the first ever DjangoCon, Andrew and I were working on rival migration systems for Django.

I had this thing called the D migrations, and he had, I think he was calling it South back then.

Um and we managed to set it up so that the first Django Con we had a um a panel about database migrations.

Um so it was myself, Andrew, and I think Russell and

Um Okay.

I saw I saw a talk he uh Russell gave where he was saying that yeah, just that, and then uh what became South came in and he was like, Oh yeah, you can you can take it.

Okay.

It's like celebrity death match here.

Um I mean, what are some of the highlights?

I mean, we definitely want to talk about Lanyard, Eventbrite.

You just gave some fantastic talks on SQLite, and then you have this journalism fellowship coming up.

So where do you want to start with all of that?

What are you most of the things?

Um, good, good.

So you gave a talk at PyCon, which we'll link to Carlton was especially Carlton's very much Team SQLite, but you've been using that in a lot of capacities around um data set, right?

So maybe you can talk about dataset as well.

So after the Lawrence Journal World, I went back to university.

After university, I bounced around doing sort of contracting and freelancing for a bunch of different places.

Um but then I ended up at the Guardian newspaper doing data journalism projects.

So data journalism, I realized recently not everyone knows what data journalism is, and it's actually a little bit difficult for me to define it.

But essentially, it's when, as a programmer, you get to work with journalists and build tools and do analysis that helps them find the stories in amongst the data.

And the the sort of most obvious example is anytime you see an infographic in a newspaper like a graph or a chart or a map, somebody had to gather the data that, and somebody had to often write the code to help pull those things together.

Um, you get to work directly with journalists breaking helping to break news stories.

And so I worked at The Guardian for a couple of years, and um one of the things that happened there is uh I was working with this chap called Simon Rogers, who was he was the nerdy journalist who gathered the data for the infographics the first time I met him.

And so the other journalists didn't really understand him.

They're like, yeah, he's he's in the newsroom, but he's always got Excel open, and he's he gets really excited about about like getting data from places.

And um, I had this, so what when I first met him, I was like, wow, what do you do with all of this data?

He goes, Oh, it's all on my desktop.

And he pointed desktop under his desk, which it turned out had hundreds of beautifully crafted Excel spreadsheets about every fact about the world you could possibly imagine.

And I'm like, okay, we need to intro we need to introduce you to the to the web team up on the up on the floor above.

And so we started brainstorming, okay, how can we release some of this data?

What's the best way for us to publish this?

In the end, we went with the simplest thing that could possibly work, which is a blog, because The Guardian was very good at running blogs.

And so we called it the data blog.

And every time we had data behind a story, we'd put something up on the data blog about it.

And then the data itself we published mainly using Google spreadsheets.

Because this was back in 2000.

You can post a link to it, other people can then pull the data out and start doing things with it.

And this worked really well.

We had a Flickr community of people building their own visualizations against the data.

There was um it was it was pretty sort of uh revolutionary at the time to sort of tr have this mentality of no, you publish the story, but you also publish as much of the data as you can publish as well.

Most of the time you click file export as and you can get the data out that way as well.

But I was always a little bit frustrated about this because while it does have APIs of sorts, they're not the most convenient things to use.

You know, I don't know many engineers who are thrilled at the chance to integrate with it to pull data out of a Google spreadsheet.

So I always felt there should be a better way of doing this.

I actually mucked around at The Guardian with um with CouchDB.

I thought maybe CouchDB could be a way to publish these in a in a more reusable format.

Um and um so anyway, a couple of years ago I was thinking about um I was thinking about Docker containers and some of these new hosting providers, like um ZeitNow was one that were doing this at the time, which can host your Docker container for you.

And you know, these they're essentially the sort of serverless Docker model.

But the problem with all of these hosting providers is they don't let you do any writes, right?

You can just serve static files and you're expected to pay for a hosted database somewhere else.

And I thought, hang on a second, if you're dealing with read-only data, the fact that you can't have writable data and you don't have a database doesn't matter anymore because you've got SQLite.

So you can bundle a bunch of data into a SQLite file, stick it, literally, bake it into your Docker container, ship it somewhere, and you've built a extremely fast, extremely powerful dynamic web application that just doesn't accept any rights at all.

The data is part of the deployment.

And I started playing with it around with this idea, and it um quickly turned into this open source project I've been running for a year and a half now called Dataset.

It's named after the Commodore 64 cassette player, which, as far as I can tell, their copyright on that term expired about 15 years ago.

So I believe the name is up for grabs.

I hope that's true.

Um but dataset is so it's a it's a couple of things.

It's a web application that sits on top of a SQLite database and exposes the whole thing.

So you get a table view where you can look at the tables and you can filter them and all of that sort of stuff.

But then more importantly, you can execute SQL queries by typing them into a form and clicking a button.

And because I've opened the database file in immutable mode or in read-only mode, you can't harm the database doing this.

You know, like allowing arbitrary SQL queries is that's the definition of a SQL injection attack.

So most web applications will avoid that like the plague.

Turns out with SQLite, and then um if you have a few safety precautions around it, it's actually okay.

So SQL becomes your API language, which is interesting.

Because Dataset also offers a JSON API.

So anything that you can see, you can get out as JSON.

It can export as CSV and stuff as well.

And it means that you can take any database you like and dump a JSON API on the top of it, which accepts SQL over a query string as the query language.

And I've been I've been cracking jokes about GraphQL on this flux because people are like, that sounds terrible.

I'm like, yeah, well, everyone's excited about GraphQL.

This is just SQL is GraphQL from the 1970s, and turns out it works really well.

So that was the sort of initial idea was publish data in a way that it's super, super quick and cheap to get it published, and other people can make any API shape query that they like like at it.

Then the other part of the ecosystem we've been building is the ways of publishing that data.

So out of the box, dataset has a command line tool for publishing.

You can say dataset space publish space Heroku space my database dot db, and you hit enter and it up and it creates an app on Heroku and it uploads your database to it, and you're done.

It spits out a URL.

It's got um two other providers, there's ZeitNow on their V1 platform, which they don't let people sign up for anymore, which is a bit of a shame.

And then Google Cloud Run, I now have support with, which only got announced back in, it got launched back in April, and somebody sent me a pull request for dataset which implemented Google Cloud Run.

So the joys of open source, I didn't even have to um I didn't even have to actually implement that.

It's tricky.

So zeit version two no longer uses Docker, it's all about um Lambda functions.

And um that's tricky for dataset because dataset was sort of built around the idea that you do things with Docker.

I think I can get it running on Zeit v2 now that Python.

The other problem is that AWS Lambda, their version of Python didn't use to package SQLite because they assumed nobody would want to use SQLite in a serverless environment.

Um their Python 3.7 does have SQLite support, so it is feasible, but there's a whole bunch of things I need to unspool and get working to have dataset fit into those Lambda environments.

And so I want to get it done.

I think actually the biggest project everyone right now is porting dataset to ASGI.

We pronounce it ASCII, right?

ASCII ASCII, yeah.

I mean pronouncing ASCII is always difficult.

But it was WISGI, so it's ASGI.

So um that's the so so once I get dataset ported to ASCII, there are various mechanisms for getting ASCII running on Zeit now version two that I want to um have a play with.

And um when I teach SQL to beginners, you know, using a data set to learn how to do basic SQL would be the perfect playground.

It's not just made up stuff, it's just and it gets read-only, and so I'm gonna start using that because I've seen that there's there's a ton of amazing uh open source data sets, but it's kind of built for playing and exploring just SQL itself.

You know, I did I was not intentionally building a SQL learning tool, but actually no, it totally works for that.

Um I'm really excited to see that kind of thing, thing, thing, um, being explored more.

Because yeah, um a SQLite's um variant of SQL is a very it's almost entirely based on the SQL standard.

So it is a very good tool for learning SQL.

And these days it's got like common table expressions and window functions just made it into SQLite about six months ago.

So there's a whole bunch of more advanced SQL stuff that you can start playing with there.

But um yeah, I should look into because there's ways to load in um database files.

Um that would be really nifty to not actually have to run SQLite itself.

Yes.

Yes, that thing's really interesting.

Um for you know there's for Python 2, there's Python, Python, I never say it right.

Um, you know, there are these JavaScript implement implementations as well that are most of the way there, and for um certainly for a learning perspective, it just solves a lot of problems.

I saw there's a project which does the entire Python data stacks, so um Jupyter Notebooks and um NumPy and everything, all compiled down to WebAssembly and running in a browser, which is an astonishing achievement.

I didn't realize it was that advanced.

I mean, it's um

And now I'm seeing all kinds of really interesting practical applications of it.

Um, my favorite example is something that the Google Chrome Develop um DevTools uh develop uh the Chrome Developer Evangelism team put out called Squoosh, squoosh.app, which um it's basically it's an image compression website.

So you drag a JPEG on and it gives you a better, like more compressed version of that JPEG.

And it does PNGs and GIFs and JPEGs.

But the real magic is the way it works is they took the best in class C libraries for JPEG compression and PNGs, so OptiPNG and all of these different things, compiled them to WebAssembly, and they run them in your in your browser.

So now your browser has the best in class implementations of compression for like three or four different formats.

You literally drag an image onto this onto this page, it shows you a preview of before and after with a little slide that you can slide back and forth.

The interface is brilliant, but the fact that it's running, it's it's it's doing the best in class compression algorithms for all of these formats by running them in WebAssembly is I thought absolutely fascinating.

Yeah, and it's I mean it can

But if that changes and if WebAssembly can allow that to change, then it's the future.

So you can download these.

So somebody compiles something for a browser, you can download it onto your computer, you can import it into a Python process and make calls to it over the over this interface, which is really exciting.

So now I can now, I haven't done it yet, but I could potentially run SQLite, the WebAssembly version, inside of Python without having to compile extensions and all of that kind of thing.

So I think that's really cool.

And then fastly, the CDN have been looking at getting WebAssembly running, sandboxed on their CDN edges.

So you can run a program in anything that compiles to WebAssembly, compile it, deploy it to what 50 points of presence around the world, and now you've got sandboxed, incredibly fast, like um stateless code running in on the CDN edge, which again, that's that's revolutionary.

There are some very exciting things we can build with that.

That's really cool.

And how does this tie into um you've got a journalism fellowship, right, at Stanford or something like that?

What's what's the story there?

The JSK.

And basically it's a um it's a fellowship program where they get in 19 people a year, they pay you to spend a year at Stanford working on projects essentially to advance the cause of journalism.

And um, I applied for this essentially with the dataset project, right?

The idea, because the idea behind dataset is help newspapers publish the data behind their stories, but it's also help provide like much more powerful local analysis and visualization tools that journalists can use to analyze the data that's coming back from all of these different places.

And really what it comes down to is if you look at newspapers like the LA Times, the New York Times, the Washington Post, they do incredible database reporting because they can afford programmers, right?

They have the funds to hire teams of experts who can work on these things.

If you're a smaller local newspaper, there is no, you're having enough trouble staying afloat as it is, you're not going to be hiring programmers.

So my pitch for the fellowship was how can we build an open source ecosystem of tools that help local newspapers deliver the same kind of reporting, take on a similar class of projects to these much larger publications who can afford the programmers.

And um, that doesn't just mean dataset.

Like, dataset is my first foray into this world, but um I plan to spend the first couple of months of the fellowship talking to as many journalists and as many local news organizations as possible, figuring out, okay, what are the tools that if we were to provide these tools, it would give you a huge, like, uh allow you to punch above your weight and give you a big boost in terms of covering some of these stories.

Um that you've pulled out.

Um, if you're gonna do that, it's SQLite files.

You need to get a way of getting data into SQLite in the first place.

And so I've been experimenting with a bunch of different ways of doing that.

The um the first tool I built was called CSVs to SQLite, and it takes CSV files and it turns them into SQLite tables.

Um, because you know, everyone publishes their data as CSV.

CSV has its flaws as a format, but it is universally understood, and there's a huge amount of data that's being published in that way.

So CSVs to SQLite can do a bunch of interesting things.

One of my favorite features of it is if you point it at a directory, just say CSVs to name of directory and then filename.db, it will recursively loop through the directory structure finding every CSV file and convert those into tables.

So you can run it against a nested folder of 400 CSV files, and you'll get a database with 400 tables in it.

And that works really well.

Yeah, so the big demo I've got is um 538, the blog, who do a lot of data reporting about sports and journalism and all sorts of things.

They publish all of the data behind their stories in a big GitHub repository with like 400 CSV files in it.

I've got a Travis CI job which once every 24 hours grabs the entire lot, just pulls the full repo, converts it all into uh a SQLite database, and publishes that with dataset.

So that's always my go-to demo for what dataset is take a look at 538.datasets with an s on the end.com, and you can start exploring all sorts of things.

They've got um data.

My my favorite dataset they have is um that TV show where that um chap paints um paints uh mountains and trees and lakes.

Uh Bob Ross, Bob Ross joy of painting, they've got every episode of his show with Ditty Paint a Cloud, Ditty Paint a Mountain.

And uh so you can run a query that says give me back every episode where he painted at least one cloud and at least one mountain and at least one hut.

And that's just that's that's nice, you know.

Um and now that's available as a JSON API.

Should you want to build the ultimate JSON exploration JavaScript exploration of um of the Bob Ross series?

So if for anyone who hasn't played with observable yet, it's like Jupyter Notebooks um do Python, observable does JavaScript, um, with a couple of differences.

It's only available as a hosted platform.

So you go to observable HQ and you can start playing.

Um but it's fully reactive.

Like the one confusing thing about Jupyter is it runs the cells in the order that you executed them.

So it's easy to end up with a notebook where everything's sort of jumbled together.

Observable, any cell that depends on another cell will automatically re-execute when the first cell is changed.

And it means it's kind of like an Excel spreadsheet in that regard.

And it means you can build really interesting interactive tools where you muck around with the slider at the top of the page and it's updating a map at the bottom of the page, those kinds of things.

So it's by Mike Bostock, is one of the three developers behind it who invented D3.

So it's very

So it's CoffeeScript.

I think he he worked on Mapbox GL and things.

So yeah, the team behind it were astonishingly um like like that well suited to building this project.

So I've done a few bits at bits with observable where I use dataset as the back end API, pull them to an observable notebook and do visualizations there.

That works really well.

And I think there's uh a lot of potential for that kind of stuff.

But then the other thing I'm trying to do with dataset is it has a plugins mechanism.

And so the idea is I want an ecosystem of plugins that can do any kind of visualization you manage.

At the moment, the two best visualization plugins are um, I mean, almost all the plugins are by me at the moment.

You know, when somebody else writes a plugin, that's a huge like win when that happens.

Um, but I've got a plugin called dataset cluster map.

And what that does is it looks at your data, and if it finds a latitude and longitude column, it draws a map and it loads the pointers on and does that um, you know, the clustering thing where you'll get a five that you can click to zoom in and see all five points.

That that kind of thing.

And um that's using a leaflet JavaScript library for the for the marker clustering.

But it works amazingly well.

It turns out in 2019, a browser will happily display 200,000 points on a map using the right.

So so I've got a 200,000 point data set of every tree in San Francisco, which the Oh

And so this plugin will port draw 200,000 trees on a map, and then you can zoom in and see all of the individual trees and their species and when they were planted and so forth.

Um then I've got another plugin called Dataset Vega, which uses the Vega visualization library to essentially do bar charts and line charts and scatter plots.

Um and I'm trying to get it so that these things kick in automatically.

So the idea is they'll analyze your data and go, oh, it looks like this could be graphed against this.

Here's a quick preview of a graph, click here to expand it.

Um that's cut, but that that that's it's really fun.

It's also a way for me to muck around with some of the more advanced JavaScript visualization things that are going on.

And again, I'm hoping I can convince other people to start building plugins for all of the different types of visualizations people might want to do.

I see uh you know, it comes up.

No, I don't I see you tweeting about it, but I see other people tweeting about it as well.

You've been putting on Glitch recently, which is is it called Glitch?

The um the Trello company that um what are they called?

So they're responsible for, I mean, Stack Overflow was a partnership with with people from Fog Creek.

Um, and then there was um then there was Trello, which they sold to it last year, and their new focus is Glitch, which is this phenomenal learning environment for programming.

Um, you know how the absolute worst thing about learning to program is setting up your development environment.

And I've been doing this for 20 years, and I still have trouble setting up a development environment for anything that's even slightly different from what I normally work on.

So Glitch's thing is it's entirely browser-based.

Um, you literally click a button on glitch.com and it gives you an environment with a running web server, with an editor built into your browser where you can edit the code.

It's got Git running, but you don't have to know about it, so it just constantly snapshots where you've got to.

So I can go to any project on Glitch and I can click the remix button, and now I've got my own copy of that project, and I can start working around with it.

So as a learning environment, as in the community of people, it's it's phenomenal.

And for the thing I got working with dataset a few weeks ago is um I've got an example, um, I've got an example project on Glitch, which you can remix.

It's called Dataset Hyphen CSVs.

You can remix it, you drag a CSV file onto your browser, and it will convert that CSV file to SQLite and serve it instantly through Dataset.

So it literally is a drag and drop to create an API for your data and uh this this interface to explore your data.

And it's I'm so excited about this thing.

Like as a demo, I've been gleefully doing this demo where I get people to run the demo themselves, and I don't even touch their laptop.

I'm like, go to this URL, download this file of Seattle public art, drag it onto your browser.

Look, you made a website, you made a map of this thing.

This this it's such a great demo.

Um hang on.

Sorry to interrupt you.

This must mean you've got Python working in Glitch because when I looked at it originally, it was JavaScript only.

You know, they they made the reasonable decision to focus on one language for all of their material around it.

Um, but it's actually running a Docker container with an Ubuntu um with Ubuntu in it.

I think I'm pretty sure it's it's definitely containers.

I think it's Docker containers.

And so they out of the box give you um they give you Python 2 and Python 3.

It's running Python 3.5, they don't have anything more recent than that just yet.

Um they actually have PHP on there as well.

If you can, you can, there are examples out there for how to run PHP on glitch.

And so you can install anything by running pip3 install something dash dash user.

You have to use the dash dash user option or it doesn't work.

And they have this configuration file that lets you say, forget about the node.js thing, I want you to run this process and bind it to port 3000 and then route traffic to it.

And so actually getting, once you know this, and it took quite a bit of digging around in their forums to where people had reverse engineered it and figured it out, you can run any Python thing on there that you like, um, which is really exciting.

So getting dataset on there was way easier than I thought it was going to be once I found the relevant forum posts.

I've got a couple of I've got a dataset basic one, and then I've got this magic dataset one that does the CSV conversion and stuff.

And yeah, and it's just a case of clicking remix.

You don't even have to log in to Glitch to play with it.

You can remix any project as an anonymous user, and they'll delete it five days later.

But it's that's fine, you know, it's enough for you to click remix on something and start mucking around and get a feel for how it works.

So, yeah, it's my go-to demo for dataset now.

Um I the dataset docs now recommend that people use glitch as the first place to start playing with it.

But would you deploy an actual application on Glitch?

They must have millions of apps running right now.

And so the first hit, you have to you sit through a loading screen while the app wakes up.

And they also limit you to 4,000 requests an hour, which is about one request a second, um, which is fine for just you, but if you put it on a high traffic website, it would fall apart.

I'm pretty sure they're going to introduce a commercial pay some money and now your app won't go to sleep.

My best guess is it's going to be they also um they have a

Um so they've got a Teams product which is currently free, but presumably they'll start charging for that.

And again, I'm assuming that they'll start charging you for permanently keeping your projects online.

So yeah, at the moment, I wouldn't recommend it for more than sort of small side projects and things where you're where you're where you're playing with it, but I'm very I'm very much looking forward to them having a enter your credit card details and and keep your thing running option.

Because that's the point when I can tell journalists that they should use it.

You know, at the moment, I can't tell a newspaper to host on glitch if their thing's gonna go offline if they get a spike of traffic.

But um

So you know, we can talk about serverless, but getting something running on Lambda is well, non-trivial.

Right.

You know, provisioning a VM is non-trivial, deploying Docker containers is non-trivial.

Yep.

If you've got if something like that would be amazing.

The way that they've built it, the way it works is absolutely fascinating.

And they've done such a great job of having this community around it of people who are teaching each other to program.

So yeah, that's that's been so that was a that was, I think, a um I feel like dataset itself has had a bit of a tipping project within the last month, partly because I got it running on glitch, so I had a much more compelling demo.

And also the talk I gave at PyCon, um, like the feedback I got after that talk was a bunch of people who I really respect who've been aware of dataset, but you know, it was Simon's side project for a year and a half, and now it feels like it's okay, this thing's sticking around.

You know, this is not a this this is this is something that we can trust to keep on developing.

And I've got the the fellowship as well, so that'll give me a solid 10 months of working on it.

But yeah, it feels like I'm now getting serious um attention from people who previously were aware of it and it was on their radar, but it wasn't something they were going to to really commit effort to exploring.

So I'm curious, and I know you had founded uh Lanyard, which was acquired.

Um I'd love for you to maybe speak briefly about that, and then what you know, what is it like working on a Django sc uh site at scale?

Because there's still sometimes a perception that Django doesn't scale despite Eventbrite Instagram.

So what does it look like, you know, actually working on a large Django site?

Okay, so I can give you the the I'll give you the very quick startup story.

Um I was a Guardian, 2009-2010.

Um uh I married my wife Natalie.

Um she married me, more to the point, but uh, we decided that we wanted to go off on honeymoon for as long as we possibly could.

Um so you know, quit our jobs, give up our apartment, just travel the world, take laptops with us and do the and sort of work on freelancing projects and stuff to try and keep us going for as long as possible.

And so we did this, and we managed six months of traveling, um, which is great.

You know, that's a pretty fantastic honeymoon.

We were traveling through mostly through um Europe and then um Morocco and Egypt and places.

Um but the freelancing work was difficult because when you're traveling and trying to freelance at the same time, you'll find that you know the client hasn't sent you the thing that you needed, and you've got like an hour now, but they haven't sent you a thing, so okay, we'll get to it tomorrow, that kind of thing.

And um meanwhile, we had we were batting around ideas for side projects together.

We had this one idea that would be a website to make sure that we didn't miss out on the great conferences and events that our friends were going to.

Um, and then we got to Casablanca in Morocco, and we got food poisoning, and uh we were really ill, and it was during Ramadan, and Casablanca is not on the main tourist trail.

So during Ramadan, everything's shut down, you know, none of the restaurants were open and so forth.

So we said, okay, well, we'll rent ourselves an apartment for two weeks and we'll try and like cook ourselves better and look after ourselves, and I guess we'll work on one of these side projects that we were thinking about.

So we picked the conference website, um, built the first version, and it was very, very scrappy.

You know, we knocked it together in two weeks, we put it live, um, and the key feature when it launched was you sign in with Twitter and it shows you the events that the people you follow on Twitter are going to or speaking at.

And that's it.

That was all that it did.

Um, but it turns out that was a really compelling, um, that was a really compelling um uh thing to offer, partly because it's a bit of a cheat, right?

Um the thing about Twitter is people who speak at conferences really love using Twitter and they've got lots of followers.

So if you build a website where and when Landed launched, Natalie and I were the only people who'd added any data to it, but it had over a hundred speaker profiles on it because there were over a hundred people who we knew who were speaking at conferences.

And so if you signed in, if you were following any of those 100 very prominent people, we'd recommend an event to you.

And it was like magic.

You know, you you click the blue button and it goes, oh, um, you're following Jeffrey Zeldman.

He's speaking at an event apart.

You're like, it's like it looked deep into my soul and figured out everything that I need to know about the world.

Um so because of that, and because Twitter is naturally viral, it started taking off way, way faster than we'd ever expected it to.

So we're trying to travel around Morocco and like have fun on our honeymoon, and this thing is is is breaking servers and we're having a lot of.

So we made it for about two months around Morocco and then Egypt.

And then we were like, you know what?

People keep on getting in touch with us and saying, hey, I need to talk to your support department.

And we're like, it's it's just us and our lap and our laptops.

This is this is a lot bigger than we thought it was gonna get.

So we ended up applying for the Y Combinator startup accelerator from Egypt.

We were in um we were in Luxor.

Was it yeah, we were in uh we were in Luxor on the Nile um when we when we filed our application.

Part of the YC application is that you have to have a vi a one-minute long video, and so we carefully positioned a Egyptian temple in the background of our video, but didn't mention it.

We thought we we'd play it cool.

And actually, I think that video is available if you want to link to it from the show notes.

Um so we did so we got accepted into Y Combinator and sort of cut our honeymoon short, moved to Mountain View for the three-month Y Combinator program, and then after that, we moved back to London and we did the whole startup thing.

We raised money, we hired people, we got an office, we um we spent um a solid two and a half years in London growing the company and trying to get the business model working on all of those kinds of things.

And then we got to the point where we either needed to raise a series A or we needed to get acquired.

And so we started looking at what the acquisition options would look like.

Were there companies that would make a good fit for this?

And um, well, yeah, we'd we'd already had conversations with Eventbrite in the past, and it turned out there was the point at their growth where they needed to really bulk up their sort of like Django engineering um team.

They had a fantastic team, but it was pretty small, and they needed to really be accelerating the growth of the engineering team.

Um and we thought there were good alignments in the product as well with some of the things that Eventbrite were looking at doing.

And so, yeah, we we negotiated an acquisition, moved the entire team to California.

So that was 11, six, six team members plus families, eleven people total, um, none of whom have gone back to England.

So it did work out well for the employees, which is good.

People were happy with the with the Californian lifestyle.

Um and yeah, and I've been I've been working at Eventbrite uh ever since.

Well, Carlton can agree with leaving England for sunny weather, right?

Mm-hmm.

I mean, because Eventbrite, you you're adding new features, but it's it's you're not spending all your time on features.

I imagine a lot of it is around scaling and so

You can scale PHP, you can scale Rails, you can scale Django.

It all fundamentally ends up as the same sort of shared nothing architecture where you've got a bunch of different application servers, they're talking to some replicated databases.

If you need to handle more web traffic, you fire up more application servers running a copy of your stack.

Um and you put caching layers in and you don't make queries in the same thing.

Um Django at Eventbrite these days, we mainly use it for the so we do we Eventbrite's never use the Django templating language.

They made a decision before I joined the company to use Mako.

Um I think because it compiled down to Python, they thought there'd be a performance increase.

I'm not a huge fan of Mako.

Um my problem with it is that it's so easy to accidentally embed business logic in Python inside of your Mako templates.

You know, it's very sort of PHP-like at that point.

Um, although these days, most of Eventbrite's new features are React components that are rendered client and server side.

So today, most Eventbrite features will be a JSON API and then a React component, which is rendered server-side for that sort of initial hit, and then takes over client-side after that.

So Django REST framework is the API layer at the front, but internally we've been migrating Eventbrite to a microservices model.

And the microservices within Eventbrite are our own.

Um we have a protocol we call PySOA.

It's a sort of um service-oriented architecture in Python using Redis as a message bus.

So you communicate and you basically um build message pack messages, stick them in a Redis queue, a service at the other end reads off the queue, does the work, sends you the message back again.

And that's um we we've also got an older SOA mechanism from a few years ago.

And the classic problem with a site like uh at the scale of Eventbrite is anytime you transition technology, finishing is really hard.

You know, you can end up with 95% of our stuff is using the new thing, but there's still this one old bit that's on the old thing, which is terrible.

And we're very much trying to develop an engineering culture where we don't let that happen.

Now you've got two problems, right?

Absolutely.

We're on API V3 now.

Um, API V1 retired several years after we said it was going to retire it, but we did retire it.

And this was a huge, like, this was a milestone.

This is, hey, look, it is possible for us to turn off these things.

Um we've got and the Eventbrite engineering team is so much larger now than it was when we joined the company.

We've got engineering spread across five cities on three continents, which is very exciting.

Um, so we've been having to figure out how to work as an international first distributed engineering team.

We've got engineering in San Francisco and Nashville in the US.

Uh, Mendoza in Argentina is one of our largest engineering offices now.

Um, we've got Vancouver, and then we also have um engineering in Spain.

Um, so we've got offices in Vienna and uh Madrid.

Um and yeah, that's that's a really interesting challenge figuring out how to productively work with an engineering team spread across that many areas and time zones.

But as soon as you throw in Spain, you're like, wow, that's like 10 hours out.

And also given like Spanish working hours, there's basically no overlap between the San Francisco office and the and the Spanish office.

But what we do have is we've got Mendoza in Argentina, is closer to Spain in terms of time zones, so they do have some overlap, and they also have like that they they also they they speak Spanish.

So we've got to be.

Can I ask you using the streams, the new streams?

Um we're we're

but is it on the roadmap?

I think we've we've explored it a little bit.

Um the the stuff we're doing is built on top of blopping, uh sorry, blocking um list operations.

You need um and that's working great at the moment.

Um the thing I'm personally excited about the Redis Stream stuff because I love Kafka, except for the bit where you have to set it up, which is a nightmare.

And so like at Event, we're running Kafka because we've got an entire ops team who can support Kafka.

Um, for my side projects, there's no way I can get Kafka running.

And if you look at Heroku Kafka, it starts at like $50 a month, so it's not easy to paste.

But Redis Streams gives you the same primitives, but it's Redis, which I can run anywhere and is already available on all of my projects.

So yeah, I'm really looking forward to exploring that for some of these smaller projects.

Do you see any of that impacting the architecture of Eventbrite, assuming it happens, or is it separate?

Like um we have so Eventbrite runs a sort of API gateway at the front that like API requests come in, we turn them into our internal service calls, pass them out through different services, they come back, we send them back out the front.

Right now, that API framework is um it's based on UWSKY, so it's lots and lots of UWISKI workers that that handle traffic.

And we know that we can, and it's easy enough to scale, you know, you fire up more UWISKY instances, more boxes running UWISKI instances.

I think for API framework stuff, that's where Python 3 async really shines, because most of it's just it's I.O.

right, you get a request, you send it off to a service, you wait for like half a second for that to return, and then you feed it back out again.

Blocking a whole worker for that feels really wasteful to me.

So I wouldn't be surprised if the first application of Python 3 async at Eventbrite ended up being something that's one of these gateways, these sort of proxies that sits at the front.

Like the way that you can nest the applications and they're all kind of everything's ASCII all the way down.

It's like

So he's been he built out um Eventbrite's internal SOA mechanism that works on Redis while he was working on both um like channels and ASCII.

So the the the design of the two systems definitely informed each other.

You know, I think there's a there's a whole bunch of ideas from ASCII and Daphne that we're now running inside Eventbrite, and vice versa.

So yeah, it's um, you know, in in 2019, the way you build software is lots and lots of little services and API requests and so on.

And async is such a natural fit for that style of development.

Yeah.

I think we're almost near the end of time here.

Are there any last projects or things you want to um mention that we haven't already covered?

You you do so many things, it's really inspiring hearing you talk, and you're so excited about them too.

I mean, uh you sound like the like anti-burnout engineer.

Um I released my first piece of ASCII middleware um a couple of weeks ago, basically to sort of test the waters of what ASCII looks like.

And that's this it's ASCII hyphen cause, so it's a uh middleware for adding cause like cross-origin resource sharing headers to an ASCII project.

But what I'm really doing with that, my my real ambition with ASCII is I'm getting a version of dataset working, the next version of dataset will be on top of ASCII.

Um at the moment, it's um it's SANEC, which is a custom web framework.

Um, and then I want the plugin system and dataset to allow you to add plugins which are basically ASCII middlewares.

So, like dataset authentication is going to be an ASCII middleware and dataset cores and all of these different features that I want to build.

And that's made me realize that because ASCII is Turtles all the way down, the interaction between ASCII and plugin systems is really interesting.

Like you can, I'm I'm hoping that we'll get lots and lots of ASCII middleware out there, and then you can use a dataset that uses plug-y, which is the plugin um library that the uh that PyTest built.

Um, but you can use that to basically make composable web applications by throwing together a bunch of weird little plugins.

It's kind of like the Django reusable apps idea, but at a slightly different level in the stack.

Like this was Adrian Holivati built this.

Um, and so yeah, given any database, it gives you basically what dataset gives you today.

And data browse, um, we eventually um it was decided to pull it out of Django, and so now it's available as a third-party like uh module somewhere.

But yeah, it's very, it's that they're very similar to each other on that respect.

But I think the biggest innovation from Dataset is this arbitrary SQL thing.

It's using using SQL as an API language, is I'm almost gleeful about it.

So I've been building this project at work, which is a search engine for our internal documentation, um, because we've got internal docs spread across like 10 different systems.

And the way it works is it's a crawler that every half hour pulls everything from these internals, these internal systems, sticks it in a two gigabyte SQLite file with the SQLite full text search stuff turned on, and then it sticks that in data set.

And the user interface for this search engine is literally 500 lines of HTML, CS and JavaScript in one file that runs SQL queries constructed in JavaScript against dataset.

And I've been literally I've been gleefully cackling as I show people, look, look, it's sending, it's generating SQL queries in the JavaScript and it's sending them to the back end.

And this sounds like an awful idea, but it's fine.

Um, my absolute favorite feature of it is that I added facet, I love faceted search engines.

Um I added facet by emoji to our internal dot separation.

Yeah, I thought I saw that.

So if you fire up the um the Firefox DevTools, you can intercept SQL queries with emoji in going over an HTTP get request.

And I think this is the best joke, right?

This is this is very much akin to my sense of humor.

I'm hoping I can open source this uh internal search engine because it's it's just really funny.

But I've been doing this, so I've partly been doing this because it's a way of trolling other developers, going, look, I'm constructing SQL in JavaScript right here.

Right, exactly.

Like none of the obvious flaws in the system have actually caused any loss in productivity or anything.

It's super easy to maintain because it's only 500 lines of code for the whole thing.

Because yeah, it's uh it started as a joke, but it's a joke that's actually turning out to be pretty useful.

So it's not the first time uh maybe a good thing has come out as a joke.

We wanted to have you on for such a long time, and it's so great to hear about the early days of Django.

I'm sure that's some stories that people hadn't heard of, and then all the work you're doing now and combining Django and all these things.

And you still get to use Django.

I mean, there's a fair number of prominent Django people we've interviewed, and they don't get to use Django day to day in their Django.

Oh, yeah, yeah, yeah.

With data sets, it doesn't use anything from Django yet, but ASCII, I think, is the way I'm gonna link it back to the wider Django ecosystem.

You know, ISC ASCII is such such a perfect fit for what I want to do.

And it also means I'll be able to do things like let you have a data a Django app with a dataset powered view that's just embedded into the URL configuration, which is gonna be really interesting.

I mean there's absolutely no reason why we can't just um embed Wiz ASCII apps inside the uh a Django ASCII app.

And vice versa.

And that I think is gonna be really cool.

Well, again, thank you so much for spending the time for being on.

Thanks a lot for having me.