Transcript: Datasette, LLMs, and Django - Simon Willison

I'm Will Vincent, joined by Carlton Gibson.

Hello, Carlton.

And we're very pleased to welcome back Simon Willison.

Welcome, Simon.

Hey, Will.

Hey, Carlton.

Hi, Simon.

I'm really excited to have you again.

He's currently working on dataset.

He writes a lot about AI, LLM, and much, much more.

So we'll get into all that.

But I'd start it off with actually, so you were at the most re uh DjangoCon US, I guess, last year.

Um, but day to day you don't do, I don't think, a lot of Django.

So I'm curious, how do you see Django 20 years in as someone who is familiar with it but isn't maybe as in the weeds as some other folks?

How do you assess its kind of strengths and weaknesses in the web framework uh landscape as it is now?

And I I'm a huge there's this incredible essay that um online named McFunnley, um, Dan McKinley put out this wonderful essay a few years ago about how you should pick boring technology, where what he means is that anytime you're building something, um, there are things that you want to innovate on and where you want to build like something new and exciting and solve problems that have never been solved before.

And then there's everything else.

And for everything else, you should pick the most obvious, boring technology you can so that you're not current constantly trying to figure out, oh, how do I do CSRF protection in this framework that I'm using or whatever?

Just make sure that your defaults are boring.

And I love that Django absolutely qualifies now, right?

I I never in my wildest imaginations dreamed that Django would be the boring default choice for building things, but it is.

And so actually, I'm building um dataset cloud right now.

It's SaaS hosting for my dataset project.

The core of that is a Django app.

I've got a Postgres and Django app which manages user accounts and manages signups and all of that kind of thing.

And then it launches Docker containers on fly.io, which run dataset and all of that.

So all of the exciting stuff I'm getting to innovate on in the corner, but the the sort of bog standard bits that make the whole thing run, it's Django, and that's great.

Um so yeah, I I love that.

I love that Django is now the safe default choice for building a web application.

Lovely.

So Carlton's had some thoughts on, you know, maybe 20 years on changing some some of the defaults.

Carlton, all right, do you want to give your quick pitch and we'll see what Simon thinks?

So my my kind of take is this is that um we've kind of got a leaky battery with the user model because we ask um users to create this custom user.

And it's it's a whole world of complexity that for the that central auth model, which is like for every single request, uh is the identity of this user is X, and not the profile data, which obviously you want custom per app, but we sort of have this custom user model which we forget to set up, and we you say we there's all these warnings in the doc how you should use it, but oh, don't migrate to it because that's too hard.

And ah, and I think we made a mistake there.

I think we what we should have done is tripped is trimmed off all the non-identity stuff from that user model and then locked up Django Contrib really tight.

There are four flaws in the default user model.

Firstly, it expects everyone to have an email address, which doesn't work in 2024.

Uh it makes people pick a username, which is very archaic, um, and it expects uh your your name to split into first name and last name, which for many cultures doesn't work.

So, yeah, I I I'm very with you that the that the user model has not dated well, unfortunately.

Trim off the bits and somehow, somehow do it.

I think we gave up a little too early on that is all that's you know, so I've been experimenting in that domain.

I mean, I use the user model as a key that other things key onto, and then I have a second table of of Google accounts that have been uh associated with an individual.

I don't make people pick a first name and last name, yeah, all of that kind of stuff.

Yeah, I know.

I think I I rolled my own Google uh Google OAuth thing, which keys against Django.

I think I've probably got both of that lying around somewhere.

Um and I've done that before in the past.

I tend to de use the default user model partly just for the admin.

It's the most convenient way to get the admin up and running.

And I always have the I mean the admin is such a key feature for me to quickly iterate on what I'm doing and build out like internal tooling and so forth.

Oh, I need half a day of planning my needs for a user model.

You just want to start and then yeah.

I'm impressed you in implemented your own iOAuth though.

Right, okay, so it's yeah.

Um, I mean uh so my blog runs on Django, um, Simonwillison.net, and that's open source.

Like the the it's it's not a very complicated application, but it's all sat there on GitHub.

And I find myself, I tweak that about once every three or four months.

I'll go in and I'll tweak something about it.

And it's it's always fun.

It also um it's managed by DependBot, so it magically upgraded itself to Django 5 a few weeks ago.

It just did it, which was great.

I I didn't have to think about it.

I've got just enough automated tests that I trust it that the thing's gonna work after I apply updates.

And yeah, that's that that's a nice sort of way of staying staying connected with what's going on in a very sort of low-risk environment as well.

It was really good.

It was like a kind of checklist that you could run through of how to build a Django blog.

And I kind of I see everybody struggling with Hugo sites and this static site and this generator or that static site generator.

And I sometimes think, no, just run your own Django app because it's great to have a that playground.

And you've been doing it for like 20 years with the same Django application just evolving.

It's lovely.

It's yeah, it's it it pretty much.

As like and was it even was it the PHP equivalent of Pickle?

I think it might have been PHP's Pickle equivalent of just a big like array of posts and stuff.

And yeah, then I I I flipped it over to Django actually only about 2000 and no, I I might it's on my blog somewhere when I first ported it to Django, and then I did a major upgrade um in 2017 when I came back after not blogging for like seven years and did the Python 3 upgrade and stuff, and I've just been iterating on that ever since.

It's it's great.

I'd forgotten that.

Oh, that's brilliant.

He has this idea of a series of posts around a certain topic.

And so I added series into my blog inspired by what he'd

been doing.

Upstream.

Anyway, we'll carry on.

I don't know if you kept up with this, but Carlton did a talk in 2019 on single file Django.

And then at the most recent DjangoCon US Paulo um Melchiori has like there's a whole repo of like, I think it's six lines of like kind of proving like you can do Django in a couple lines.

Um and I guess I wonder, I think about this as teaching, because I again like my brother-in-law is going through a boot coding boot camp, and I'm like, hey, I'm here, let me help you with like oh, we're doing Flask.

I'm like, why?

Like, I almost feel like I don't know if it's worth doing showing like single file blog on Django or something just to make the point that, like, hey, it's possible.

Because even in Flask, like you don't no one does it that way.

You could, but no one would do it that way.

I actually I built my own Django single file thing like 10 years ago, something called DJ N G Jing.

It's Django without the vowels.

And yeah, that was basically just trying to do a little thin shim that lets you do a flask um imitation on top of Django.

Because I love that for just hacking out quick things.

I I don't not having to bother about the directory structure and so forth.

So yeah, I'm I I'm thrilled to hear people are still pushing ahead on that.

It's it's it's it's a great idea.

If we were to j design Django today, I'm certain it would be like capable of doing single file out of the box.

That just makes sense to me.

Uh this question comes from Eric Mathis, uh, who who wrote Python Crash Course, and he was asking, you sort of answered it, but what is your preferred way of building web apps today?

I think specifically on the front end, having seen it go from you know server-side rendered to jQuery to spas, and now I guess HTMX.

Uh but where do you fall on that pendulum?

The thing I hate it, it's the build script.

I hate it when you have a front-end project which you work on every six months, and you come back in six months and nothing works.

It's it's all you have to re-spin up your webpack configuration, all of that kind of stuff.

And so a few years ago, I said, you know what, I'm gonna give myself permission to write JavaScript like it's 2008 again.

And so, no libraries, no build scripts, no TypeScript, nothing like that.

Just like little bit because the thing is that we used to use jQuery because of the browser differences.

But the browser differences are gone, right?

Today, document.query selector all and all of that stuff, it works exactly the same across everything.

So you can build code like you're using jQuery, but without using jQuery.

You just write like event handles and so forth.

And it was so liberating.

Like suddenly I enjoyed front-end development again because I didn't find myself fighting Webpack and whatever and Vt, whatever the new cool stuff is.

And I and I could go back to projects I wrote like this two years ago, I can drop in and I can maintain them and I can add new features to them.

And on top of that, the um like language model stuff, ChatGPT is really, really good at all forms of JavaScript.

So it's not like I ever find myself stuck trying to remember how a certain API works.

If there's something which is going to be a bit tedious because the JavaScript's gonna be 20 lines of boilerplate, it'll spit out the 20 lines of boilerplate.

I could just let it go and let me get on with it.

So yeah, I've I've got really into that.

I have played with um HTMX on a couple of projects.

I really like it.

HTMX is so good for that kind of thing.

Um and so I really I I like that.

I love that that's getting popular, and I love the performance that you get from it because you don't have to serve a megabyte JavaScript bundle just to like show a contact form or whatever.

Um and then dataset itself is very strictly, it's just HTML, and when you click a link, it loads a new page.

But I've been playing with the Chrome view transition stuff recently, which is um super, super interesting.

Like um cutting edge Chrome, I think you might still have to turn on one of the experimental flags.

You can actually serve up CSS that says, and when the user navigates from this page to this page, keep this area of the page stable and sort of like blur update this other bit, and it's like a couple of lines of CSS, and suddenly it feels like a SPA.

Like you click a link and only parts of the page updates and so forth, but it's a real navigation, there's no JavaScript involved.

That's thrilling.

I can't wait to see that roll out to other browsers as well.

Just with bundling, because I just I just did a redesign of my main site, which is um using Tailwind.

Um and it I loved I like Tailwind, but it's a little disappointing.

I now have to have like node and stuff running to it's almost like it's switched from JavaScript to CSS now to have a build script for everything.

The stuff I do, I always try and keep it small and simple enough that you don't necessarily need that.

And then they just become friction.

Like it's it's just something that that prevents me from being able to because I have so many projects on the go at once.

I've got what a hundred and nearly two hundred.

It's it's I've got some ridiculous number of actively maintained projects, and the only way to do that is to make it as easy as possible to drop into something that you've almost forgotten all of the details of and get it up and running again.

I feel like with the the front-end build stack, if you do you work on the same projects every day, it's completely fine.

It gives you a huge productivity boost.

There's none of that friction because you're you've constantly got that stuff sort of warm in your head.

If you drop into a project every six months, it's completely different.

And that that's so I like to optimize for being able to hop across hundreds of different projects and make small changes to them without getting stuck on the building.

Is if you focus your if you focus on one or two or three technologies, then you're able to really get the most out of them rather than spreading yourself thin over, you know, say half a dozen, and when and that slows you down.

It's a sort of saying.

Like I've got a hundred odd repos, they're all Python plug-y plugins, ginger templates, um like dataset plugins.

They're all the exact same shape.

They've all got GitHub actions running workflows and so forth.

It it it just works.

So you mentioned uh LLMs there and chat GDP and things like that, but we go I wanted to to ask you something before we you know talk about those in more depth, which is you're you're not as well as doing all this amazing working open source, you're now on the board for the PSF.

Yes.

Because you're new.

This this is your first year on the it's

I just hit the 12-month point.

And it's interesting.

So the reason I'm on the board of the PSF is that I'd been hassling the PSF on a sort of low-grade basis.

Every now and then, I'd go, I'm really annoyed that the PSF isn't doing more to help make Python easier for people to get into, like solving the horrors of the Python learning development environment, all of that kind of stuff.

And um, and also the the fact that it's very difficult to distribute applications written in Python, because you know, you if you want to, you don't want people to have to install Python to use your stuff.

And I realized I almost have a snap judgment one day.

I was like, you know what?

It's not reasonable for me to complain at the PSF and not offer to help and not try to do something.

So I put myself up for um election on the basis of I want these are the problems that I think the PSF should be addressing.

And I got elected, which was a little bit of a surprise because I didn't really I mean I think it is name recognition because you show up on the list of names, people are, oh, I recognize that person or whatever.

Um and of course then I made it I now that I'm in the PSF, I realize that the PSF is not particularly well equipped to solve the problems that I was most interested in solving.

Because the PSF is a lot of people.

The PSF is basically a it's about it's about money that's that's raised and is distributed around the Python community and initiatives to um the the the PSF's focus is on the community and the health of the community.

There's a huge amount of sort of sponsorship of events of um of initiatives like that, which is fantastic.

The stuff I care about is it's not completely aligned with what the PSF is for, but it's not unaligned either.

So what I'm having to learn is okay, how do what how do I align what the PSF can do with the things that I want to get done in a way that supports the the missions of the organization and so forth.

So it's been a huge learning curve.

You know, this is my first time on the board of a nonprofit.

It's understanding what levers are available to pull and what priorities make sense and so forth.

Um, and yeah, so the first year I was in mainly in sort of just trying to understand what this how this thing is shaped and what it can do.

Now that I'm through that, I'm looking forward to to maybe trying to tweak those levers a little bit myself.

Because it basically takes a year to get up to speed.

And during COVID, we had less turnover, and I feel like we got a lot done because we had largely the same crew for two or three years.

Um because it does it takes a it takes a year to just understand how it works.

Right.

Sorry, I interrupted though.

You were gonna No,

I mean, we're gonna have Jacob Kaplan Moss on in a couple weeks, um, who just joined back the DSF after obviously working on Django and being one of the I think the first president.

Um but when when I joined, I had a similar list of things I wanted to do, which I guess I in hindsight, I guess I was lucky that they aligned with they were all things that could be done around like sponsorship and um God, I I forget, I have a blog post on it.

But I didn't I hadn't thought about the fact that maybe the things I wanted done didn't align directly with the mission, but but you're right, it's fundamentally these organizations are about money and community and helping others.

I mean, one thing DSF is is now doing is having working groups, which the PSF has had mixed success with, but at least some success.

Whereas historically it's just been everything goes to the DSF.

And when you're on the board, that's kind of its own thing.

It's unreasonable to be on the board and spearhead an initiative from what I've seen.

I I imagine it's similar on the PSF.

Or I don't know, are you thinking of like you actively doing it or more you can like help spin up a group that

The PSF is a like like the staff to

I mean, PyPI, I think, is one of the most um impactful things that the PSF does outside of the uh PyCon and event sponsorship and so forth.

And so the directors are not there to do the work, the directors are there essentially to sort of help make those high-level decisions, help set strategy um and yet make decisions about where the where where the money goes to a certain extent.

And yeah, it's it's it's understanding, okay, what also what's ethical and responsible to do?

Like if I throw all of my weight in trying to push the PSF in one direction, am I actually starving other important initiatives that the PSF are doing that just don't happen to align with my own personal interests?

Well, in in a sense, you're pure is not the word, but you know, quite a few people um work for big tech companies, and so it's there's even more of a potential of I don't know, not conflict, but it gets a little, we've got to watch what you're doing there.

Yeah, I'm I'm being unemployed by a large company, that gives me an aspect of independence.

But most of our board members are independent.

So it's it's not uh I'm not unique in this.

Oh, okay.

Okay.

Um so again, as this Django person, but a little bit of an outsider, I can ask you all the questions that uh uh you know I want someone to weigh in on.

So an executive director.

We're gonna have Deb Nicholson on the podcast in a couple weeks.

There has been talk of Django potentially having one.

Um what has your experience been seeing an executive director at work in one of these organizations?

So can you imagine one of these organizations without one?

I think a lot of the stuff won't happen absent someone full-time to do it.

Absolutely.

Like having having this is one of the problems with boards of directors is if everyone's just a a volunteer who's investing a few hours of their time a week or maybe a month, it's very difficult to make progress on things.

You find you'll you'll have a meeting and it'll be that not much will have happened since the last meeting.

And with once you've got an executive director and staff, that completely changes.

You know, there is constant forward motion.

Well, I keep on telling people the the I think the best thing about the Django Software Foundation has always been the fellows, because that's the and that this is something I I say I don't understand why other open like community-driven open source projects aren't trying to imitate this exactly, because it works so well.

The PSF now has, I think, at least two fellows inspired by the Django Software Foundation, and those are incredibly impactful.

Um, you know, the work that they're doing, the work that Seth's been doing around security is a sort of relatively new addition, absolutely extraordinary how much impact you can have with that.

So, yeah, I'm I'm very, very keen on the idea of these non-profit open source supporting foundations that actually have staff that can could just keep on making progress on things.

But it wasn't really the fellow role.

And there wasn't enough capacity to make any sort of significant progress on, you know, for instance, you know, reworking the Django Project.com website.

We okay, do a little bit of work in it, but it was literally an hour or two here or there, and not the massive month-long project, months-long project that's going on now to actually do a proper assessment and what does it need and how do we refresh it in a sort of professional, you know, to a 24 2024 kind of standard, you know, rather than just oh yeah, can you make a tweak here?

That's exactly um, yeah.

Um so yeah, I put that out there, but I'm not on the board now.

So but Carlton, yes.

So I want to get so I've been using Copilot and whatnot, and I think it's it's awesome.

And it's it's you know, you you mentioned JavaScript earlier.

Like my JavaScript's come on so much because they're a bit like how do I how do I filter this array to get this the one value that I need?

And previously that would take me 10 minutes of looking up because it's not something I do, you know, do it once every six months, but now I can just ask the LA and it's got it, and it's not it's not rocket-breaking code, it's not, it doesn't have any value other than it saved me 10 minutes.

Um so I guess my question is how can I leverage that?

And how can I leverage uh continue to leverage that?

And and your tooling, how can I install that?

And can I get something equivalent to the closed source that I can use that's open source?

Yeah, so I'm gonna.

But that's kind of let let's get into it.

Yeah.

Um I'm with you, like the thing that excites me about LLMs is I love them as as sort of teaching assistance, right?

It's something I can ask questions, I can ask the dumbest question in the world at three in the morning, and I'll get an answer, and it doesn't judge me, and I don't like like and I don't feel the knowing how to do a for loop in bash or whatever it is.

I love that, and I love that um it lets me be so much more ambitious with the projects that I take on because like um a great example, I shipped code in Go for I needed a little like high-performance network proxy router thing.

And I I ended up writing it in Go because I don't know Go, but ChatGPT GPT-4 knows Go throughout.

And I know Go just well enough to read the code and be able to tell if it's doing the right thing.

And I can I can get it to write tests.

So I ended up building this like hundred-line little custom Go server thing with comprehensive unit tests and GitHub actions running, continuous integration I got, continuous deployment running, all of the things that I I consider to be important for a like robust project, and I shipped it, and it's great.

Like last month I had to make a change to it, and I fired out GPT-4 and I worked with it, and we figured out what what to do.

And and I absolutely mean that was extraordinary because normally I would never write something in Go because I'd be fine tinkering with it, but I'm not gonna write production code in a language that I'm not completely fluent in.

In this case, I'm I feel like me plus GPT-4 is fluent enough that I'm willing to deploy code written in a language I'm unfamiliar with.

I've written code in AppleScript.

Apple script is notoriously a read-only language, like you can read it and see what it does.

There's AppleScript on one end and Perl on the other, like read-only, write only.

Um but yeah, I'm I'm I'm using AppleScript for things.

I'm using all of these weird little domain-specific languages.

I use JQ all the time now because JQ is really powerful, but I can never remember the syntax for all the different things it can do.

So I love that.

I love it as a sort of um an accelerator for me doing lots of things.

I'm taking on more projects, which is terrifying because I already had too many projects, and now I'm like, oh, I mean, me plus chat GPT, I can probably get something working in 20 minutes.

And of course, it takes two hours, but still at the end of that two hours I've got something that works and is interesting that I wouldn't have built otherwise.

I can use the code interpreter mode where it actually runs the Python code it generates.

So I can get back from an hour-long walk with the dog, and I've got 50 lines of Python that I know works because it actually ran the code, found the bugs, fixed them, all of that kind of thing.

It's incredible.

I can literally talk to it while I'm on a walk with the dog and it writes code for me.

That's utterly surreal that that's even possible.

So yeah, I love I love that aspect of it.

Um, and yeah, but the as you mentioned, the problem with ChatGPT is it's a it's for a company called OpenAI, it could not be more closed, right?

It's this proprietary hosted model.

They change it all the time without telling you what they've changed.

So people keep on complaining that it's got weaker or it's worse at X and so forth.

I never know if that's actually true because it's basically a random number generator.

So it's very easy to assume that it's changed when it hasn't.

But that's really frustrating.

And then but the great news is that in the past like 12 months, we've had so many new options for running these things ourselves, these openly licensed models that you can run on your own hardware, and they're beginning to get pretty good.

Like, I don't use any of them on a daily basis because GPT-4 is so good, so it's sort of my default, but I'm constantly experimenting with them.

My favorite at the moment, um, I have two favorites, are these Mistral models.

There's Mistral 7B, which literally runs on my telephone.

Like there's an app that runs it on my phone, and it's not awful.

Like, I was on a plane and I was using it to do the kinds of things I might have looked up on Wikipedia, and okay, it'll probably hallucinate stuff, so don't depend on it telling you the truth, but it's still useful for sort of getting things, getting just starting to explore different ideas.

And then the other one, there's this new one called Mixtral, which is a Mistral mixture of experts model.

They just released that um a month ago, and that runs on my laptop and is feeling it, the quality begins to feel like ChatGPT 3.5.

Like it's very, very good.

So if you've been resisting using these things because you don't want to use some weird hosted model by some like closed open company, Mixrel is something you can run on your laptop right now.

It's a it's Apache licensed.

The whole thing is Apache licensed, although whether it's truly open source is up for debate because they won't release the training data that it was trained on, which is, I think that's the source code.

Right.

I think for these models, the raw training data is the source code that was used to compile the model.

Of course, you can't open source that training data because you ripped it all off.

It's full of copyright data, and you can't just slap an Apache 2 license on someone else's copyrighted works.

But yeah, so this stuff is really exciting.

It's it's really interesting.

And so there's this um these lots of cases where the the the LLM will reproduce its training data almost exactly in cases.

So the the New York Times um we've got this um lawsuit.

Perhaps you could explain the thing that I I can't I kind of see it though, and I'm like, oh wow, yeah, it is actually reproducing, you know, you type in underwater sponge and you get a sponge called Bob SquarePant come out from the one of the image generators, for instance.

The ethics of this entire space could not be more murky.

Like every aspect of the space, you're like, wow, is that okay?

And the answer is maybe not.

It's all very, very and that so a lot of people have ethical qualms against this, and I agree with everything that they're saying.

You know, the the New York Times thing is um is is the so the most recent thing is the New York Times filed a very big lawsuit against OpenAI a few weeks ago.

Um it was against OpenAI and Microsoft, and it was complaining about three different things.

It was complaining that firstly, you took all of our work um without permission and it's copyrighted work and you used it to train your model.

And I don't think anyone is is is disputing that that is what OpenAI did.

They used OpenAI, they used New York Times data as part of a vast amount of training data that went into these models.

So effectively, you could look at it as it's a crawl of a sizable chunk of the internet that was used to train these things.

But that included New York Times data.

The New York Times say that OpenAI put more weight on the New York Times data than they did on other data that they trained on because of the high quality of that that training data.

I don't know if that's conclusively proved or not.

I think the GPT-2 paper a few years ago did explicitly list that the New York Times data was being used like that.

So they might be assuming that that's still true.

It probably is still true.

But this is part one of the things I'm excited about, this lawsuit is I want discovery because I want to know how GPT-4 was trained.

Because they haven't told us.

But yeah, so complaint number one, they trained without permission.

Complaint number two is that the models can spit out exact copies of New York Times articles.

And this was news to me.

I didn't, I thought that the act of training muddled the stuff up to the point that it won't spit out exact copies.

It turns out if you set the temperature to zero and then feed it the first two paragraphs of a New York Times article, it can often spit out the next four paragraphs.

And sometimes there are very slight differences, like one word will be changed, but effectively it's it's it's memorized and it's regurgitating the same thing.

It would be

And that's I mean, so many of these things, I don't think there's a obvious legal, I'm not a lawyer at all, but that there's a reason this is going to go to court because that these are legal questions that are very blurry and unanswered.

So complaint number two is it can reject regurgitate their content.

And they've said um this means that people bypass our paywall by getting the model to spit out articles, which is a bit of a loose claim because you've got to have the first three paragraphs of the article anyway.

But they did have a really interesting thing where they talked about the wire cutter, right?

Where the wire cutter is a New York Times company, it does product recommendations.

If you ask ChatGPT for product recommendations, it will often spit out the wire cutter's picks, but it won't give you the referral link that's the wire cutter's business model.

And this is the the the definition of fair use in American law specifically talks about um whether the thing is competitive with the thing that it ripped off.

And so the New York Times case, the main thing they're trying to demonstrate is this competes with us, this is harming us financially because you can bypass our paywall, you can like rip off wildcatter recommendations, all of that kind of stuff.

So that's argument number two.

Complaint number three is actually about retrieval augmented generation.

It's about the thing that um Microsoft Bing does and uh Chat GPT Browse does, where you can ask it a question and it goes and does a search on the internet, and it'll find the New York Times article about something, read bits of it, and then like summarize that and give you the summary back again.

And so there the New York Times is saying, well, look, you're clearly subverting our paywall, you're you're profiting from content that's derived from us.

Now that one, that's one one, it's almost the one that worries me the most in terms of I think they've got a completely fair point in complaining about this, but summarizing stuff is my favorite use of LLMs.

Like if we come up, if we end up with legal precedent that you can't even copy and paste data into an LLM to get a summary back out again, that would be very harmful for the sort of the ways that these tools are most useful.

But that's the problem, is that I read the 69-page lawsuit, and it's very clean, it's very well argued, and I think, like all like I said, not a lawyer, but all of these points feel to me like points that are worth putting in front of a judge and jury and and and and trying to get answers about.

I think I mean two things come to mind from what you've just said.

One is um, I know Google uh has been um told it has to pay news publishers in various countries at various times, because it does exactly that.

If you Google the news in a country in Australia, I pick Australia, I don't know if it's applied in Australia, but it will go and you know get the Sydney Morning Herald and summarise it without you ever having to leave Google.com.

And they were you know, the world thing about that

And Google are clearly gonna be if the the the the Google are clearly gonna be on the chopping block next after after OpenAI and Microsoft because they've got a prototype um like an alpha version of their search page that does exactly that.

It just adds generative AI and it spits out a generated answer to your question at the top.

They've been doing this with their like little content snippet boxes and so forth as well over the past few years.

And it's super worrying, right?

If you've got a web where nobody ever clicks a link from a search result because they just get their answers right there in search, what point is there in trying to like build a profitable web business anymore?

You know?

So all of these ethical complaints are very, very legitimate.

So we know now that LLMs are being used to generate a lot of the content on the internet.

How do you see this going forward if the LLMs are going to be really trained on themselves?

Do you think that like is this is 2021 the the you know the the high point?

Or is there a way out of that?

Because it seems a bit like a sort of vicious circle.

It seems like an aura boris.

And it's people have been talking about this for a couple of years now.

And um at one point I heard that OpenAI, the reason they hadn't updated their training data, like there was a training cut-up of what, September 2021, I think.

And the reason they'd updated it is that after that point, there was enough usage of these tools that the internet was beginning to fill up with LLM generated text, and they didn't want to train LLMs on LLM generated text because of the Auroboros effect.

At the same time, in the openly licensed language model community, almost all of the really good ones are actually trained on GPT-4 output.

Like the way you the way you build a really useful um like chat-tuned language model is you need to give it 20,000 examples of of good conversations.

And the easiest way to get those is to get GPT-4 to spit them out, and then you train your model on GPT-4.

And so if it if that was such a bad thing, we wouldn't be seeing models that were trained almost exclusively like that show up at the top of the leaderboards.

So I think this is all, I mean, this is all part of the larger problem that we really have very little insight into how these things work.

They are giant like 16 gigabyte blobs of floating point numbers.

We're and and and we're still trying to figure out just the basics of how you sort of poke around inside that weird matrix brain and figure out how it's working and what it's doing.

And so yet maybe the fears of LLM's training on the LLM output are don't don't actually work out.

Maybe it's okay, maybe it's complete catastrophe.

We have no idea.

And it's funny that we had no idea six months ago, and it feels like we still have no idea now.

So despite the rate at which this technology is improving, the rate at which our understanding of it is very sort of dubious in in terms of how how much we can figure out.

And as a computer scientist, it's infuriating, right?

Because I like computers that do exactly what you tell them to do, and you can write tests and you can fire up a debugger and everything is is is is repeatable and understandable.

And these are not that at all.

It's like a completely sort of weird, blurry alternative world in which yeah, everything's based on vibes.

You you come up with we can you you pick a model and you poke around with it and you see if the vibes feel right, and then you tweak your prompts.

And does that seem better?

I mean it kind of does, but it's it's it's awful.

It's really difficult to do sort of responsible developments on

top of it.

And especially like I'm in Boston, there's a lot of research places, they're like, we can't use an open uh LLM thing, but these closed things are definitely being sold and used on whatever industry company has huge amounts of their own data.

I I would say I almost feel like that's got more promise than this, like the entire web being you know, stolen approach in the long run.

Well, I don't know where you call that.

It was supposed to be the best possible LLM for finance.

And then it turned out that GPT-4 came out, and as a general purpose model, it was beating the Bloomberg one on financial tasks.

Because the the the this is one of the things that's so challenging right now is the rate of improvement of these things is such that if you've got a project that will take six months, you maybe shouldn't do that project because you might spend six months on it, and then GPT-4.5 comes out and it solves the problem that you just spent six months trying to solve.

And so there's this interesting strategic problem where at what point do you actually settle down and start building on this stuff, as opposed to thinking, you know what would be quicker is if I waited two months and then started building, because I'd get a better result than if I started building today.

And that's absurd, but that's genuinely the position that we find ourselves in.

Completely.

And I read that, I think I read that five years ago.

He had he summed all this.

This is before um OpenAI came out, but he basically said you need three things.

You need the algorithms, which finally like we had at that time, you need training data, and then you need processing power.

And he argued with the cloud, that basically it all came down to data.

You know, this is back in the day because we had the algorithms, they're basically open source, we have the cloud computing, and so it's really all about training data.

I think he went on to say he thought China would surpass the US for that reason because it has uh no privacy controls.

But all of that is to say to you, where do you do you see it as tweaks in the is there more to juice to squeeze out of these LLM models, do you think, or is it really more about crap like a data science thing where it's all about what you put in and trying to optimize that?

I'm very confident.

Um mainly because if you look at the open model community over the past like since since since February, people just keep on coming up with new little tricks that make the models run faster and smaller.

Like the fact that I can run a chat a GPT 3.5 class model on my laptop now, and I certainly couldn't do that a year ago because like the models that were coming out, um, like the first versions in Lama and stuff were much larger, required much more hardware, much less optimized.

Um, so there are so many techniques that can be used to make these things.

And I like smaller and faster, right?

I want I want a model that works on my phone and can do the things that I need it to do.

I want it to be able to summarize and extract facts and call functions and all of that kind of stuff.

Um, but at the same time, people keep on finding that the higher quality the data, the better.

Like there really is so much to be said, especially when you're fine-tuning these models for just having super, super high quality data that you feed into them.

Um if the New York Times thing plays out one way, we may find that it's no longer possible to just steal the entire internet and train your models on it, at which point that becomes raises some really interesting questions.

The thing that worries me most about that is does that mean that LLMs then become incredibly expensive to build because of the licensing costs, to the point that you don't give them away for free?

And so does that mean that only people who are very wealthy can afford to use these tools?

Whereas today, anyone who can afford an internet connection has access to some of the best in class of these models.

So that really scares me.

Like the the the that that I feel, despite the fact that the um the the ethics around copyright, I mean, that there are very, very real concerns here, but at the same time, a world in which only the vo the vo the most wealthy have access to the the to these toolings, that feels unfair to me as well.

Like to take them away would be foolish.

Like also,

You create a blank market of people, it's very cyberpunk, right?

People swapping USB sticks with like with the last version of Mistral that was released on them.

So there was a paper a little while ago just to pick up what you said that about OpenAI saying we haven't got a moat or something like that.

Um there was it was

It was somebody within Google put this memo together saying, saying there is no moat for this technology.

Um it's interesting to revisit that.

That I think that was it was quite, it came out in maybe March or April of last year.

And it's interesting to look back at that now and say, okay, how much of this played out?

Because uh one of the real challenges with this stuff is um if it's all just driven by human language prompts, the cost of switching to another language model might be as simple as saying, okay, we'll run this against Claude instead of GPT-4, and maybe that will give you the exact same effect, right?

Um or maybe it won't, because so much of the prompting comes down to these very small tweaks that you make where you're like, oh, okay, if I capitalize the instructions to output in Markdown, maybe it'll actually listen to me this time.

But that effect itself is kind of hurt by the fact that OpenAI upgrade their own models.

So just because that won't work now, will it still work in a few months' time?

It's it's kind of uncertain.

Um so that's part of it.

That's something I really like about the open model community, is there's all of this sharing and this acceleration that comes from just having tens of thousands of people worldwide all trying to solve these problems.

And OpenAI are an incredibly talented, experienced set of people, but I still don't like their chances against tens of thousands of people around the world.

Although, of course, when those people around the world figure something neat out, cool, open AI can just take that research and use it themselves.

So you can they can sort of keep up that way.

Um but yeah, it's and there's also there's the compute, right?

Like it's we still don't know why GPT-4 is so much better than everything else.

Um the most likely thing is that they ran it for they trained it for longer and they trained it on more data than anyone else has been able to do yet.

But still, that's people are catching up.

Now that there is, if you have a hundred million dollars, maybe it's worth trying to funnel that into data and training.

You know, that the that it's it's not like there's a shortage of investor money floating around the space at this point.

Yeah, I guess.

When people people often complain about the environmental impact of language models, where they say, well, look, training, like training a language model takes this enormous amount of carbon dioxide, which is true.

At the same time, it's about the same amount of carbon dioxide as flying a Boeing 747 across the Atlantic twice, you know, which is a vast sum.

But I would argue it benefits more people because your your airline flight benefits the people on that plane.

The language model, if it's then used by a few million people over the course of six months, it feels like you are getting uh more value for your for your for your sort of carbon dioxide at that point.

So I my understanding of ML, which is machine learning, which is quite limited, but that was that the training was the hard bit.

But then once you what you get out of um the training algorithm is a kind of um vector operation which you can run almost, you know, quite cheaply.

And then I saw though people complaining, um, and I didn't have the time to to follow up, but that every time you generate an image with DALI or or or whatever, that uses so much water or so much this because it's still computationally expensive to run the model, not just train the model.

Is that true?

So, like I said, I run models on my iPhone, I run models on my laptop.

I'm not worried about their resource constraints.

Um, but again, I don't know what GPT-4 is running on.

I'm pretty sure it's running on a full server rack of GPUs.

So my hunch is that for the very large models, yeah, there's a lot of cost in the inference.

I still think it's a fraction of what it costs to train them.

That's that's the intuition that I've I've gotten from this.

Um and you know, like the image, like stable diffusion also runs on my phone.

So there are versions of these models where the environmental impact of running them is no worse than than turning your laptop on.

That's but but I don't really have good insight into what the large hasted models are doing.

I wanted to ask about your tool because if I want to run this, you've got the perfect tool for me to download and do.

So please tell us about that because we've we've talked about all the exciting things.

But okay.

If I actually want to do it, what do I have to do?

I got lucky, LLM was still available on the package index.

Um so you can PipEx install LLM and you get a command line tool for interacting with models.

But what's really fun about it is that it's um it's inspired by dataset.

It's all based around plugins.

So out of the box, you can give an OpenAI API key and it will run against OpenAI.

And then there's about a dozen plugins you can install that will add additional models, including models that run on your own machine.

So you can essentially pip install my tool and then pip install a plugin that adds a language model to it, and now you've got a four gigabyte file on your computer that you can start interacting with.

But crucially, the interface is the same no matter what model you're using.

So it's llm space double quotes your prompt, or you can pipe things into it as well.

You can do cat myfile.txt pipe llm.

And then if you by default, it'll use your default model.

If you stick dash m space clawed on the end and you've got the clawed plugin, it'll run it against clawed and so forth.

And um, of course, everything it does is logged to SQLite because I do everything with SQLite.

So I just use it on a like daily basis for all sorts of different bits and pieces.

And I'm I've accumulated like a few thousand prompts and responses in my SQLite database of things that I tried out.

Maybe at some point I'll do some analysis on that and try and start comparing models that way.

But really, the fun thing about it is I'm trying to make it so whenever there's an interesting new model, you can install a plugin and start playing with that model.

And that works for hosted models and it works for local models as well.

Um and yeah, it's it's really, really fun to hack on.

One of the things I've realized from playing with it, one of the original ideas, is um the Unix philosophy, the Unix command line of piping things to other things, is an amazingly good fit for language models.

Because the language model is it's a function.

You you pipe it a prompt and it gives you a response.

And so one of the things that I use my tool for is um it's um it ties into this concept of system prompt, which is something that OpenAI did originally and other models have started picking up, where you've sort of got a second prompt that gives you instructions about what to do with your other data.

So a great example is I can take a I can take a um file and I can say cat my file.py pipe llm dash dash system write me some unit tests, and then the model gets the prompt, write some unit tests, and it gets a bunch of piping code, and it'll spit out a bunch of unit tests.

And of course they won't be exactly what you need, but it's that skeleton that you can start hacking on.

It's really good at explaining code.

I pipe it code and say, explain what this thing does.

Um I use it for uh release notes, not to publish.

I I kind of feel like it's rude to just straight up publish something that an LLM wrote for you, because I mean, what are you doing, right?

Like it's it's fine to take as long as it's fine to publish something which you're willing to sign your name to because you at the very least reviewed it extensively and hopefully revised it and tidied it up.

But there are lots of projects out there that don't bother writing good release notes.

And what you can do is you can check out that git repository and you can do git diff between this version and this version, pipe llm system, write release notes.

Um, GPT-4 can understand a diff format, it'll it'll read it, and it'll spit out release notes, which in my experience are about 90% correct and 10% slightly wrong, or maybe there's a hallucination in there, and that's fine, right?

That's good enough for my purposes of just saying, okay, what have they done in this release that they didn't bother writing release notes for?

So yeah, I I I recommend trying this thing out, partly because it's fun to play with models.

Um, something I'll say about the models you can run on your own laptop is they are kind of crap.

Like they are they are very, very weak compared to GPT-4, but that's a feature because it's easier to build a mental model of how they work when you work with the weak ones.

Like GPT-4, because it's so good, you can use it for a few days without really seeing the weaknesses and the flaws in it because it gets most things right.

But it's still just, you know, guessing what word should come next.

It's doing the same kind of thing.

The little ones will hallucinate wildly, which is so useful for getting a feeling for okay, these things are not intelligences, these things are dumb auto-complete that's just been scaled up to be able to cope with lots of things.

Um, I love, I use myself as a test thing because I've been around on the internet for long enough that these things can answer questions about me, like I can ask for a bio, and some of the models will get most of the details right.

They might say I went to a different university or whatever, and some of them will just hallucinate wildly.

And so I've had models tell me that I I co-founded GitHub and things like that.

Um and it's amusing, but it's also quite good as a sort of like just an initial sniff test to see, okay, how good is this model when it comes to hallucination and that kind of thing.

Okay, super.

I wanted to add one positive note, which I've heard about, you know, we mentioned that these tools could further uh increase the economic divide, but they are democratizing a lot of things, like in unexpected ways, at least to me.

Like, for example, someone I know is an admissions director at UC Berkeley, and someone and a friend asked that person, hey, you know, what is it now that what is it like with these college essays now that um ChatGTP exists?

And he said it's actually great because it's an equalizer, because rich kids have had private essay tutors for forever, and now everyone has, you know, 80%, 90% of it.

I mean, it probably makes them all sound kind of the same anyways, but it's it's a tool that people who don't have these external resources, they if they know how to use it, can you know, up the you know, just like Grammarly and all these tools to help increase the writing and and so that's I was pleased with that because I think it's very easy to get a little doom and gloomy about it, but it is for almost no money bringing uh these resources to so many people who didn't have them before.

I feel like we we always get very hung up on the many ethical flaws of this technology and the harmful ways they can be used.

The positive ways they can be used are just enormous.

Like the reason I'm spending so much time with this tech is that I do believe that it's genuinely useful and it does genuinely provide enormous amounts of value to enormous numbers of people.

Um if you have English as a second language, this tool is phenomenal, right?

You can now, you're no longer cut out of those things in your life, those parts of society where you need to be able to write like somebody who's a native speaker who's had a certain level of of education.

And that's that that that has been completely flattened.

I um like people sometimes say, oh, it's not worth learning to program anymore because the the Chat GPD will just do it all.

I think that's complete rubbish.

I think now is the best time it's ever been to program, because anyone who's coached somebody learning to program has seen that the first six months are just utterly horrific.

Like it's it's so frustrating because you try something and you get this obscure error message that doesn't make sense to you, and you can bang your head against it for two hours, and maybe you give up.

Lots of people do give up.

They assume that they're not smart enough to learn to program.

And it wasn't that they weren't smart enough, it's that they weren't patient enough.

Nobody warned them how tedious and stupid this stuff is.

And now we can give them a tool.

We can say, look, if you get an error message, paste it into Chat GPT, and nine times out of ten, it will tell you what to do next and how to get out of that condition.

That's phenomenal, right?

The uh the the flattening of that learning curve, getting more people.

My ideal endpoint of all of this is I think every human being deserves the right to have computers automate stuff for them.

Like, I can do this, right?

I've got 20 years of programming experience.

If there's anything that I can tedious in my life that the computer can automate, I can get it to automate that thing, but it's ridiculous that you need 20 years of experience to do that.

Like, that should be a universal human ability.

And I think this technology might get us there.

I feel like if we get to a point where people are able to get those tedious automated things just done for them because they didn't have to learn to program first, that feels enormously valuable to me.

Yeah, absolutely.

It's like one of the sort of parallels to that is why hasn't technology sort of pervaded more deeply throughout the the clerical world, for instance.

It's like people are still doing with paper or sort of manually repeating a task on a computer.

Yes, it's in a spreadsheet, but it's not automated.

Why not?

Because they never picked up that programming skills.

But all of a sudden, if these assistants are built into Excel or built into Word or built into what the software they're using, it can be automated easily.

This is somebody who has a very real, very important job to do.

And I think this pattern plays out a lot.

A lot of people with a lot of important things in life end up stuck for a day doing something tedious and manual because they didn't, we haven't given them the tooling that lets them not have to have to do that.

So yeah, so I'm really excited about that.

I think as an educational assessment, it's amazing.

I think one thing that isn't necessarily talked about enough is these things are actually very difficult to use effectively.

And like they feel like they should be easy because it's just a chatbot, but actually, to really get the best out of them, you have to understand the prompting techniques, you have to know what it can do, what it can't do, what are the things that it's going to break on.

I love that we've created computers that are bad at maths and can't look up facts for you, which are the two things that computers have always been best at.

And like, so people sit down with chat GPT, well, like it got maths wrong and I asked it for a fact and it couldn't tell me the answer, because those will that's not what it's for.

But that's really not obvious, you know.

Isn't that the whole thing with the Sam Altman one of the things is it can do math now, allegedly?

So ChatGPT, like um the paid version now has access to Bing search, so it can look up facts, and it has access to code interpreter, so it can run mathematics using Python, which on the one hand, it does fill those giant gaps.

On the other hand, it makes it even harder to use because now you have to know what Bing search is, you have to understand bits of Python, you have to know this is the kind of thing where it's gonna it's got compete, it's got vision support, so it can read documents, but the interaction of all of these features is incredibly complicated.

A great example is um sometimes I will give it like a photograph of a receipt and ask it to add up the numbers in the receipt.

And it will then write Python code that imports tesseract, it will use Tesseract OCR to pull out the numbers, and then it'll try and add them up.

But of course, tesseract isn't as good as GPT vision, right?

If it had in if it had taken that image and used its OC, it's like built-in OCR to pull the numbers out and then pass the Python, I'd have got a more reliable result.

How the heck am I supposed to explain that to anyone?

Like, oh I don't know what Tesseract is.

Kind of.

But you see what I mean?

Yeah.

So having expert skills to use this stuff gets harder.

I mean, using a chatbot is one of the worst user interface.

It's like terminal, right?

So it's do you have any, and I guess it's a final question for me.

Do you have any predictions on you know the mouse equivalent of where this stuff goes?

Because we're not all gonna be using chatbots forever, I don't think.

Yeah, like like I mean, yeah, like you said, the problem with chatbots is there's no discoverability.

Like you've you've just got a blank box to start up.

And they might give you a couple of suggestions, but it's it's it's a terrible user interface.

Um, but that's the thing that's really exciting about the space right now is there is so much low-hanging fruit.

Like you could sit down and just crump come up with an alternative UI for interacting with language models, and right now, maybe you'll invent the thing that everyone will be using for the next six years because there's been we're so early in this process.

There are so much scope for innovation around how we use these, how we interact with them.

And that I find really exciting.

I love that now that people are beginning to understand this tech and what it can do, like we need designers on this stuff.

We need user experience people.

We need that it turns out machine learning nerds are the worst possible people to actually make use of this technology.

Because that thinking in terms of, you know, that that that thinking in terms of, okay, well, I've got to optimize my gradient descent or whatever.

You don't need to know what gradient descent is to innovate on top of language models.

That's almost a distraction from what we're trying to what what we can achieve with them.

Okay, so we we are going over.

I did want to ask you about data sip, so but we're it's quite in a run short.

So can you give us the 30-second what's new in DataSix?

Yeah, we've talked to you about it before, but what's what's hot?

So you need to geocode those addresses.

With enrichments, you can have a plugin that lets you select the address column and say, geocode this, and it'll go and churn away against the geocoder of your choice, and it'll populate a latitude and longitude column next to it.

But crucially, these things are all built as plugins.

So you can have a plugin that does geocoding, I've got a plugin that does just regular expression extraction of things, and I've got a GPT plugin.

So you can say, take this, take this database table, run this prompt against every single row, and then put the output of that prompt in this other column.

And there's one example of that.

I've um it can do the GPT vision thing.

So I actually fed it a database table with 100 URLs to images and told it to write me descriptions of those images.

And I got about three or four paragraphs per image describing what was in the image, right there in my table.

And of course, now I can search against that and do all of that sort of stuff.

So I'm really excited about that.

It means that dataset is evolving into more of a data cleanup and manipulation tool, which is a departure.

Originally it was about publishing and exploring data.

Um, but I realized that the problem I most want to solve, especially around journalism, is if somebody gives you a hundred thousand rows of data, what the heck are you supposed to do with that, right?

If especially if it's slightly too big to put in Microsoft Excel, but you can't afford to hire some programmers to build you like a custom Django Postgres app for this thing, what do you do?

And if I can build plugin-based tools, especially with dataset cloud now, so I can host them for people, where you can upload your CSV file, click, click geocode, wait a couple of minutes as the progress bar fills in, now it's all geocoded, now you can visualize it from a map, that's really exciting.

And yeah, the and the enrichments, I try to make it as easy as possible to write additional enrichments as plugins.

So I'm hoping to see people building their own enrichments for all sorts of other data transformations they might want, might want to pull off.

If you're doing it on dataset cloud, uh cloud, sorry, and I've got an enrichment and I'm going to give you some software.

Uh, did you find a solution to how you can run my software in a sort of trusted way?

And then I've actually, dataset cloud has a feature now where I can basically say, for this customer, pick install this additional package.

So I do have that now.

And also, dataset cloud is I built it on top of fly.io precisely because they offer secure containers.

So with dataset cloud, every customer gets a separate container.

So if you somehow manage to like screw up the security in your container, it's isolated.

Like that's a problem for you, it's not a problem for other customers in the system, which felt really important to me.

Because I know you've been noodling on that problem for quite a long time.

That's like my my my my my ultimate goal.

Because yeah, I want users to be able to say, here is some Python code, run this against all of my data to transform it without risk of them breaking things or whatever.

And I it feels like we're almost there with WebAssembly.

And that would be amazing.

Okay, absolutely.

We're gonna have links to everything and you know, dataset, dataset cloud enrichments.

Those are, I think, the three big things that fans of yours should go take a closer look if they're not already familiar.

Cool.

Yeah, I'll um I'll I'll I'll put together some links to this as well.

Thanks so much for coming.

That was really awesome and illuminating and you know, filled in so many questions around a really hot topic.

So super.

We're at DjangoChat.com and we'll see you next time.

Bye bye.

Bye bye.