Transcript: Postgres - Craig Kerstiens

Hi, welcome to another episode of Django Chats, a podcast on the Django web framework.

I'm Carlton Gibson, joined as ever by Will Vincent.

Hello, Will.

Hi, Carlton.

Hello, Will.

And today we've got with us Craig Kirsteins.

Craig, thank you for joining us so much.

Yeah, thanks for having me.

No, again, I'm really excited to have you on because you're a massive hero of mine.

For people who don't know you, I'm going to say you're the database guy, right?

yeah i think i've definitely been pegged as that like uh basically it's been a lot of time in in

dev tools and around postgres um i i keep joking that like the next time i do something it's going

to be not postgres and back to dev tools but no it's not it'd be postgres again oh no it's uh

third time and i'm done with it but uh no i've been around you know postgres and in that community

a while but um really you know a mix of dev tools and kind of work with a bunch of dev tools

companies know those circles well um spent a long time at heroku um like came on heroku not to do

anything postgres at all i came on there to actually build and run their python team so

um and ended up then doing postgres there again so so how okay so that's a nice little segue well

how does one start i'm a python lead and now all of a sudden i'm you know get further and further

into the python well just one one index at a time yeah yeah um so i guess i can you know back way up

right so i've been at like fortune 100 companies um had joined a small database startup way back

um in the the last recession not the current one but um and uh they were they extended postgres to

become a streaming database and basically do MapReduce on data as it came in. So back

in the, at a time, like complex event processing, CEP was a common thing. So streaming databases

were a thing. There was like AstroData, Netezza, Greenplum, and I joined Truvizo. And it was

interesting. We had, I think MySpace was a customer. Like if you remember back when MySpace

was like hey i do i do we do yeah that's that's where i learned the web on myspace um and uh like

we were handling all their ad traffic right like how do you like process all this in real time and

manage it and report on it and uh so i joined there they were 15 people they grew to be about

30 people dwindled back down to 15 and was kind of ready to move on and did some consulting

bootstrap my own thing um actually in the wedding space it was one of those like scratch your own

inches right was engaged getting married at the time and uh was like built this product didn't

know anything about marketing and like got written up in like the wall street journal like top wedding

apps of the sea like it's like oh wow cool um and then i got married and i'm like i hate this

like i was literally going to meetups talking about the season's dress color and uh uh i had

known some of the heroku folks had been there did some consulting with them and uh but it's like

it's all ruby right and i'm like i don't want to go work at a ruby company like hiroki was the cool

thing at the time but myself being more in like the django world i'm like yeah it was like this

is a cool thing i'd heard about right but i'd never experienced it and uh so i show up there

and uh i'm doing a little bit consulting um some business analytics stuff and other things for them

and uh they're like so can we show you this thing we've got this like build pack thing and you know

when i was bootstrapping my own thing we spent two or three weeks like getting our deploy process

good and like cool you tar it up you ship it over here like everyone did this right

and like to have stable builds and releases and i forget we were running on a line node vps or

something you know like everyone did back then and uh i was just like wow there was three weeks

of my life i wish i could get back because i could have just get push and it was up and running

and uh yeah python was slated to be like the i think the next the second or third language

and it got pushed back because of java and others so it like ended up being one of the last major

ones released at the time um but i was like okay who's releasing this who's driving this and they're

like oh we don't know like we just have this thing like we built it and here it is and i'm

so i ended up joining to like i was the first kind of build pack maintainer there then i hired

kenneth because i'm like i'm not actually a good python developer like i know that ecosystem and

world but like there's so much better out there so i i joined heroku to kind of focus on that

not postgres at all okay okay fantastic can i can just go back and ask so how did you get into

programming initially do you sound did you have a like undergrad degree like how why why why did

python hit for you because this is before python was kind of the hotness um so i started coding at

like age 10 like um to me like code's always been an interesting means to an end like i i'd like

solving problems but like to me that intersection of like code and technology and business so like

my my degree in college was actually over in the business school but a minor in cs um so like i i

i generally say in like you know the best of the best like developers i'm i'm like i can't keep up

but like put me in your average eight to five engineering job right like i think the people

listening to like the podcast right deeply care but there's a whole world of developers that like

that show up write some code and go home and don't think about it right and don't care if they get

massively better and i think it's i'm kind of envious of that at the time i mean i was just

gonna say that doesn't sound i mean maybe not to not caring about getting better but the rest of

it sounds good yeah i mean like i'm kind of envious like hey you can turn it off right for

me uh you know i'm out here in the bay area and it's like everything is code or tech constantly

so i i appreciate the like you know the family friends that are doctors and nurses and not in

tech right um so i don't i don't subscribe to the idea that you have to to truly to be a good

developer and contribute you don't have to like live and breathe it and say this is you know the

end all be all but uh yeah to me always that intersection of of tech and business was

interesting um we i think it was a colleague of mine daniel farina that like he was like

yeah this Django thing it seems like it has good docs it's like well opinionated it's well

structured I mean this was back in like was it like 092 maybe that we got started on it like

early days um it was kind of data rails and it's like the docs on Django seemed better so

um and I think it had the time the the batteries included tagline which I loved like it's like

great I need auth I need an admin panel I need like it was all I mean all the things that are

still true today were incredibly true way way way back 15 years ago yeah right i mean um that talk

about how you might you might as well ask for a pony because it's got you know it's got all these

things that like now it you know when people say well where's the pony come from and you tell that

little story they sort of glaze over a bit because they're like well every web framework's got that

but you've got to understand that no really every web framework didn't have that yeah i mean it was

you know you look to the ruby community in the rails community and there's like still five

different off and finally an off gem one out but for the longest time it was everyone was changing

every you know few months yeah and ruby's still part of the i mean you know rails is still very

much batteries included full stack framework right so you expect yeah i mean to me i think

jangle goes even further like rails is close to it but like misses some of those pieces um and i

have to have to be careful because you know i have many many friends in the rails ruby community

who do i we had dhh on here so we you know we're we're all compatriots trying to solve the same

problems i mean okay so what we've got what you're talking about rails i mean one thing that people

often say about rails is it's more opinionated um i think it so it's more opinionated i think

there's less batteries included i think there is right um like there's not an admin dashboard

right there's not user management there's not like there's there's a few things that

now it's standard oh yeah you pick this up off the shelf but i mean even 10 years ago that wasn't

i decided you know like how do you do user management rails i was gonna say one of the

things um that rails had as an advantage was that heroku you could pretty much just start it and

push it to heroku and it worked and that i think the rails book you know the first chapter i remember

when i was learning django back 10 years ago was just like you know do do do boop you're live and

there's nothing like that on Django so like I have a couple books on Django and my goal was actually

like I want to get you as quickly there as we could but it's still four five six steps just

because so that I think comes to I like that distinction you made of batteries versus a pin

of you know guardrails but definitely feels like a beginner still thinks rails is like

gets you what you need it's like kind of but then once you actually know what you're doing

you're a little bit more on your own but but that's um anyways we'll talk later about this

But there's the whole new post-Horoku hosting platforms.

And I just wrote the Django docs for Fly.io, which is one of them, because they didn't have any.

But it's interesting how there's not quite a Rails bias.

And there's also you don't use Git at all.

In fact, if you don't have a Docker ignore file, your Git will be pushed up there.

So it's a whole kind of different.

It's similar, but quite different in subtle ways if you're used to the Heroku way.

yeah i i think we're gonna have a few interesting years you know i don't know if we want to get into

it now or later on but like a few interesting years where i don't i don't know what the the

i mean and heroku's still there right the free tier is gone but um i think it's you know it's an

interesting time and to me it's kind of the a little bit the writing on the wall right like

i saw someone that was you know ex heroku used to work at heroku and was you know thought it

was the right thing to do to get rid of the free tier and then came back a few weeks later and was

like hey i just ran like a rails uh girls boot camp and like i didn't think about the implication

of this like like developers learned on heroku like if you were at a jango boot camp pie ladies

like i mean they every pie ladies boot camp ran on heroku that i i'm aware of like i helped with

a bunch like we sponsored stuff and it was like here it's just easy right like getting a dev stack

is still a huge pain and you didn't have to have a full dev stack to be able to push it up to

heroku and see something at the end of the day yeah every con like you know my books everyone's

everyone's stuff is broken now um or not broken but you have to pay and it's less than having to

pay i mean i think like we've carlton carlton and i've had some discussions like there's a valid

thing to say you shouldn't have unlimited free because there's a real cost and there's fraud

but um you know not like they all those one of them told us like they all want to be the second

place right they want the marketing benefit but not the hassle of being the free place but it is

a yeah it's harder it's harder to learn um i mean i i'm trying to tell people like i'm switching to

fly just to my books because i need to pick one like you know you hopefully you paid for this

book and didn't steal it so like paying a couple cents to host a site like you know that should

feel normal right like it there are real costs involved but it is just a drag on the whole

cycle and to your point it's going to be several years before there's a clear winner right like a

lot of these are series b and c they've raised money probably one or two will get acquired one

will win and the rest will go away so in the interim it's frustrating for people teaching

beginners yes carlton well i i said i mean unless you want to say something to that point craig but

i wanted to ask you more about the dx side of it the developer experience side of it so heroku

clearly got it right and from your perspective which is an amazing perspective having been there

and worked in it and you know created it what do you think um sort of like the key things that

people should look out for or that are missing from the the new ones or you know what do you

think the sort of the holy grail is like the the you know yeah there's two pieces in there right

like at heroku we we agonized over dx like we spent so much time discussing and debating

you know like i mean at the time requirements.txt was not a standard right okay like you go i mean

i remember speaking at a pi code comp in miami asking how many people actually had a requirements.txt

in an audience of python and there were like three hands um it was like it was not a thing

and and it was like but then you go and talk to people and how did you manage like a reproducible

build and it was all over the map um and now it's like you know whether it's you know

pipin or like what i mean i know there's plenty of debates on but it's like okay requirements.txt

generally works as well and like well enough a standard now right without it being as that i'm

not aware of it as a standard but like it may be right like codified somewhere but we agonized over

that and kind of we looked across language communities and said what do we need to push

as an agenda to like not for us but to like better improve how can python go and learn from ruby

and how can ruby learn from java and how can um database url was a canonical thing and like i i

had yeah i forget how many debates i had whether it was with jacob or andrew or who they were like

no you just do this and i'm like no you shouldn't be putting config in code um and they're like yeah

but it's it's like and man it was some hard-fought battles um and like persistent connections in

django uh i remember at a a django con europe standing up giving a talk and android argued

tooth and nail with me that like it's not a problem and i showed him the chart of like latency

to do tls i'm like okay do you connect to your database with tls and he's like yeah you should

it's secure and i showed him the tls negotiation and it's like oh here's 100 milliseconds and like

that's just to grab a connection he's like like and and i asked how many people uh and so this

is andrew he knows how to run production postgres right and i'm like all right how many people are

using pg bouncer because that's what you should do he'd tell you to do that and there were like

three hands in the audience out of like 400 people and it was like i i loved publicly trolling him

right like he's sitting there on the front row and i'm just like i'm giving him this like subtle

stares i'm asking the question in the audience because it's like no it's not a problem this is

what you do and everyone says to do this and we agree that it's the right thing to do and no one

does it so um being highly opinionated and just pushing for the thing that was right right like

taking the long database url wasn't a standard for three years right and we just gradually pushed

and nudged and had the conversation and and built the community and the connection right like i i

have a lot of very good friends having never contributed a line to core of jango at all

that like when they're in when jacob's in town like he's coming over for dinner to catch up

um and uh i mean that invite goes broadly out to like anyone i know in the dev world like i'm very

happy to like like it's a small world and community that we're a part of but um really pushing for

that like this is the right way to do things and and just seeing it through it not giving up on it

um and that's a hard thing to measure like yeah i you see it when you know it and if the answer

is go to docs or oh here like here's this error message or here's how to treat like it should just

work like it should just fade away and be magic okay good that's really nice because that's what

One thing I sort of see with a lot of the new batches,

they're trying to cover every possible use case

and every possible way of doing it.

And because of that, it's quite complex.

There's too many moving parts sometimes.

It's hard.

And I'm excited by Fly and what they're doing.

I think a lot of what they're doing

with the networking stack's really interesting.

Like there's a whole bunch out there

and some are more exciting to me than others.

And they're exciting because they have an idea

opinion and a vision yeah i don't think any of them have nailed the same simple developer experience

of heroku yet well they don't you know maybe this leads into crunchy data i don't think any of them

truly have managed databases yet um which you know is like they have they'll give it they'll

give you postgres but it's just a running in a vm it's not not like heroku has it and not like

dedicated services like crunchy data and it might send my just from the outside senses it's taking

them a lot longer to do that than they thought it might it's it's it's hard and i wouldn't wish

it on it it sucks um like i wish i had someone to run my databases for me um i mean at the time

at heroku we we joked that we had all these rails developers asking for database and we thought how

hard could this be turns out it was a whole lot more work than we ever expected um like wally

which did all of the like underlying disaster recovery continuous protection stuff um

like we were good and lucky like we wrote that in a weekend thinking hey we needed something like

and shipped it luckily about a month before the great ebs apocalypse i think that was april 2011

i don't know if people remember when like netflix read it like the entire internet was down for a

week it was terrible and like a lot of times like when heroku would go down in the early days like

we would we would go down and we'd check and see like is reddit up and netflix up and if they're

down too like we're screwed like we just wait for like can yeah it's the whole availability

zone that's gone it's not just yeah it's like northern virginia is offline and half the web

is gone yeah it was i mean that happened like we i mean aws was offline for a week in april of 2011

pretty much um and we had shipped wally just like luckily a month before and so we were able to

recover databases with no data loss whereas rds just told customers like sorry we lost the volume

like back then yeah um and so i mean i'd like to say that was amazing foresight and like we

knew what we were doing it's like a little bit of lucky timing and uh i mean now we very much

know what we're doing like a very experienced team that you know at the tail end of our time

at my time there it was after five years we were running managing like 1.5 million postgres

databases we we joked that a one in a million problem was once once a week and that was across

eight engineers so very small proficient engineering team uh but uh i know kurt over

at fly really well and his viewpoint is like it's at this point it doesn't make sense for

a pass to also run a database as a service like they're specialized enough like it's a distraction

i think i think we may be past the point where like if you want the best service for postgres

you come to someone that does postgres you want the best service for reddish you go to someone

that does redis you want the best pass you go to the pass for the past and it's um i know kurt's

gonna you know ping me after this and be like all right craig so that means you're gonna run

the databases for fly right and we we go back and forth every few months and we have customers that

use fly and use crunchy and like i i've i've run a pass i've worked at you know heroku and and ran

a lot of heroku postgres but a lot of other areas of post of heroku as well um it's a lot of work to

do either one of them and right now we're really focused on postgres really really really well

okay so given that we've talked about crunchy let's dive in there so this is a managed database

service a bit like rds but you know more um as i say boutique more bespoke more so i'm a big fan

of rds i always recommend people get a managed um postgres why should i use crunchy data what's the

you know what's the yeah so i mean i'll i'll uh i'll give a few reasons right like um

one we contribute about a third of the upstream code to open source postgres itself right so

you know if you care about supporting open source right like we contribute to a lot of it right if

you're like how do i contribute to postgres well guess what we we employ people that contribute

to upstream postgres about a third of the code so if there's a you know there's sometimes not a lot

of love lost for amazon right out there making a ton of money how do i how do i contribute in a

meaningful way to open source supporting us is is one um we we are just postgres like at whether

you go to you know microsoft or google or rds like they they have people that run databases and they

have you know app dev teams and all sorts like we are just postgres and we also understand it from

an app dev perspective like i've been trying to make a positive impact on postgres and for app

developers like at heroku we contributed a lot to get jsonb committed to um to postgres itself um

we at situs when i was there we donated a portion of our equity to the postgres community and

foundations so like when situs was acquired it meant money for the postgres community

um so like we're deep in there and because of that you're going to get a great postgres

experience within rds like you open a ticket and you you may or may not get to the product team

like yeah no you won't no you won't yeah i'll say oh even if you won't say that i'll say that

you won't and for us right like if there's a bug in postgres like i can reach over to a colleague

and have them upstream a patch to it right so like there's no issue too deep there um i i was

told by our marketing team i can't say like our support isn't crap but like our support isn't

crap like we give really good support like um i tell people like open up a ticket ask a question

like um and i think we have a unique like it's not on the product side necessarily i think it

sort of is and sort of isn't in that like i get app developers like and app developers

are terrible with your database like yeah it's true like how many don't run with pg bouncer and

you should and like even if you have persistent connections in django you're like no i set my

connection pool i'm like that actually wastes memory in postgres like you've reduced latency

of establishing a connection but every idle connection to postgres is about 10 megs of

wasted memory like oh so i should run with pgbound yes but like one connection pool is not the same

as another connection pool um yeah like they're like which one do i want i'm like both you want

both of them uh but you're talking about performance things i mean i remember i was a startup called

Quizlet, which has decent traffic. And it was MySQL. And at one point, we didn't have a lot of

permissions for who could go in and do something. And the founder went in to change a record and

dropped the entire database, which was an oops. We got it back. But then after that, we had Percona

on the highest level of support to your point of people saying and not doing. I mean, if you're

running a website or running a startup, you're definitely not focused on the database to the

degree you should be just because you have a million other things to do yeah no i mean i i

laugh a little at like situation but like i've seen it before i've had customers friday at 4

p.m be like uh-oh we ran our test suite against production what's the first thing in the test

suite like truncate database like oh and i'm like all right take a deep breath tell me exactly when

you ran the command right do you have and we can like restore your database to exactly no database

database data loss like it's going to take a little while but like we can watch the right

ahead log replay and see when it like the exact time that it happened and give you the exact

moment before you dropped all your data and so like we have your back there i mean we've got

some stuff that detects like n plus one queries in your database coming soon like none of you

in the django orm have ever written an n plus one query at all i'm sure right yeah user.groups.name

and it's it's not that like developers are bad but you've got other things to worry about right

you can't be an expert on everything and like i found the django community loves postgres

but also you you're missing half the features and half the perks of it because you don't have time

to dig into it well this as well was one so one of the one of the things that crunchy data's like

on my radar for is that you've got um support for uh pl python so the you know embedded python in

there was rds like you go down their long list of which supported pls and python's not there and

it's like but i you know i'm not going to go and learn this one in order to write that stuff

so there are more features there as well like you can get a richer postgres experience yeah i think

it sums it up like richer postgres like because we are everything postgres because we believe so

strongly in it like i want to give you everything that postgres can do right like i you know when

we shipped pl python i'm like i wonder what if and we so we ship it also with sci-pi numpy and

pandas in there and i was just like you know what i wonder if i could write a recommendation engine

inside my database i i started live tweeting this like when we we shipped it and uh i was like i

should have tested that i could do it first so like could this how wrong could this go

and there's like 20 lines of pl python and two hours later like here's a recommendation engine

inside my database and there's all sorts of reasons you shouldn't put that in production

but maybe you should yeah no well well this is one question is how much how much of my application

logic should i move to the database because i remember years ago and it must have been years

ago because i was using mysql and i learned about triggers i was like this is going to be quite good

because you know round trip to the database that's costly let's see if i and it was fine to sort of

start with but then it started getting out of hand and I had no real management of it so the

things that I missed and I still don't know if they really exist are source code control or

version management and like testing decent testing and so like I you know one question I have for you

is okay let's say I use PL Python how much of my logic should I embed in there and how can I have

good software engineering practices so that I can roll back that I can test I can sure and are there

frameworks to help me do this because if i was doing this in python i just use django if i was

doing it there's a little bit more than there used to be but it's still not perfect right there's

still not the easy release process version management as much as you'd expect um testing

there's a little bit more with things like pg tap or like you think about the um the other kind of

supporting pieces right that i can go and take a fork in my database so i can grab a snapshot of

production down to staging and i can run this as it like we have more tooling around the database

of like, I mean, we have customers all the time that take a copy of production, move it to staging

like via automation on a Friday night, slim it down. So they don't, you know, developer, all

their developers aren't working with a one terabyte database, but it's like, here's the latest 10 gigs

of production data. So we have realistic test data that we get every Monday morning, right?

Every, every developers get a real copy. And I, you know, I think they built it in less than a day

with all our APIs and automation and the, hey, we give you these constructs to build and work

how you want, right? And we see organizations work very differently. So we haven't said,

here's the codified one way to do this. Now we have thoughts of how it evolves, but

it's still, I think, early for that type of CICD for your database, right?

So it's not perfect. And that's why I don't think you should always go and code, swing all the way

back and do all of the logic in the database. I think if you're doing data intensive operations,

that's a good line where you can do it easier in the database. And my Postgres, I think, can do a

lot. And so it's start there instead of adding another tool or system. Full text search, start

there before you add Elastic. Because how are you getting the data into Elastic? Now you've got two

systems to maintain. Geospatial stuff, PostGIS is great. You start to talk about a recommendation

engine and it's like it's 20 lines of pl python and it's going to be slow and not great but you

refresh it every hour or something like that right now if you were to do this otherwise you're going

to do be like okay i'm going to go and install debysium and do change data capture from my

database feed this into kafka i'm going to run some model and spark and now i'm going to cache

the results in redis and now and i like that's not an exaggeration from 90 of companies and now

i've got five systems to maintain versus like 20 lines of pl python inside my day like and your

point of like well how do i version it how do i test it well manually like it's 20 lines of pl

python as opposed to five systems right yeah yeah okay that's super as well because that's that's

that ties it i saw you tweet something about how um like postgres and either rails or django you

know but then being like the money stack like if you just want to get something done it just works

like it's it's it's it's um it's is profanity okay on the podcast or not

you get the idea get whatever done right like that's that's what it is um it's like i just want

to and one fun thing a total kind of non-sequitur but like one fun thing for me of like running a

past or databases of service is seeing everything like people build like i talked to one person

who's uh uh he he surveys snakes for canada like that is his job he's like half like engineer

and half like out in the woods surveying snake and populations of snakes and he uses like

post gis because it's like the best geospatial database and he's like yeah i'm not a real

developer engineer i'm like you totally are like the stuff you're doing here is he's like yeah

but i'm not a real eight to five software i'm like because he's his background's in snakes but

he's using like post gis and so getting to see all of these fascinating things like come in full

circle right to me like tech is a means to an end it's a fascinating one and so by being at kind of

the dev tools database as a service intersection i can see like um you know we have one person that

our team jokes is like the cow social network that like tracks cows like it's an app that like

tracks their eating habits and other things and it's like who how is this a business and a startup

in like what and but we know this like walk down the the street and like walk into a retail store

i'm sure there's a problem you can see tech solving so it's it's fascinating to be at that

intersection and like django postgres django or rails to me they're very similar just just work

like they're not new and shiny they just work and now i can go focus on solving a problem

yeah and you haven't got 15 bit moving parts which then you need three engineers just to

keep the moving parts in the air never mind about write the application logic yeah yeah absolutely

and so that's why i i mean i totally appreciate the question right i'm like okay well how do you

test this how do you version this but sometimes you may not if it doesn't change once a year and

you you know how did we how do we test things before well we with like extra code review and

eyes on it and like you know manual testing it's like oh my goodness crazy that we could do those

things again i've got such a little evil devil on my shoulder going oh yes yes

it's like it's like when i got the remote um development extension in vs code and all of a

sudden i can so i can just open a folder on my server now oh i can i wasn't good all these things

in balance right if you've got a 20 person team how are you doing this how much is it changing

this is you know we used to like rsync files to deploy right like like hey guess what it still

worked back then like we we brand websites on the internet same as we do today um just fine

okay well i wanted to um ask you one more thing about um crunchy data and perhaps we can swing

around to um you know perhaps a few more django questions that we've got lined up but um the

The question I had was, I saw you recently,

so when you first launched, I was a bit like,

it's quite a lot more expensive at small scale than an RDS because a small RDS costs you I know

eight ten bucks a month and you know then you can scale up as you need to whereas Crunchy Data was

more like in the 30 40 dollars which is absolutely reasonable but you know it's for a smaller project

that's perhaps prohibitive but then last month month two months ago you did you went for 10

bucks a month or something for the minimum which I just think is a power move so tell us about that

yeah i mean so initially we were really focused on like being a production ready and like data

absolutely backups disaster recovery monitoring metric like everything you truly need right like

pg bouncer is built in like you don't need to go and layer it in um i don't know of other providers

that give it just right out of the box i think heroku kind of has an add-on for it now um but

like on rds there's this separate rds proxy that you go pay for it like no this is what it takes

to run a production database and um we're running a lot of very large production databases now i

care about hobby developers but we needed to nail the the advanced absolutely like we now we're going

kind of down market that it can be there for everyone and it's you know as you say 10 bucks

a month for a full postgres database with pl python and with you know post gis and you can

create more databases heroku you know when they had their kind of like lighter tier plans it's

like here's a million rows and i'm like a million rows what is that like um and it's one you get a

shared multi-tenant database thing this is your own dedicated instance um and then the key point

the key point there for the hobby developers is i can create as many databases in that instance as

i want and so you know if i've got if i'm running 50 micro projects hobby micro projects i'm still

running it on the same one 10 buck instance and then that that's as close to zero as you're ever

going to get like well so we also it's not technically zero but it's effectively zero

and so uh you know we were talking kind of like about pile ladies and some of those workshops

uh if you spend less than five dollars in a month we don't build so it's not a five dollar credit

but if you're below that five dollars in a month like great it's on us um and so you can go and

like suspend your database and idle it so if you're like building and developing right and you're like

hey i'm um we've got a couple of universities now that are pointing their students at us and like

great idle it like 5 p.m and on the weekends and like it's there and then when you're you know

developing and learning and building out your app or your project turn it back on right and then if

you want to hey if you want to have your portfolio live to demo and show off what you can like great

it's 10 bucks right it's man like less than you paid for the textbook right um but then archive

of it it's sitting there and you're just paying for the storage and if you're below that five

dollars it's it's there on us see i think that's i just think that's a super power move i think

that's you know that's going to grab a lot of people's attention i think and rightly so i think

when i saw you put that out i was like yes that's good move so absolutely so can i ask about because

i'm interested in how people learn how to code are these do you have more insight on like these

these universities like is this their databases course is this like like an intro like cs50 kind

of thing do you have like how are people from where you said how are people kind of learning

how to code these days because i'm always curious and can't it's i never get a consistent answer

from people on like how they're actually like like building a website right still many universities

don't have a course on it if anything it's an adjunct or like it's not a core thing web in

like an undergrad cs degree i i've talked with you that seems to be a little more common now

and it's you know i think it's very much the basics of building a web app to showcase something

right um one of these is actually in a i don't know if it's an mba or a master's program i think

it's a master's program but it's typically like people out in industry so it's not like i'm

getting a cs master's like coming back and wanting to kind of level up on their technical skills to

kind of advanced their career right um so and this one is like on the geospatial side and so

geospatial it's like you do anything geospatial you're using post gis likely and so like how do

you build an app to showcase this and it's you know a lot of the apps are you know simple little

apps but it's hey i can showcase that hey i'm doing this analysis here i want to show off a

website not deep deep on the sequel side but they touch just enough of it right where hey the

databases here um i'm definitely seeing more universities teach here's how to build a web app

and but the the problem is man like the way they they plan out a course right they'll work on a

course and develop it and um plan it and establish that curriculum for like two years and they do it

and you know guess what everything about that we talked about on web development and how to build

the web app has changed like um what is it like remix and next js now are the front-end frameworks

and i'm like i i don't even man i miss jquery right like i like i don't know htmx yeah it's

htmx yeah but yeah so um i i think it's hard for like universities to like how do they with the

pace at which things change how do they keep up and do all that right i don't know that i have

should use Django the answer is they should use Django well but even five years ago well like I

had like a Django con there you know a couple professors who are using want to teach Django

and are looking at my books and you know yeah to your point they have to have it they're planning

now for next fall and it's like well the you know the book's gonna change like Django might change

a little bit and it's not not so much if you've been in the space but if you're a beginner a

newcomer it's enough to kind of break everything and they can't really plan around it yeah and

And I mean, for me, especially on the front end side, that the pace.

Oh, yeah.

Well, forget about forget about that.

Yeah.

That's the one that's like, oh, man, it's just like anxiety immediately.

I don't know anything about like the front end world anymore.

Like, like it changes so, so quickly.

Can I ask you, so you're sort of unique in that you you're not a pure Django person, though.

You have a strong background in Django.

Like, are there other frameworks that the Django community should look at in terms of, hey, there's some good stuff going on there?

databases or otherwise like i'm constantly looking at um laravel impresses me um in terms of being

batteries included but from where you said are there other ones beyond you know rails that

django people should look at i think you nailed it on laravel that's the big one that comes to

mind of like a similar and uh like kind of modern you know php is a new thing like yeah things i

didn't i and i so i i launched the python support at heroku i also launched the php and the go

support so um and it was you know the php one was like what what really like heroku official peach

it's a different world now right um it's huge i've got a buddy who's a consultant and he's

constantly i mean it's it's you know i don't want to throw out stereotypes but it there's a lot of

money in in php and a lot of people tech stacks that just work i mean hosting is easier right

they've got you know laravel itself has forge or whatever it's called you know so whereas in

django it's like you know so um i mean on the you know the front end side i think next jx uh next js

and remix um like we we've gotten on the remix run like bandwagon ourselves and i i'm not knee

deep in it but it as a front end framework it seems like next and remix both solve some of

the problems that i think existed with vanilla react and and you know we we complicated things

on the front end and it feels like they're getting a little better like part of me is still confused

it's like why do we need a state machine on the front end like why why doesn't like request

response just work i there's there's people in that world that'll educate me at some point like

i'm i'm sure there's a good reason i hope maybe but um it's just the pendulum just swings right

yeah it entirely does right and i feel like i'm an old curmudgeonly like uh you know yeah just

use django and postgres like build your app make your money solve your problems and like you don't

need anything else um uh but i also think that's that's fairly true like you can live in that world

and it's great you don't have to relearn something every 12 months um not to say you don't learn new

things in that that stack like postgres advances django advances but um that stack works pretty

well i was gonna ask are there obvious things django could add to our official postgres support

i mean i think the last the last big one was i think full full text search um i mean we've we

django has felt historically like we couldn't pay for code or really push things absent a kickstarter

or some things and we found out through the python software foundation that's not necessarily the

case so this is one of the things is we're kind of looking for doable you know three six month

projects that django can support and especially around postgres so what we have what cool features

does Django not have support for that maybe we should it's a great question I feel like Django's

done really well like the the JSON support's good the full text search support is good um I I don't

I don't know right all I mean it's I don't know that it fits in Django but maybe it does some of

the stuff around like um constraints like I don't like um like enums aren't you know there's an

internal slack discussion about enums of like hey has anyone seen them out in the wild these days

with postgres and the answer is most uniformly no um it's like use a check constraint instead

because then you can easily change it and relax it and you don't have to do a table rewrite it's

not as invasive like oh okay like i i don't know if an enum and django maps down to like

a check constraint which is maybe a more modern way of doing that right or um exclusion constraints

with range types how do you say like you can't be registered for a university class that you

know overlaps times like range types and check and exclusion constraints are amazing like if you

actually had to enforce that in code like you're going to wind up with like race conditions and if

you remember back to like university scheduling it's like no no it's just like a class with 30

people how like there's no way two people would be registering at the same time and i remember like

oh yeah my registration opens at this time and i i want the 1 p.m i do not want the 8 a.m class so

i'm like there when it opens ready to go and yeah did you just fill up a class because like they

both had their session working and you know you know multiple web servers and like so like relying

on the database i think a little bit more for like constraints is a big one constraints are

so valuable and yeah go ahead so that so it's good you mentioned that because that's one of the good

feet that has come on quite a long way and um django for the last few versions um so we first

all unique constraints and now you know i have to check exactly which you know which constraints

we've got covered there um but um django 4.1 now that has it so when you do validation on a form

the database constraints you've got checked for that form validation as well so it goes back and

we'll go and so it's like it's not we first of all it was like yeah you get an integrity error

when you save because you violated the database constraint but now you get the nice form error

and you can go and handle it and it's like yes that's so that's lovely that you say that because

it's one of the features that's been pushed hard over the last few versions and has really flourished

so that's lovely i don't i feel like django in general does a great great job um so um one one

question i had on this topic was one thing that's always really interests me about postgres is being

able to um take a create a custom type so you've got a um a relation and then you can use it as a

field yeah yeah in another relation it's like i've never sort of sat down and thought if that

would be if that's even feasible to make work at the django orm level but that would be kind of

amazing yeah and that's one of those things where it's like when you understand that you're like oh

man now what can i do with this and it's like wait a few things not as many things as i thought it's

like um and like i mean i've seen it used out in the wild but not as common right like it's one of

those like all right this is a a deep database hack thing for a very specialized case it's not

one of those i would be more scared if django supported it like easily in the orm that it's

like oh man we've got people now building monstrosities um i i so i mean i i could

actually see a reason for django not supporting it because you know cool i've built this table

and now i want to embed the table and embed the table inside a you know another column in another

column and now i built a no sql database inside postgres that's also relational like it i could

just be see it being turtles all the way down okay good good that's a good answer don't go there

i wanted to ask you um some fun questions about personal coding setup if you don't mind like so

let's maybe we'll go quickly so text editor of choice these days uh usually sublime text

oh okay nice um mac or windows mac or linux yeah okay good man um get visual editor what

do you use to see your git i know command line all the way okay like that um but what else what's

another like cool personal like tool setup you have that maybe others aren't using um so i mean

i you know uh my i don't spend as much time in code right so my stack is like notion linear uh

for like project management um superhuman for email um i think my like my default these days

is like when i pick up a new tool uh if it doesn't have like a command palette like a command k

like then cool i'm not i'm gonna like look for another tool so um it's actually a thing that

we're building into crunchy bridge i'm hoping it ships early q1 next year of like command palette

inside crunchy bridge where it's like great you want to find slow queries command k database name

slow query is like you can imagine all these things that like i forgot where to go into what

to like queries over five minutes right on your database just command k start typing and it'll

take you to that screen where you could kill a query so i'm i'm stoked to i think maybe be the

first depending you know who listens and tries to outrun us and ship it um yeah i'm a big fan of

tools with like a command palette to just like i can be a power user but also like this take like

this as an average user turns me into a power user it's discoverability right because you just start

typing and all the possible matches come up it's oh it's that one right brilliant yeah how do you

think so the fact that you mentioned that crunchy is like around a third of open source contributions

to postgres like django doesn't have anything like that we have massive companies that use it

that it's sort of like the bigger you get the less you're you contribute to django like

any any thoughts on how we can do better about getting some of these massive organizations to

actually that are using django because what happens i think is that they kind of get off

their versions and then they have a custom fork like instagram has this and they're kind of not

really using django itself or i don't know like it it we we'd love to have them you know that's

such a great thing for postgres right we're always like like postgres had a table at django con like

django has no tables anywhere right like postgres seems to have figured something out there that

django needs to well but i think it's uh it's not the big companies right it's you know the

smaller ones that are doing kind of like you know we're a hundred person company right like

right look look at aws right like are they running a proprietary fork of postgres that they should be

contributing things back yeah probably like like there are instagram right or instacart or like

um so yeah i mean i think there's i don't know how to get the big ones too right and i think

that's the reason of like hey how do you support and sustain open source it's you know some smaller

companies building models around this right um we have people that are full-time working on

Postgres and like, we're, we're, we're a hundred person company.

And it's like, you know, can we take that person and that salary and build

it back into product or otherwise?

Sure.

But we're committed to Postgres.

Right.

And I think there's an opportunity for Django, um, you know, the mix of, of

consultancies and smaller shops, like to, to do that, right.

And, and product companies, right.

Like at Heroku, we were for a little while headed down that path.

Right.

And I remember way back at Heroku, there was, there was Jenge and there

was um uh what was andrew's uh startup um i forget oh uh no no before that he was building

a python for heroku oh i don't oh i don't know about that yeah so heroku for python um i emailed

them ahead of time there were three of them that existed um it kills me that i can't remember now

what the name of it was um but uh yeah how do you build product companies around you know django

that then also contribute back that's a good question i think i'd say postgres is a little

lucky maybe a little bit of skill and then some some super intentional kind of companies doing

that can i can i ask is there any um sort of business case to be made that it pays for itself

in terms of expertise and you know the ability to react to problems more quickly and so that

you're paying for these engineers to work on Postgres but because of that you have in-house

expertise that's at a significantly higher level than you would otherwise have I think so I mean

it's the pitch I give right like you can't but right but does does it does it does it cut the

mustard with the business types do they go oh yeah actually yes yeah I mean for for us absolutely

right so i think it's a case by case but it's like you know you you don't need a postgres

contributor on staff as a django development shop right does that mean you can't discover

a postgres bug and need help with it yes now guess what even if you wanted to pay the salary

of a dedicated postgres person as a django shop you you couldn't hire them because you don't have

interesting enough problems like you you uncover one bug a year right it's like they're bored with

to work now as a django shop right man having someone that's like knee deep in django core

and you got you know five clients simultaneous projects like great oh yeah here's an interesting

one we heard this from users right here's where we can go and upstream something like

that model can work really well and what i tell our customers like why do you pick us over someone

else like great when you have that postgres bug that you could never get to the bottom of like

forget even if you you could if you wanted to pay the person you can't hire the dba because you

don't have interesting enough problems but guess what here like your problem is interesting to them

and like let's pull them in right and so when you need that that one week of the year you've got it

with us yeah i mean i've done a lot of freelancing and one when i started contributing to drf in

particular it absolutely changed the dynamic because instead of going to shops they'd be like

oh you sure you can do this they'd be like oh you maintain that that dependency that we're building

yeah and that just changed everything massively i mean zapier they have um simon charette who's

one of our big orm contributors like you know he's but they give him a lot of time to work

on django itself and they must obviously see the benefit of doing that yeah so i mean it's a model

that i think works and i wish i would see more of it um we we did some of it at heroku um managed

to do some of it at situs and and when i was at microsoft a little bit and and crunchy were doing

a ton of it you know um at heroku we did it across various languages and other areas too

so i've seen the model work outside of the database world um it it takes some buy-in right

at the business level um and it takes interaction on the technology side too like it i found that

like hiring someone just to work on open source off on an island isn't as good it's like you gotta

you gotta get the communication like what are you seeing out in the field what are customers like

what problems are your developers right how do you get that communication like communication's hard

you've got to build that in and then then it really shines okay so what's the ai disruption

for databases right like what's the co-pilot or is there one coming uh there so much is already

there in postgres right like you point me at a postgres database and i can tell you like

you have this query that was probably generated by this orm and it's like missing with like

uh select related right like i can look at the query from inside the database pg stat statements

already has it and so i and you know i'm optimistic we're we're doing a lot of r&d

and investing there to be like hey i can tell you like from not knowing you are a django app or not

knowing you are a rails app because like django has these conventions on the tables are named

this way the join looks this way i can probably tell you to search for this in your app code and

change it and now this will improve your your performance so um that that's like my favorite

like there's like a a meme of like you know ai and then like you pull back the curtain it's like

if then if then like you know but it works it works so well so i mean i feel like we're really

close to a lot of this with postgres and so um i'm maybe oversimplifying it but i think it's

like a lot of this simple stuff goes a really really really long way and is that what you're

excited about coming up in the postgres world you know if i would say you know what's what's on the

horizon that we should look out for for the postgres i mean so for that's definitely what

i'm excited about for crunchy and what we're focused on and like coming soon from us from

the postgres world you know the pluggable storage stuff right like postgres got pluggable storage

backends a couple years ago but we haven't seen like another predominant one emerge so like there

z heap for a little while then z store um there's there's a few others we haven't seen one win yet

so i but those sort of things move at a different timeline and pace right on like years so it's not

like oh we'll see it next year and it's gonna win i think it's like five years before we have

another standard pluggable storage back end um extensions like are awesome like postgres that

I'm aware of is the only database that allows you this like deep low level extension hooks that can

change the query planner change the storage engine. It's super exciting. And it's not like there's

like one extension. It's like as Postgres evolves, it's like more hooks get exposed. So you can do

more things that you never thought of possible, right? Like Postgres is great at full text search

at geospatial JSON. I think it's like just missing like being a graph database. Like it's good at

time series um there's the apache age project there's one thing that's i get asked about that

every like nine months so it's been around a while i don't know of anyone using it out in the

wild and someone's using it on the wild like drop me like i'd love to hear about it what's the

experience like i'm pretty easy to find search like craig and postgres and like shoot me a note

a dm and email um i i haven't heard of cases out in the wild so i don't know enough like i get

asked like every six nine months like what about this one i'm like i still don't know and it's been

like five years now because it's not a new project it's new to the apache project but it's it's not a

new extension okay so so we're coming up on time a little bit are there any questions or

recommendations you could give to to us you know what could django do to make crunchy or postgres

his life easier does anything jump off your jump to mind i think the biggest thing is like don't

i find developers and i hate to use the term developers like i am a developer just a really

bad one that's like developers developers uh don't be afraid of the database like i i find that way

like like an app developer that's a really good developer shows up and apologizes or like i'm not

database expert so like don't judge me and it's like i like don't judge me for all of my like we

all have terrible code like all over our apps right and like don't judge me from a year ago

because now i'm a better developer like just you know don't talk to my future self you know because

then i'll have been a bad developer right now um like don't be afraid of the database don't be

afraid of sql right like you don't have to become an expert at it like but like that willingness to

learn i feel like sometimes we abandon it when it comes to databases as developers like we'll we'll

dig into advanced caching and you know templating and um you know like how does dns work does do any

of us know right but we'll go and learn about that before we'll learn about the basics of sql

and i i find there's just like fear and aversion to databases that i wish people would just kind

of like don't be afraid of it be a little willing to dig in and then it you get so much more out of

it yeah good to see and there are these tools like there's the what is it i think one of the

like the harvard places has these like sequel like ways to go deeper a sequel right because i think

part of it is you if you're using django right like okay learn just like very basic sequel and

then you can use the orm but like there isn't that need unless you're like at scale to kind of dive

into sequel as part of it like part of why it's a gap for developers whereas maybe as you grow like

caching okay like i don't know yeah i don't know it's kind of like how do you become a

a database person right like you said you like is there a straight yeah it's just like a weird

sort of you sort of end up there as opposed to like starting there in a career whereas like web

you can kind of stay in web yeah so i mean like so one thing like crunchy we have like a postgres

playground which we put like postgres in your web browser oh yeah that's right i thought yeah

that's that was this year that came out right you can like follow along like it it's running

postgres not on the cloud somewhere connected to it like it's in your web browser and it'll load

we have like exercises and guided tutorials that it's like yeah i'm gonna link to that i'm sorry

yeah i remember that that was really exciting when that came out oh cool i just i just looked

and we shipped like four or five new tutorials there um that i i didn't know about um since

yesterday i think um uh but it's like you you have the question like a query i write at every

single job is like like user signups or like revenue and then it's like growth week over week

or month over month right and i i bet your first thing is gonna like i'm gonna jump to the jingo

orm and write a query that's probably terrible and then i'm gonna iterate over stuff and do

comparisons over a list um why not just write like try to write it in sql and see how it comes out

because it's going to be more performant than anything you do in the orm for that kind of like

reporting stuff um and like just a willingness of like hey if i'm asking questions of data and

it's some kind of report like can i can i try sequel can i give it a shot right that's i don't

know that there's a an answer for like how do you get better at it except for like when you have the

question see if sequel can answer it some of the time doesn't have to be all the time but at least

some of the time yeah there i mean there's for me with beginners i often say if you're writing raw

sequel you're doing it wrong um so i think that's part of it is like you know yes you can do it to

see things and at scale but i think maybe that's also part of it is there's this sense of like

you know there's the simon charrettes of the world who've were smarter than you so like don't even

touch it right i mean that was simon's whole talk at jango con was like it's the orm is not this

scary thing that only you know three sorcerers understand i think it's like if you're asking

questions of data right if i'm not doing a quick like you know the like quick look up like if i'm

asking a question about the data right i'm doing some kind of analysis sql is the right tool for it

right if i'm and so like those cases right when you want to ask how many users did this how many

people did that like what happened to this over this period of time sql is a great thing right

and that's that's when it's useful to dig into yeah and the other thing is like when you dig

into sql like don't worry about it being performant worry about it being readable sql is a powerful

language um if you like you know do do either of you like writing sequel i like the results

okay um if i ask that to 100 people i'll probably get like five to ten

yeses right like it's not a lot of people have any of you have either of you ever liked

reading someone else's sequel no

but like i can write sql that it's like you know with like use ctes use common table expressions

like build like building blocks that it's like no like a dba is going to be like cringe this is this

is so like unoptimized and it's like yeah but developers can read it and you can come back and

see what's going on and tweak it and change it and like right ctes are amazing for readable sequel

and like i'll indent my sequel like i will my code that is and it's like like writing readable

sequel and people are like oh actually this doesn't have to be what i thought about which is

like three joins on one line like what's happening and you just cry right yeah yeah no entirely i

think that was that's why we ended up in this position where it's like no use the ORM don't

use SQL because you'd have basically junior mid-level coders writing hand writing the worst

SQL in the world and then you'd come and you'd need to maintain that and it was unmaintainable

and so it was like no no the correct thing to do is to use the ORM and the abstractions it provides

but there's another there's another use case there's another category of things which isn't

like right in the middle of your application code,

you're slapping some horrible, you know, insert query.

I think I have like some block, I forget.

It's like, I thought it was some like horribly executed joke

about like MySQL, like how I, like my, not MySQL,

but like how I do like queries.

And it's like some basics of indentation and CTEs.

Those two things will make all the difference.

yeah okay super anything else carlton i think we're well i was just gonna say do you run

post chris weekly uh i helped start it i do a very very very little bit for it now it's

peter cooper does all the heavy lifting it's kind of um like early on it was you know hey there

wasn't now that you we've gotten it going right it's kind of like great it's easy to discover

the things people tell us that it's there um there's there's clear kind of outlets and mediums

way back then it was like where is postgres content so um i did a lot more curation in the

early days um now it's a lot less and a lot of peter cooper but like um yeah it was years ago

we were both speaking at a conference and i'm like so postgres weekly are you gonna do it and

he's like because he had the domain and he's like but i i don't know anything about postgres and i

can't curate it myself i'm like well he's like i got all the mechanics in place i'm like this

sounds great so i i did start curating it and like he's got an awesome back-end system that

manages it ships it out like it's so it's great so it's i highly recommend it you know i tweet a

lot about postgres um there's the crunchy data newsletter postgres weekly um a number like if

you're hey i want to dig in a little bit and postgres weekly has always been targeted more

at app devs like people think oh a database newsletter that's we're not targeting dbas if

you want to go like deep on like postgres stuff join the postgres mailing list it's a great

resource um the hackers that postgres hackers mailing list if you want to learn about databases

it's a wonderful place to lurk but if you're just hey i'm a django dev and i just kind of

want to know about the cool new things that i can take advantage of postgres weekly is an

awesome resource yeah no i would i subscribe to that i have done for a number of years and it's

like my little window into the postgres world and i i truly appreciate it so i want to say thank you

to you and to peter for doing that because it's important all right i think we're we're at time so

um we'll say thank you again craig this was this is great this is really interesting to hear all

these things and i you know i think my biggest takeaway is your your point about the next

generation of host like nothing's gonna replace heroku it's gonna be like it's enough to do a

database well it's enough to do a pass well like that makes all the sense in the world even just

talking to you about some of the challenges and things that you see so i'm gonna have to think

about that as i change my recommendations to people in my various books um yeah i think it'll

make the world a little messier but i think it's like it is it's probably the right thing for a

little while cool thanks for having me it was fun no well thank you for coming a real honor to have

you on and like you know to have a chat with you it's super thank you so we have links to everything

crunchy craig's personal site um and thank you craig thank you carlton we'll see everyone next

time bye-bye take care bye-bye