Transcript: Django @Instagram - Carl Meyer

Hello, and welcome to Django Chat, a fortnightly podcast on the Django web framework. I'm Will

Vincent, joined by Carlton Gibson. Hi, Carlton.

Hello, Will.

And this week, we're very pleased to have Carl Meyer join us. Welcome.

Thank you. Good to be here.

Thank you for coming on.

Again, thank you for joining us. Maybe could you just quickly mention your backstory?

You're a big name in the Django world. But for those who don't know you,

how is it that you've come to be on our podcast?

well that's a good question um i don't know i besides we asked you yeah you emailed me and

asked me to come on no i i guess i uh i started a company in 2008 with my siblings doing a web

application development a consulting shop and we settled on django for back-end stack and so i

started building sites in Django and often fixing bugs in Django was more interesting than

building whatever the client needed so I ended up spending time doing that and

got into Django development that way I guess originally so what what version of that I mean

that was was that pre-1 when you first started getting involved it was pre-1 let me think it

was post magic removal this is gonna really be one for the old timers here um it was a pre-query

set refactor when i first started using django it was in the 0.97 days i guess which lasted for

quite some time well there's still some people on it no probably never managed to what day

and then you you're a major figure right because like so i you know acting as a fellow as i do now

like daily i'm trawling through the history of track and very often i'll come across you know

comment by comment you know how did you get involved in the mate in this um sort of maintenance

and running just just contributing or just just fixing the bugs as you said or yeah just uh

contributing and i guess having opinions um i attended that helps yeah and it happened in so

in 2008 when i started the company and started doing more with django i was living in northern

Indiana, so just an hour or two away from Chicago. And as luck would have it, PyCon was in Chicago

that year, 2008 and 2009. So I said, why not? I headed up to Chicago and attended PyCon for the

first time in 2008. And so I've been to every one since that actually happened, unlike this year.

And yeah, so I guess I attended a number of Django sprints at PyCons for a number of years in a row

and got to know the Django team, and I just became more involved.

When I think it was 2014 when you joined the technical board,

was that the inaugural technical board?

Yeah, that was the first technical board.

Yeah, I think I officially joined the Django team in 2010 maybe,

but then we had kind of the first big project governance restructuring

that Amrik Augustine spearheaded in 2013, 2014,

and that resulted in the creation of the technical word structure.

And so you must have been involved in the discussions

around the creation of the, say, for instance,

the API stability policy and these kind of decisions

where we don't, the deprecation policy,

where, you know, if we are going to change something,

we bring it in, we deprecate it, two releases later,

we'll take it out.

And that's, from my perspective, that's working phenomenally well,

especially compared to the early days when you know i was using 1.2 and upgrading and

ah it was always it was always really difficult right um so perhaps you could give us some light

in that discussion or because it for me it's just like oh wow it's but it's it's kind of why jango

is what it is at the moment yeah my memory is that that was kind of always the intent from the

beginning that jacob and adrian kind of set that uh set us down that path i think at various times

we've been more or less successful in maintaining the backwards compatibility that we intend to

maintain. But I think that was one of the reasons that 1.0 took so long to arrive because there was

I guess this idea that 1.0 was the moment when the backwards compatibility guarantees would

really kick in. And so there was a hesitance to actually declare 1.0. We wanted to make sure we

had all of the everything in place the way we wanted it. And after a while, it became clear

that that was kind of a fool's errand,

that the APIs will never be perfect.

And we were already trying to maintain backwards compatibility

even pre 1.0, because in practice,

it was a problem for people if we broke it.

So at some point, we just said,

yeah, it's time to do 1.0 now.

Yeah, but I don't recall a lot of changes

or discussions around changing that policy.

I could be forgetting something, but memory is,

we pretty much always had that intent

of giving a couple releases before changing something because okay that's interesting because

i would what would i say i'd say it's a different world now from say 1.5 1.6 1.7 where things were

still changing quite a lot and you know for good reasons and django grew with that but you know

comparing that to say 1.9 1.10 1.11 into 2.0 it's it's it's very easy to update now um and it's part

of that do you think it's just the maturity of django or do you think yeah i would put it down

to the maturity of Django mostly.

I mean, I think backwards compatibility

for a widely used library in Python is always tricky

because nothing is really private.

And as much as you might try

to maintain backwards compatibility,

it's very hard to know what people are relying on.

And I think in the 1.0 through 1.7, 1.8 era,

Django was changing very quickly

and there were major new components being added.

And I think that just made it hard

to always maintain the backwards compatibility

that we intended to.

And I would guess that if it's significantly better now,

it has a lot to do with just the core parts

not needing to change as much.

Yeah, they don't change.

Yeah, they don't change.

Okay, cool.

Well, to circle back to PyCons,

I was watching for the second time a PyCon 2012 talk

I think you gave on Django testing

because there aren't actually that many testing Django talks.

And one of the things you said in there,

which I thought, which just stood out to me was write less view code and maybe don't even test it

with unit tests, just do it with a functional or integration tests. I wonder, does that still hold

up for you? Do you still stand by that? Because I love the line you said something like, anytime I

see a view of more than 10 or 12 lines, it's sort of a smell test because I feel that way.

So I'm just curious, you know, eight years later, does that sort of general advice still

hold for you or how have your views changed how my views changed pun intended on testing in django

no no i still write short views so my views haven't changed um i think uh yeah that's still

basically what i believe i mean these days i don't it's been a number of years since i've really

written a django view to be honest uh so uh in practice it's not that particular question isn't

an issue for me but i still certainly believe in uh pulling pulling business business logic out

into testable components and isolating it from io uh in order to make things testable and that's

essentially what that advice boils down to right yeah no it's it's fantastic advice and i'm uh for

me i'm thinking of doing an in-depth django testing course at some point so i'm trying to review

to the extent there are best practices because that's one of them use pi test but it seems kind

of all over the map to me honestly aside from i don't i don't see any pure canonical way to

test in django but perhaps i'll find it yeah it's interesting uh coming to instagram and seeing sort

of what happens in a very large code base many years later uh it it certainly highlighted some

of the some of the things in early django's testing frameworks that maybe weren't ideal

like the the emphasis on the test client and on writing lots and lots of kind of end-to-end

functional tests and uh you know it's easy to say that that doesn't scale but seeing in practice

how it doesn't scale is another thing entirely you made you mentioned the big i word so i have

to ask so you can't you came to instagram it was built on django yes and then you know for me this

is you know super interesting topic because i've never worked on anything like that kind of scale

but i've worked on some big sites and people say can you scale django i'm like yeah of course you

can scale django but then there's a level above that so i guess you know what what can can you

sort of talk us through the the evolution of um to the extent you can for instagram and how they

hit those scaling problems and that you know the approaches that are needed to take and things

because it's something that i think most of us will never see yeah no it's a great question um

i gave a talk in 2016 at django under the hood in amsterdam where i really kind of go through a lot

of that history in in detail so it'd be an interesting thing to look up if you're curious

about sort of the scaling challenges as they came my memory from the research i did for that talk

is that, as you'd expect, I think the ORM was one of the first...

Well, there were some kind of funny minor things

that hit earlier on, like turns out sending an email

to the site maintainers every time there's a 404.

didn't scale particularly well that became evident early on uh but but yeah as far as

sort of big components i think the orm was the first one there was a uh there was a need i think

after about a year already to to be able to start the database and at that point it was still pre

multi-database i think uh i'm not sure multi-database support might have happened

around that time but i don't instagram certainly didn't have access to it and uh and there was no

real support for uh for sharding uh databases in the orm so mike krieger the the founder of

instagram basically sort of wrote a sharded orm kind of a very bare bones sharded orm

and moved from the django orm to that uh and then later from that to tau which is a

graph database built at facebook which is what instagram uses for the primary data store now

oh i recall i don't know here's an anecdote i'm not sure where i heard it but i believe it to be

true that in the early days uh there was a postgres issue with the space allocated to users

was so big that they couldn't figure it out and i think frank um at uh revsys there was some hack

at the time but helped them with but just these crazy scaling challenges i think it was just

you know the two founders and pushing the limits of software at the time yeah one of the funny

stories that oh definitely people told when i first arrived at instagram was in the early days

uh once instagram sort of really kind of made it mainstream and i think justin bieber was one of

the first big instagram users and uh because of the way that the sharding scheme was done

essentially like a user and all of their media were sharded together and which makes sense for

sort of access patterns but uh it caused a bit of a problem when you have sort of very popular users

like justin bieber would make a post and then everybody you know goes to like and comment on

that post and all of a sudden you have a ton of database activity all concentrated on one shard

uh and that apparently caused this was before my time at instagram but apparently that caused a lot

of problems and sort of memorizing justin beaver's user id was a requirement for uh instagram

operations for a period of time there because sort of every time he posted made a popular post it was

sort of an operations emergency that everyone had all hands on deck kind of thing but those things

are in a lot better shape now yeah i could i could similar stories about um twitter who where

they got popular users they sort of pre-populate that in a separate thing whereas the the average

user you well not you or me but like just a small small people you'd fetch them on demand because

they're not many fetches so python instagram you're wearing a python instagram shirt uh so

they're still using python but they're not using so much django can can you talk about where the

stack is today in terms of is there any django left in there and how much is python as opposed

to other back-end languages uh i mean instagram server is still a big python code base a big

and it's django at the core i mean it still runs through the the django request handling stack and

the middleware and views and url routing all of that is still django would look familiar to any

django developer just very big lots of views lots of urls lots of code um there's a lot of i mean

obviously the orm we don't use anymore the admin is very much based on the orm so that's not in

use anymore i mean there's a lot of components where django provides kind of pluggable backends

like sessions and authentication and for all of that stuff we've essentially rewritten the entire

system but we're actually still using the uh sort of django api for those things because it's

pluggable so i would call that a success story i mean in that we've been able to very smoothly

migrate away from the components that uh that no longer worked at our scale in many cases without

even having to change the client code uh touching those subsystems because django provides this

ability to swap out the uh swap out the back end so yeah and you know the django core request

support that we still use we're 100 async now so we've had to uh i mean already a few years ago we

kind of forked and modified a bunch of that to support uh async or concurrent handling of multiple

requests using async io uh so there are have been some changes but it's kind of cool that you still

got the that core request bit in there because i gave a talk at django con last year the one

that happened not this one that didn't um but talking just about that that core land there's

only like 160 lines in base handler right there's nothing there so there's no the point i made in

the talk was there's no room for it to be slow and clunky i mean that django sometimes gets

described as going down but when it's not doing anything it can't you know it can't be it's still

doing that same thing that all the red frameworks are doing so that's kind of that's really nice i

was wondering would would instagram at the scale they're at need needed to have moved away from

python to a you know a faster language quote unquote i mean you know there's certainly a

discussion of that uh python is not particularly efficient and how it uses uh cpu and there would

be more efficient languages and that would certainly save on server costs but uh there's

also a lot of code that would have to be rewritten and that also has a cost so there's certainly

always a debate about the pros and cons there uh so far the conclusion has been that we'd rather

work with python and try to improve python and make it more efficient uh rather than trying to

rewrite all the existing code in a different language okay that's cool and you said you said

you said improve python there so can you what might that mean i mean yeah well i mean that's

the team that I work on now is kind of actively looking at the Python language and runtime and

where we can tweak things to get a little more efficiency out of it or more developer efficiency.

So one of the projects we've talked about publicly is something we call strict modules. So like one

problem that we have with Python is the fact that when you import a module that can execute

arbitrary code which can do anything it can go out and talk to the network it can end up being

very slow it can fail and when the code base gets very large and there's really so so you can have

all these side effects of import and sometimes one side effect of importing one module can be a

prerequisite for some side effect of importing another module so they have to be imported in

a certain order. And that situation can arise even accidentally over time. But yet the order

that modules are imported isn't really fixed or determined anywhere. It's kind of an emergent

property of the entire system and all the imports and all the modules. So you end up with this very

kind of fragile state where somebody can remove or add an import in some module in one part of

the system. And that has a cascading effect of changing the order of imports. And then something

can fail just because two modules are being imported in a different order. So another side

effect of that or another consequence of that is that if you make a change to a Python module

and you have all of these modules loaded and running in your web server, say you're a developer

working on the code base, you make a change, there's really no safe way to say, let's just

reload the one module and sort of hot patch this one change that's been made. There's nothing you

can do that's safe other than start from scratch, start the entire process over again, and begin the

entire process of importing and loading all those modules. And, you know, when your code basically

is very large, that takes 30 seconds, 40 seconds, 50 seconds, a minute, which is a long time for a

developer to wait for making a simple change to one view before they can test that view in their

browser and see the effect of their change. So something that we're trying to do with strict

modules is do some static analysis to restrict what is allowed at the top level of a module

to say you know at the top level of a module you can define classes you can define functions

you can use decorators as long as they don't reach out and have external side effects

basically like you can do all the things that we normally associate with python

at the top level of a module but they have to be contained the only effect they can have is within

that module and it once you can guarantee that that's the case that unlocks a whole lot of

possibilities because now you can just safely uh re-import modules it doesn't matter what order

modules are imported in all these problems go away so that's one one kind of example of a project

we're working on more in the developer efficiency side of things okay that sounds really cool when

you talk about the static analysis i mean how does that tie into the kind of thing you see with the

um type hinting that is coming in and um like the pylance um thing that microsoft have got

doing the static analysis there and my pine related tools is it the yeah it's clearly it's

the same ballpark right so but you're using you're leveraging those yeah for sure yeah yeah it's i

mean so strict modules in particular doesn't make too much use of of uh of type hints but it's

It's definitely all in the same kind of area,

which I think is a really productive

and growing area right now of figuring out

how can we do more advanced analysis

make stronger assertions about what Python code does and doesn't do without actually having to

run the code to find out. And so type hinting is one example of that, that we've been very

excited about. We started adding type hints to our code base, I think already in 2017, maybe 20,

yeah, I think 2017 already. So at this point, our server code base is basically 100% typed,

everything has type annotations and we found that to be a big productivity boost for developers

and makes the code safer and so we're looking at you know how to further leverage type hints and

and static analysis to to make python both faster and and more efficient for developers okay and can

i ask a follow-up then because we had a discussion about whether we would bring type hints type

anything into django itself and we were the conclusion that the technical world came to

well not right now because it's difficult and you know the technology is still nascent and

so if you could give us some advice like what would you say to you know if how is it feasible

are we going to be able to get django there what should we be looking at can we start small and

grow in what just as your sort of you know yeah no i'm a big fan of of typed python i think it's

improvement to the language. But I think that the decision the technical board made on that was a

reasonable decision, probably the right decision. It is very early days still. I mean, at a place

like Instagram, we can kind of go all in on something that's still a little rough around

the edges. And our developers are kind of willing to put up with some rough edges in order to get

the benefits of typing. But for a framework like Django that is so widely used, I think it does

make sense to wait until the technology has matured a little bit and some of the rough edges

of usability are smoothed out. So I think that was a good choice. I think it's always a challenge

to bring static typing to code that wasn't developed with static types in mind, because

patterns are used that just don't lend themselves to static typing. So I think

it's been fine for us at Instagram because we don't use the ORM. There's parts of Django that

are particularly challenging for static typing that we don't use anyway. So for us, it's been

pretty easy to just have some stub files that cover the APIs that we use. And that pretty much

covers our needs as far as typing Django itself, it will be challenging to do a good job of typing

all of Django, especially given the backwards compatibility constraints. The other thing is

when we're typing Instagram code that was written in a very sort of dynamic way that doesn't lend

itself to static types, you know, if it's internal code, we can just rewrite it or, you know, change

the pattern. And for Django, that's much more challenging because of the backwards compatibility

constraints so uh i think it'd be great if someday yeah you can say for this code you have to pass an

integer not int or any right exactly and we can just you know go through the entire code base and

change all the call sites and make sure they all do the right thing and just ship that as a diff

and you know you can't ship diffs to all of the code that uses django in the world so question i

have is so you talked about what you've been working on recently but it's been what i think

five, five years or so since you've been at Instagram, there's always this, I always have

this mystery of when someone like you, so Django core developer goes to a large company, it's a

little bit of a black hole in terms of like, what do they do there? Like, how does it work? Do you

come in and they say, what do you want to work on? Or do they say you have to do this? Like how,

how does that evolve? Right. When you, I mean, you're focusing on, you know, eking out performance

in Python and, and advantages for developers, but yeah, I'm just curious what your story was like.

And if there's any sort of standard paths when, you know, a well-known figure goes into

a large company and then, you know, isn't doing open source work all the time.

Yeah, I'm sure it varies by company.

So I can only really speak to my experience in general.

And this would apply to any engineer, not just someone who's coming from an open source

community.

Facebook and Instagram have a very kind of engineer driven culture.

so you know there's a pretty wide latitude for an engineer coming in to pick the teams and the

projects that are interesting to them and then the culture is sort of you you decide what you

want to do and then you have to show that the things you did had some benefit rather than

somebody telling you what to do so there's a lot of opportunity in that it can also be a little bit

scary. Uh, but, uh, but yeah, so that was my experience coming in was that there was a lot

of freedom to, to say, well, this is, this is what I'm seeing here. This is what I think is

important for us to work on. And so I'm going to go work on that. One of the public facing things

I saw, I forget how many years ago is that there was a small team that switched from Python two to

Python three over the course of, I believe was only like three or so months at Instagram, which

was perhaps an example of people wanting to make a change, taking on their responsibility,

um but i'll link to that post i mean that was it was pretty amazing that even at the size of

instagram's code base back then a small team was able to do it relatively quickly yeah no it was

very impressive that was one of the first projects i worked on it was already uh kind of it was kind

of it had just been started when i joined so it was kind of something that was already happening

that was easy to jump in on and i thought it was important for us to do so uh i worked quite a bit

on that on that project um and yeah i mean it it felt like it took us a while but i think it was

under a year which is impressive given the amount of code that we were talking about

for django maybe this is a prosaic question but so what version of django to the extent that there

is django at instagram are you running right now yeah unfortunately we're still on um 1.8 so when

i joined in 2016 we were on django 1.3 which was already quite old at that point and uh we upgraded

to 1.8, the biggest motivation for that was to allow the Python 3 upgrade, because Django 1.3

didn't support Python 3. So we did the 1.8 upgrade, we did Python 3. And, you know, the problem at

this point is just given the amount of Django code that we use, you know, there's some big

deprecations that happened in 1.9, 1.10, things like string views went away, string view names

and url configurations went away and there's a bunch of new restrictions around like uh the

import of models files anyway some things that would be a fair amount of work for us to adapt

to and there just really isn't a lot of corresponding um benefit uh because the i mean

you know i don't think django's development should be oriented around the needs of a company like

Instagram. That's not the 90 or even 99% use case. And so the things that are being done in Django

by and large are things that we either already have our own solutions for or just aren't going

to be relevant to what we're doing. So I think at this point, given the forks we've already had to

make for things like async, the most likely path forward is that we end up pulling in the remaining

django code that we use and just kind of continuing to iterate on it as kind of our own internal thing

rather than probably ever doing another django upgrade yeah that's kind of

clearly going to be the case i guess as you say it's not even the 99 that you know five nines out

it's like the one in a one in a million um but it's also kind of sad it's like oh you know

yeah yeah yeah no it's definitely been one of the one of the sad things about

uh the way that my work has evolved over the last four or five years has been that i'm very

disconnected from django development these days i don't even really have a clear sense of what's

in the recent django versions because i'm not getting to use them obviously so that is a little

sad but you know i'm still having fun working on python so it's all good that's fair enough you'll

have to start a side project i wanted to um ask you then so if you're not getting to contribute

to django and work on django anymore what about other things open source things are you are you

able to do open source work or i've made uh i've made and helped with some um contributions to

python itself some small things in the standard library and um a few other patches that we needed

One thing was we had kind of a silly issue with the way that our development process works via the local development server.

We actually run uWSGI, which we run in production and also for local development.

But it runs as root just because of kind of legacy of how the operational systems developed.

And so if you enabled PYC file creation, it would sprinkle all these PYC files owned by root through the code tree.

that obviously doesn't really work.

So at some point years ago, before I even got there,

someone just turned off a PYC file creation for local development.

And at some point after I got there,

I was looking at this problem of how long it takes to start up the server

in local development when someone makes a change,

how long it takes to restart it.

And I was like, well, we have no PYC files.

So not only are we re-importing all these modules every time someone

makes a change, we're actually like recompiling them from scratch

because we don't even have PYC files and that's costing us 25% time, which is really silly.

then there's there wasn't really a great solution to that so i just wrote a feature for python where

you can set an environment variable and say put all my pyc files over in this other directory

somewhere else and just kind of have a parallel directory tree elsewhere of all your pyc files

which works great for us now we can do that and we get the benefits of pyc file caching

so that's just one small example of like uh and then yeah i contributed that upstream i think it

came out in python 3.9 maybe 3.8 i think 3.8 came out in so so yeah a few things here and there with

python but not as much as i would like okay and do they do they give you time i mean like yeah

i don't think they'd still do it anymore but google used to have this 20 time or whatever

that i don't as i said i don't think it's still there but do you get that kind of thing or is it

only if it's project related you don't have to yeah there isn't a kind of a explicit 20 time

or anything like that um it's kind of like i mentioned earlier like anything you work on here

it's like you can do what you want you just have to at some point be able to say hey hey i did

something useful this app so it's kind of up to you how to how to allocate your time okay that's

cool i'm curious about feature flipping uh at scale you know in django there's like i like

there's this project called django waffle that's really great but instagram is a whole different

beast um are there things what things crop up you know flipping features like i think i've seen

instagrams and playing around with removing the like button in canada or a whole bunch of countries

like what does it even take to roll out something like that yeah if you ask me about specific

features and where they're being rolled out or what's being experimented with i have no idea

that's like very far away from the things that i deal with on a daily basis but certainly feature

flagging in general is, uh, is critical. Um, and you know, not even, not even for sort of big

experimental features, but for almost anything, uh, that gets rolled out, gets rolled out behind

a flag. So it can be turned off. And one, I mean, in some ways, uh, that's not,

it's not just about sort of experimentation. It's also just a necessity of working at this scale

because the changes to the code base are coming so quickly

that there's really no possibility

of maintaining a long-running branch for a new feature.

So you can't have this development style

where you sort of branch and work on your feature

for a long time until it's just perfect.

And then like a month and a half later,

you merge a big branch back in.

There's just no way that would work.

It'd be a huge-

No home run merges, yeah.

No, exactly.

So it really forces you into this

sort of continuous integration development style

where everything has to land in the same trunk branch

that's constantly being rolled out to production

in fairly small increments.

And so that means if you're developing

any significantly sized feature or change,

it forces you to figure out

how can I do this incrementally and backwards compatibly

and have things in a fully working state

at every point along the way

and gate the feature off

until it's really ready to be rolled out and all that stuff so so yeah it's a very big big part of

what we do so you're not using the orm but you're like the issue there is that always happens is

migrations right it's migrating the database store and how i when you take the field out when you put

it in or yeah yeah i mean it's the same issue just even worse because like uh you know there's

There's no possibility, you know, with a large number of servers and obviously many, many database servers, there's no such thing as like, okay, I'm going to synchronize this code rollout with this database migration and everything will roll out together.

like every change you make to the data store i mean first of all it's a big deal to roll out any

uh change to the data schema but any change you make has to be backwards and forwards compatible

you always have to have things work if it's new code running against the old schema if it's old

code running against the new schema all that has to work so um there's that certainly has a a cost

in terms of attacks on development speed but it's just it's the only option can i ask how long does

it take to run the test suite you know i mean because i i know of like small sites that take

an hour you know because they have an optimizer done this and the other thing i mean how is it

not take months for a little committed instagram so the good news is that we have a lot of tests

pretty good test coverage um the bad news is that i've mentioned earlier about sort of how early

django really pushed you into this path of writing a lot of kind of end-to-end http tests using the

test client and there's a lot of that legacy still in the code base so the bad news is that

the good news is we have lots of tests the bad news is a lot of them are slow tests when you

have a lot of slow tests you have a very slow test suite uh so really i mean our our only solution to

that is that we uh throw hardware at it so when we i i literally think that nobody in the world

has any idea how long it would take to run the instagram server test suite and and sequentially

one test at a time because i don't think i think it's been six years since anyone has done that

like when we actually run the test suite it runs uh sharded out to a bunch of different machines

um so like there's like a bunch of machines all running tests in parallel and sending the results

back and with that kind of a system you know we can effectively run the test suite for 15 15 or

20 minutes which is uh good enough um but yeah enough to go get a drink but not yeah enough to

a drink, but not so much that you're like on Twitter for an hour in between every.

Yeah, exactly. And obviously, people don't generally run the whole test suite every time

they're working locally, like they'll figure out which specific tests are relevant and run

those locally. And then we'll run the whole test suite before we ship it.

So that's something I'm going to keep in mind when people tell me about test suites that take

more than 20 minutes. I'd be like, well, I heard Instagram is under 20. So, you know, think about

that. I like your point, though, on hardware, though, like, because that reminds me that

the only thing at scale I worked on is Quizlet, which was a top 100 site at the time. And we were

on joint and we had the whole executive team come in and this, that, and the other thing after a

full day, they're like, yeah, you just need more hardware. It's just like the people who wrote it

were just like, eh, hardware. Sometimes throwing a computing power at the problem is the most

efficient option. Yeah. Oh, there's something in a way beautiful, like, you know, from a business

perspective, if you can just throw money and solve an engineering problem, that's sort of great,

actually it's like you'd take that we didn't talk about odd bud i don't know what that is or other

things i mean yeah odd bird is the company i mentioned that i founded with my siblings in 2008

ah we just said yeah odd bird is still going strong i'm not i'm not there anymore on a daily

basis but uh yeah they make you work on thanksgiving i checked in with my siblings now

and then see how things are going if they need help but yeah it's still going still going well

still using a lot of Django. I think they're up to six or seven people now. So it's a, if you need

a Django application, Django site or application built, and you're looking to contract it out,

I mean, Oddbird does everything from the backend to the UI design and kind of figuring out what

the user needs are. So it's a great option. Well, maybe one final question. So we're on video.

You're not in California, you're remote. I think you've been the whole time you've been

remote and Instagram. Is that right? How has that been? Cause my sense is most of the team is there

and obviously now everyone's remote, but I'm just broadly curious how, you know, remote learning the

last odd number of years. And then today where everyone's out of the office. Yeah, I've been

remote pretty much my whole career. So, uh, it's kind of like a fish in water. Uh, I never really

had a a big issue with it uh even when i was one of the few people remote i did travel out to the

office in california pretty frequently which i think is important there's really no substitute

for getting to know people in person at some point and so just to sort of have those relationships

with the team and i think it was important to make those trips out but on a daily basis you know we

of good video conferencing uh pretty much everybody's already using you know the funny

thing is like the the instagram or facebook offices are so big that half the time people

are meeting over vc anyway just because they're in different buildings and they don't want to walk

across to the other building so so really everybody's already using kind of these remote

working tools on a daily basis anyway so i didn't feel that out of place using all the same tools i

felt like i was able to keep up and now of course yeah it's the world that everyone lives in so

right well that's one one step further than i remember in uh in san francisco i was actually

i think on the same block as the instagram founders i was on second street um you know

around the same time uh 2010 but it was it you know it's even if i'm sitting right next to someone i

would send them a you know a chat message because it's rude to poke someone with headphones on the

middle of something right it's like you know unless it's lunch like i don't want to get

poked on a question just like you know toss it my chat thing and i'll so now here we are you know

extending that out but you know yeah it makes sense right it's it takes however many 10-15

minutes to get into a flow state anyways and if you're constantly being interrupted even if you're

right there yeah it's rude to poke a co-worker yep great so carlton can you take us out well

all i can say carl is thanks for coming on it's really interesting chat and love um love to get

the back story and then you know the story of instagram and how they grow up and you know

that's super thanks for coming thank you it's been a pleasure i've really really enjoyed the

conversation well as ever we are at django chat.com chat django on twitter were you gonna say

something carlton or did i talk over you no i was just gonna say join us next time all right

bye-bye everyone bye-bye thanks guys