Transcript: Django @Instagram - Carl Meyer (Ep78 Replay)

Hello and welcome to Django chat, a fortnightly podcast on the Django web framework. I'm Will

Vincent joined by Carlton Gibson. Hi, Carlton. Hello, Will. And this week, we're very pleased

to have Carl Meyer join us. Welcome. Thank you. Good to be here. Thank you for coming on. Again,

thank you for joining us. Maybe could you just quickly mention your backstory? You're a big

name in the Django world. But for those who don't know you, how is it that you've come to be on our

podcast? Well, that's a good question. I don't know. Besides, we asked you. Yeah, you emailed

me and asked me to come on. No, I guess I started a company in 2008 with my siblings doing web

application development, a consulting shop, and we settled on Django for backend stack.

And so I started building sites in Django and often fixing bugs in Django was more interesting than building whatever the client needed.

So I ended up spending time doing that and got into Django development that way, I guess, originally.

So what version of that? I mean, was that pre-1 when you first started getting involved?

It was pre-1. Let me think. It was post-Magic removal.

This is going to really be one for the old timers here.

um it was a pre-query set refactor when i first started using django it was in the 0.97 days i

guess which lasted for quite some time there's still some people on it no probably

never managed to what day and then you you're a major figure right because like so i you know

acting as a fellow as i do now like daily i'm trawling through the history of track and very

often i'll come across you know comment by comment anyway you know how did you get involved in the

made in this um sort of maintenance and running just just contributing or just just fixing the

bugs as you said or yeah just uh contributing and i guess having opinions um i attended

that helps right yeah and it happened in so in 2008 when i started the company and started

doing more with django i was living in northern indiana so just an hour or two away from chicago

and uh as luck would have it pycon was in chicago that year 2008 and 2009 so i uh said why not i

headed up to chicago and attended pycon for the first time in 2008 and so i've been to every one

since that actually happened unlike this year and yeah so i guess i i attended a number of

Django sprints at Pycons for a number of years in a row and, uh, got to know the Django team and

I just became more involved. Okay. When I think it was 2014, when you joined the technical board,

was that the inaugural technical board? Yeah, that was the first technical board. Um, yeah,

I think I joined, officially joined the Django team in 2010 maybe, but then we had, uh, kind of

the first big project governance restructuring that Amrik Augustine spearheaded in 2013-2014

and that resulted in the creation of the technical word structure and so you you must have been

involved in the discussions around the creation of the say for instance the API stability policy

and these kind of decisions where we don't with the deprecation policy where you know if we are

going to change something we bring it in we deprecate it two releases later we'll we'll take

it out and that that's that from my perspective that's working phenomenally well um especially

compared to the early days when you know i was using 1.2 and upgrading and ah it was always

it was always really difficult right um so perhaps you could give us some light in that discussion

or because it for me it's just like oh wow it's but it's it's kind of why django is what it is

at the moment yeah my memory is that that was kind of always the intent from the beginning that

jacob and adrian kind of set that uh set us down that path i think at various times we've been more

or less successful in in uh maintaining the backwards compatibility that we intend to

maintain but i think that was one of the reasons that uh 1.0 took so long to arrive because there

was i guess this idea that 1.0 was the moment when the the backwards compatibility guarantees

would really kick in and uh and so there was a hesitance to actually declare 1.0 we want to make

sure we had all of the everything in place the way we wanted it and after a while it became clear

that that was kind of a fool's errand that the apis will never be perfect and we were already

trying to maintain backwards compatibility even pre 1.0 because in practice it was a problem for

people if we broke it so uh at some point uh we just said yeah it's time to do 1.0 now yeah but

i don't recall a lot of changes or discussions around changing that policy i could be forgetting

something but memory is we pretty much always had that intent of giving a couple releases before

changing something because okay that's interesting because i would what would i say i'd say it's a

different world now from say 1.5 1.6 1.7 where things were still changing quite a lot and you

know for good reasons and jango grew with that but you know comparing that to say 1.9 1.10 1.11

into 2.0 it's it's it's very easy to update now um and it's part of that do you think it's just

the maturity of django or do you think yeah i would put it down to the maturity of django mostly

i mean i think backwards compatibility for a widely used library in python is always tricky

because nothing is really private and as much as you might try to maintain backwards compatibility

it's very hard to know what people are relying on and i think in the 1.0 through 1.7 1.8 era

Django was changing very quickly, and there were major new components being added. And I think that

just made it hard to always maintain the backwards compatibility that we intended to. And I would

guess that if it's if it's significantly better now, it has a lot to do with just the core parts

not needing to change as much. Yeah, they don't change. Yeah, they don't change. Okay, cool.

Well, the circle back to PyCons, I was watching for the second time, PyCon 2012 talk, I think you

gave on Django testing, because there aren't actually that many testing Django talks. And

one of the things you said in there, which I thought, which just stood out to me was

write less view code, and maybe don't even test it with unit tests, just do it with functional

integration tests. I wonder, does that still hold up for you? Do you still stand by that? Because I

love the line, you said something like, anytime I see a view of more than 10 or 12 lines,

it's sort of a smell test because I feel that way. Um, so I'm just curious, you know,

eight years later, does that sort of general advice still hold for you or how have your

views changed? How have my views changed? Uh, pun intended. On testing and Django.

No, no, I still write short views. So my views haven't changed. Um, I think, uh, yeah, that's

still basically what I believe. I mean, these days I don't, it's been a number of years since

i've really uh written a django view to be honest uh so uh in practice it's not that particular

question isn't an issue for me but i still certainly believe in uh pulling pulling business

business logic out into testable components and isolating it from io uh in order to make things

testable and that's essentially what that advice boils down to right yeah no it's it's fantastic

advice and i'm uh for me i'm thinking of doing an in-depth django testing course at some point

so i'm trying to review to the extent their best practices because that's one of them use pi test

but it seems kind of all over the map to me honestly aside from i don't i don't see any

pure canonical way to test in django but perhaps i'll find it yeah it's interesting uh coming to

instagram and seeing sort of what happens in a very large code base many years later uh it it

certainly highlighted some of the some of the things in early jango's testing frameworks that

maybe weren't ideal uh like the the emphasis on the test client and on writing lots and lots of

kind of end-to-end functional tests and uh you know it's easy to say that that doesn't scale but

seeing in practice how it doesn't scale is another thing entirely you made you mentioned the big

i words so you have to ask so you can't you you came to instagram it was built on django yes and

then you know for me this is you know super interesting topic because i've never worked on

anything like that kind of scale but i've worked on some big sites and people say can you scale

django i'm like yeah of course you can scale django but then there's a level above that

so i guess you know what what can can you sort of talk us through that the evolution of um to the

extent you can for instagram and how they hit those scaling problems and that you know the

approaches that are needed to take and things because it's it's something that i think most

of us will never see yeah no it's a great question um i gave a talk in 2016 at django under the hood

in Amsterdam, where I really kind of go through a lot of that history in detail. So it'd be an

interesting thing to look up if you're curious about sort of the scaling challenges as they came.

My memory from the research I did for that talk is that as you'd expect, I think the ORM was one

of the first, well, there were some kind of funny minor things that hit earlier on, like

It turns out sending an email to the site maintainers

every time there's a 404.

didn't scale particularly well that became evident early on uh but but yeah as far as

sort of big components i think the orm was the first one there was a uh there was a need i think

after about a year already to to be able to start the database and at that point it was still pre

multi-database i think uh i'm not sure multi-database support might have happened

around that time but i don't instagram certainly didn't have access to it and uh and there was no

real support for uh for sharding uh databases in the orm so mike krieger the the founder of

instagram basically sort of wrote a sharded orm kind of a very bare bones sharded orm

and moved from the django orm to that uh and then later from that to tau which is a

graph database built at facebook which is what instagram uses for the primary data store now

oh i recall i don't know here's an anecdote i'm not sure where i heard it but i believe it to be

true that in the early days uh there was a postgres issue with the space allocated to users

was so big that they couldn't figure it out and i think frank um at uh revsys there was some hack

at the time but helped them with but just these crazy scaling challenges i think it was just

you know the two founders and pushing the limits of software at the time yeah one of the funny

stories that oh definitely people told when i first arrived at instagram was in the early days

uh once instagram sort of really kind of made it mainstream and i think justin bieber was one of

the first big instagram users and uh because of the way that the sharding scheme was done

essentially like a user and all of their media were sharded together and which makes sense for

sort of access patterns but uh it caused a bit of a problem when you have sort of very popular users

like justin bieber would make a post and then everybody you know goes to like and comment on

that post and all of a sudden you have a ton of database activity all concentrated on one shard

uh and that apparently caused this was before my time at instagram but apparently that caused a lot

of problems and sort of memorizing justin beaver's user id was a requirement for uh instagram

operations for a period of time there because sort of every time he posted made a popular post it was

sort of an operations emergency that everyone had all hands on deck kind of thing but those things

are in a lot better shape now yeah i could i could similar stories about um twitter who where

they got popular users they sort of pre-populate that in a separate thing whereas the the average

user you well not you or me but like just a small small people you'd fetch them on demand because

they're not many fetches so python instagram you're wearing a python instagram shirt uh so

they're still using python but they're not using so much django can can you talk about where the

stack is today in terms of is there any django left in there and how much is python as opposed

to other back-end languages uh i mean instagram server is still a big python code base a big

and it's django at the core i mean it still runs through the the django request handling stack and

the middleware and views and url routing all of that is still django would look familiar to any

django developer just very big lots of views lots of urls lots of code um there's a lot of i mean

obviously the orm we don't use anymore the admin is very much based on the orm so that's not in

use anymore i mean there's a lot of components where django provides kind of pluggable backends

like sessions and authentication and for all of that stuff we've essentially rewritten the entire

system but we're actually still using the uh sort of django api for those things because it's

pluggable so i would call that a success story i mean in that we've been able to very smoothly

migrate away from the components that uh that no longer worked at our scale in many cases without

even having to change the client code uh touching those subsystems because django provides this

ability to swap out the uh swap out the back end so yeah and you know the django core request

support that we still use we're 100 async now so we've had to uh i mean already a few years ago we

kind of forked and modified a bunch of that to support uh async or concurrent handling of multiple

requests using async io uh so there are have been some changes but it's kind of cool that you still

got the that core request bit in there because i gave a talk at django con last year the one

that happened not this one that didn't um but talking just about that that core land there's

only like 160 lines in base handler right there's nothing there so there's no the point i made in

the talk was there's no room for it to be slow and clunky i mean that django sometimes gets

described as going down but when it's not doing anything it can't you know it can't be it's still

doing that same thing that all the red frameworks are doing so that's kind of that's really nice i

was wondering would would instagram at the scale they're at need needed to have moved away from

python to a you know a faster language quote unquote i mean you know there's certainly a

discussion of that uh python is not particularly efficient and how it uses uh cpu and there would

be more efficient languages and that would certainly save on server costs but uh there's

also a lot of code that would have to be rewritten and that also has a cost so there's certainly

always a debate about the pros and cons there uh so far the conclusion has been that we'd rather

work with python and try to improve python and make it more efficient uh rather than trying to

rewrite all the existing code in a different language okay that's cool and you said you said

you said improve python there so can you what might that mean i mean yeah well i mean that's

the team that I work on now is kind of actively looking at the Python language and runtime and

where we can tweak things to get a little more efficiency out of it or more developer efficiency.

So one of the projects we've talked about publicly is something we call strict modules. So like one

problem that we have with Python is the fact that when you import a module that can execute

arbitrary code which can do anything it can go out and talk to the network it can end up being

very slow it can fail and when the code base gets very large and there's really so so you can have

all these side effects of import and sometimes one side effect of importing one module can be a

prerequisite for some side effect of importing another module so they have to be imported in

a certain order. And that situation can arise even accidentally over time. But yet the order

that modules are imported isn't really fixed or determined anywhere. It's kind of an emergent

property of the entire system and all the imports and all the modules. So you end up with this very

kind of fragile state where somebody can remove or add an import in some module in one part of

the system. And that has a cascading effect of changing the order of imports. And then something

can fail just because two modules are being imported in a different order. So another side

effect of that or another consequence of that is that if you make a change to a Python module

and you have all of these modules loaded and running in your web server, say you're a developer

working on the code base, you make a change, there's really no safe way to say, let's just

reload the one module and sort of hot patch this one change that's been made. There's nothing you

can do that's safe other than start from scratch start the entire process over again and begin the

entire process of importing and loading all those modules and you know when you code basically is

very large that takes 30 seconds 40 seconds 50 seconds a minute which is a long time for a

developer to wait for making a simple change to one view before they can test that view in their

browser and see the effect of their change so something that we're trying to do with strict

modules is do some static analysis to restrict what is allowed at the top level of a module

to say you know at the top level of a module you can define classes you can define functions

you can use decorators as long as they don't reach out and have external side effects

basically like you can do all the things that we normally associate with python

at the top level of a module but they have to be contained the only effect they can have is within

that module and it once you can guarantee that that's the case that unlocks a whole lot of

possibilities because now you can just safely uh re-import modules it doesn't matter what order

modules are imported in all these problems go away so that's one one kind of example of a project

we're working on more in the developer efficiency side of things okay that sounds really cool when

you talk about the static analysis i mean how does that tie into the kind of thing you see with the

um type hinting that is coming in and um like the pylance um thing that microsoft have got

doing the static analysis there and my pie related tools is it the yeah it's clearly it's the same

ballpark right so but you're using you're leveraging those yeah for sure yeah yeah it's

i mean so strict modules in particular doesn't make too much use of of uh of type hints but it's

It's definitely all in the same kind of area,

which I think is a really productive

and growing area right now,

figuring out how can we do more advanced analysis

make stronger assertions about what Python code does and doesn't do without actually having to

run the code to find out. And so type hinting is one example of that, that we've been very

excited about. We started adding type hints to our code base, I think already in 2017, maybe 20,

yeah, I think 2017 already. So at this point, our server code base is basically 100% typed,

everything has type annotations and we found that to be a big productivity boost for developers

and makes the code safer and so we're looking at you know how to further leverage type hints and

and static analysis to to make python both faster and and more efficient for developers okay and can

i ask a follow-up then because we had a discussion about whether we would bring type hints type

anything into django itself and we were the conclusion that the technical world came to

well not right now because it's difficult and you know the technology is still nascent and

so if you could give us some advice like what would you say to you know if how is it feasible

are we going to be able to get django there what should we be looking at can we start small and

grow in what just as your sort of you know yeah no i'm a big fan of of typed python i think it's

improvement to the language, but I think that the decision the technical board made on that was a

reasonable decision, probably the right decision. It is very early days still. I mean, at a place

like Instagram, we can kind of go all in on something that's still a little rough around

the edges, and our developers are kind of willing to put up with some rough edges in order to get

the benefits of typing. But for a framework like Django that is so widely used, I think it does

make sense to wait until the technology has matured a little bit and some of the rough edges

of usability are smoothed out. So I think that was a good choice. I think it's always a challenge

to bring static typing to code that wasn't developed with static types in mind, because

patterns are used that just don't lend themselves to static typing. So I think

it's been fine for us at Instagram because we don't use the ORM. There's parts of Django that

are particularly challenging for static typing that we don't use anyway. So for us, it's been

pretty easy to just have some stub files that cover the APIs that we use. And that pretty much

covers our needs as far as typing Django itself, it will be challenging to do a good job of typing

all of Django, especially given the backwards compatibility constraints. The other thing is

when we're typing Instagram code that was written in a very sort of dynamic way that doesn't lend

itself to static types, you know, if it's internal code, we can just rewrite it or, you know, change

the pattern. And for Django, that's much more challenging because of the backwards compatibility

constraints so uh i think it'd be great if someday yeah you can say for this code you have to pass an

integer not int or any right exactly and we can just you know go through the entire code base and

change all the call sites and make sure they all do the right thing and just ship that as a diff

and you know you can't ship diffs to all of the code that uses django in the world so question i

have is so you talked about what you've been working on recently but it's been what i think

five five years or so since you've been at instagram there's always this i always have

this mystery of when someone like you so django core developer goes to a large company it's a

little bit of a black hole in terms of like what do they do there like how does it work do you come

in and they say what do you want to work on or do they say you have to do this like how how does

that evolve right when you i mean you're focusing on you know eking out performance in python and

and advantages for developers but yeah i'm just curious what your story was like and if there's

any sort of standard paths when you know a well-known figure goes into a large company and

then you know isn't doing open source work all the time yeah i'm sure it varies by company uh so i

can only really speak to my experience in general and this would apply to any engineer not not just

someone who's coming from an open source community facebook and instagram have a very kind of

engineer driven culture. So, you know, there's a pretty wide latitude for an engineer coming in to

pick the teams and the projects that are interesting to them. And, uh, then the culture

is sort of, you, you decide what you want to do, and then you have to show that the things you did

had some benefit, uh, rather than somebody telling you what to do. Uh, so there's a lot

of opportunity in that it can also be a little bit uh scary uh but uh but yeah so that was my

experience coming in was that there was a lot of freedom to to say well this is this is what i'm

seeing here this is what i think is important for us to work on and so i'm gonna go work on that

one of the public facing things i saw i forget how many years ago is that there was a small team

that switched from python 2 to python 3 over the course of i believe was only like three or so

months at instagram which was perhaps an example of people you know wanting to make a change taking

on their responsibility um but i'll link to that post i mean that was it was pretty amazing that

even at the size of instagram's code base back then a small team was able to do it relatively

quickly yeah no it was very impressive that was one of the first projects i worked on it was already

uh kind of on it was kind of it had just been started when i joined so it was kind of something

that was already happening that was easy to jump in on and i thought it was important for us to do

So, uh, I worked quite a bit on that, on that project.

Um, and yeah, I mean, it, it felt like it took us a while, but I think it was under

a year, which is impressive given the amount of code that we were talking about.

For Django, maybe this is a prosaic question, but so what version of Django to the extent

that there is Django at Instagram, are you running right now?

Yeah, unfortunately we're still on, um, 1.8.

So when I joined in 2016, we were on Django 1.3, which was already quite old at that point.

and we upgraded to 1.8.

The biggest motivation for that was to allow the Python 3 upgrade

because Django 1.3 didn't support Python 3.

So we did the 1.8 upgrade.

We did Python 3, and the problem at this point

is just given the amount of Django code that we use,

there's some big deprecations that happen in 1.9, 1.10,

things like string views went away.

string view names and url configurations went away and there's a bunch of new restrictions around

like uh the import of models files anyway some things that would be a fair amount of work for

us to adapt to and there just really isn't a lot of corresponding um benefit uh because the i mean

you know i don't think django's development should be oriented around the needs of a company like

Instagram. That's not the 90 or even 99% use case. And so the things that are being done in Django

by and large are things that we either already have our own solutions for or just aren't going

to be relevant to what we're doing. So I think at this point, given the forks we've already had to

make for things like async, the most likely path forward is that we end up pulling in the remaining

django code that we use and just kind of continuing to iterate on it as kind of our own internal thing

rather than probably ever doing another django upgrade yeah that's kind of

clearly going to be the case i guess as you say it's not even the 99 that you know five nines out

it's like the one in a one in a million um but it's also kind of sad it's like oh you know

yeah yeah yeah no it's definitely been one of the one of the sad things about

uh the way that my work has evolved over the last four or five years has been that i'm very

disconnected from django development these days i don't even really have a clear sense of what's

in the recent django versions because i'm not getting to use them obviously so that is a little

sad but you know i'm still having fun working on python so it's all good that's fair enough you'll

have to start a side project i wanted to um ask you then so if you're not getting to contribute

to django and work on django anymore what about other things open source things are you are you

able to do open source work or i've made uh i've made and helped with some um contributions to

python itself some small things in the standard library and um a few other patches that we needed

One thing was we had kind of a silly issue with the way that our development process works via the local development server.

We actually run uWSGI, which we run in production and also for local development.

But it runs as root just because of kind of legacy of how the operational systems developed.

And so if you enabled PYC file creation, it would sprinkle all these PYC files owned by root through the code tree.

that obviously doesn't really work. So at some point years ago, before I even got there,

someone just turned off a PYC file creation for local development. And at some point after I got

there, I was looking at this problem of how long it takes to start up the server in local

development. When someone makes a change, how long it takes to restart it. And I was like,

well, we have no PYC files. So not only are we re-importing all these modules,

every time someone makes a change, we're actually like recompiling them from scratch,

because they don't even have PYC files, and that's costing us 25% time, which is really

silly.

then there's there wasn't really a great solution to that so i just wrote a feature for python where

you can set an environment variable and say put all my pyc files over in this other directory

somewhere else and just kind of have a parallel directory tree elsewhere of all your pyc files

which works great for us now we can do that and we get the benefits of pyc file caching

so that's just one small example of like uh and then yeah i contributed that upstream i think it

came out in python 3.9 maybe 3.8 i think 3.8 came out in so so yeah a few things here and there with

python but not as much as i would like okay and do they do they give you time i mean like yeah

i don't think they'd still do it anymore but google used to have this 20 time or whatever

that i don't as i said i don't think it's still there but do you get that kind of thing or is it

only if it's project related you don't have to yeah there isn't a kind of a explicit 20 time

or anything like that um it's kind of like i mentioned earlier like anything you work on here

it's like you can do what you want you just have to at some point be able to say hey hey i did

something useful this app so it's kind of up to you how to how to allocate your time okay that's

cool i'm curious about feature flipping uh at scale you know in django there's like i like

there's this project called django waffle that's really great but instagram is a whole different

beast um are there things what things crop up you know flipping features like i think i've seen

instagrams and playing around with removing the like button in canada or a whole bunch of countries

like what does it even take to roll out something like that yeah if you ask me about specific

features and where they're being rolled out or what's being experimented with i have no idea

that's like very far away from the things that i deal with on a daily basis but certainly feature

flagging in general is, is critical. And, you know, not even, not even sort of for sort of

big experimental features, but for almost anything that gets rolled out, gets rolled out behind

a flag, so it can be turned off. And one, I mean, in some ways, that's not, that's not just about

sort of experimentation. It's also just a necessity of working at this scale, because

the changes to the code base are coming so quickly

that there's really no possibility

of maintaining a long-running branch for a new feature.

So you can't have this development style

where you sort of branch and work on your feature

for a long time until it's just perfect.

And then like a month and a half later,

you merge a big branch back in.

There's just no way that would work.

It'd be a huge-

No home run merges, yeah.

No, exactly.

So it really forces you into this

sort of continuous integration development style

where everything has to land in the same trunk branch

that's constantly being rolled out to production

in fairly small increments.

And so that means if you're developing

any significantly sized feature or change,

it forces you to figure out

how can I do this incrementally and backwards compatibly

and have things in a fully working state

at every point along the way

and gate the feature off

until it's really ready to be ruled out and all that stuff so so yeah it's a very big big part of

what we do so you're not using the orm but you're like the issue there is that always happens is

migrations right it's migrating the database store and how i when you take the field out when you put

it in or yeah yeah i mean it's the same issue just even worse because like uh you know there's

There's no possibility, you know, with a large number of servers and obviously many, many database servers, there's no such thing as like, okay, I'm going to synchronize this code rollout with this database migration and everything will roll out together.

Like every change you make to the data store, I mean, first of all, it's a big deal to roll out any change to the data schema, but any change you make has to be backwards and forwards compatible.

You always have to have things work if it's new code running against the old schema, if it's old code running against the new schema, all that has to work.

So that certainly has a cost in terms of attacks on development speed, but it's just it's the only option.

Can I ask how long does it take to run the test suite?

You know, because I know of like small sites that take an hour, you know, because they have an optimizer done this and the other thing.

I mean, how is it not take months for a little committed Instagram?

So the good news is that we have a lot of tests, pretty good test coverage.

The bad news is that I mentioned earlier about sort of how early Django really pushed you into this path of writing a lot of kind of end-to-end HTTP tests using the test client.

There's a lot of that legacy still in the code base.

So the bad news is that the good news is you have lots of tests.

The bad news is a lot of them are slow tests.

When you have a lot of slow tests, you have a very slow test suite.

So really, I mean, our only solution to that is that we throw hardware at it.

So when we, I literally think that nobody in the world has any idea how long it would take to run the Instagram server test suite and sequentially one test at a time.

Because I don't think, I think it's been six years since anyone has done that.

Like when we actually run the test suite, it runs sharded out to a bunch of different machines.

So like, there's like a bunch of machines all running tests in parallel and then sending the

results back. And with that kind of a system, you know, we can effectively run the test suite for

15, 15 or 20 minutes, which is good enough. Um, but yeah, enough to get a drink, but not,

yeah, enough to get a drink, but not so much that you're like on Twitter for an hour in between

every, yeah, exactly. And obviously people don't generally run the whole test suite every time

they're working locally. Like they'll figure out which specific tests are relevant and run those

locally and then we'll run the whole test suite before we ship it so that's something i'm going

to keep in mind when people tell me about test suites that take more than 20 minutes to be like

well i heard instagram is under 20 so you know think about that i like your point though on

hardware though like because that reminds me at um the only thing at scale i worked on is quizlet

which was a top 100 site at the time and we were on joint and we were had the whole executive team

come in and this that and the other thing after a full day they're like yeah you just need more

hardware you know it's just like the people who wrote it were just like hardware yep sometimes

throwing uh computing power at the problem is the most efficient option yeah oh there's something

in a way beautiful like you know from a business perspective if you can just throw money and solve

an engineering problem that's sort of great actually it's like you'd take that we didn't

talk about oddbird i don't know what that is or other things i mean yeah oddbird is the company

i mentioned that i founded with my siblings in 2008 ah we just said yeah odbert is still going

strong i'm not i'm not there anymore on a daily basis but uh yeah they make you work on thanksgiving

i check in with my siblings now and then see how things are going if they need help but

yeah still going still going well still using a lot of django i think they're up to six or seven

people now so it's a if you need a django application django site uh or application built

and you're looking to contract it out i mean odd bird does everything from the back end to the ui

design and kind of figuring out what the user needs are so it's a great option well maybe one

final question so we're on video uh you're not in california you're remote i think you've been the

whole time you've been remote at instagram is that right that's right how has that been because my

sense is most of the team is there and obviously now everyone's remote but i'm just broadly curious

how remote learning the last odd number of years and then today where everyone's out of the office?

Yeah, I've been remote pretty much my whole career. So it's kind of like a fish in water.

I never really had a big issue with it, even when I was one of the few people remote. I did travel

out to the office in California pretty frequently, which I think is important. There's really no

substitute for getting to know people in person at some point and so just to sort of have those

relationships with the team and i think it was important to make those trips out but on a daily

basis you know we have good video conferencing uh pretty much everybody's already using you know

the funny thing is like the the instagram or facebook offices are so big that half the time

people are meeting over vc anyway just because they're in different buildings and they don't

want to walk across to the other building. So really, everybody's already using kind of these

remote working tools on a daily basis anyway. So I didn't feel that out of place using all the same

tools. I felt like I was able to keep up. And now, of course, yeah, it's the world that everyone

lives in. Right. Well, that's one step further than I remember in San Francisco. I was actually,

I think, on the same block as the Instagram founders. I was on Second Street around the

same time uh 2010 but it was it you know it's even if i'm sitting right next to someone i would

send them a you know a chat message because it's rude to poke someone with headphones on the middle

of something right it's like you know unless it's lunch like i don't want to get poked on a question

just like you know toss it in my chat thing and i'll so now here we are you know extending that

out but you know yeah it makes sense right it's it takes however many 10-15 minutes to get into

a flow state anyways and if you're constantly being interrupted even if you're right there

yeah it's rude to poke a co-worker yep right so carlton can you take us out well all i can say

carl's thanks for coming on it's really interesting chat and love um love to get the back story and

then you know the story of instagram and how they grow up and you know that's super thanks for

coming thank you it's been a pleasure i've really really enjoyed the conversation well as ever we

are at jango chat.com chat jango on twitter were you gonna say something carlton or did i talk over

you no i was just gonna say join us next time all right bye-bye everyone bye-bye thanks guys