Transcript: Django @Instagram - Carl Meyer

I'm Will Vincent, joined by Carlton Gibson.

Hi, Carlton.

Hello, Will.

And this week we're very pleased to have Carl Meyer join us.

Welcome.

Great to be here.

coming on.

Maybe could you just quickly mention your backstory?

Uh, you're a big name in the Django world, but for those who don't know you, how is it that you've come to be on our podcast?

Um, I don't know.

I I

No, I I guess I uh I started a company in 2008 uh with my siblings doing uh web application development uh consulting shop and uh we settled on Django for our backend stack and so I started building sites in Django and often fixing bugs in Django was more interesting than uh building whatever the client needed.

So uh I ended up spending time uh doing that and uh got into Django development that way, I guess, originally.

I mean, that was was that pre-1 when you first started getting involved?

Uh let me think, it was post-magic removal.

This is gonna really be one for the old timers here.

Um it was uh pre-query set refactor when I first started using Django.

It was in the point.97 days, I guess, which lasted for quite some time.

Probably never managed to update.

And then you you're a major figure, right?

Because like so I, you know, for uh acting as a fellow as I do now, like daily I'm trawling through the history of track, and very often I'll come across, you know, a comment by Carl, you know, it's a how did you get involved in the mate in this um sort of maintenance and running?

Just just contributing or just just fixing the bugs, as you said, or

Um I attended uh that helps uh in so in 2008 when I started the company and started doing more with Django.

I was living in uh northern Indiana, so just an hour or two away from Chicago.

And uh as luck would have it, PyCon was in Chicago that year, 2008 and 2009.

So I uh said, why not?

I uh headed up to Chicago and attended PyCon for the first time in 2008, and so uh I've been to everyone since that uh actually happened, unlike this year.

And yeah, so I guess I I attended a number of Django sprints at PyCons for a number of years in a row and uh got to know the uh Django team and I just became more involved.

Okay.

Was that the inaugural technical board?

Um yeah, I think I I think I joined officially joined the Django team in 2010, maybe, but then we had uh kind of the first big project governance restructuring that uh Emric Augustine spearheaded in 2013, 2014, and that resulted in the the creation of the technical word structure.

And that that's that from my perspective, that's working phenomenally well.

Um especially compared to the early days when you know I was using 1.2 and upgrading, and it was always it was always really difficult, right?

Um, so perhaps you could give us some light in that discussion.

Or because it it for me it's just like, oh wow, it's but it's it's kind of why Django is what it is at the moment.

I think at various times we've been more or less successful in uh maintaining the backwards compatibility that we intend to maintain.

But I think that was one of the reasons that uh 1.0 took so long to arrive, because there was uh, I guess this idea that 1.0 is the moment when the the backwards compatibility guarantees would really kick in.

And uh and so there was a hesitance to actually declare 1.0.

We wanted to make sure we had all of the uh everything in place the way we wanted it.

And after a while it became clear that that was kind of a fool's errand, that the APIs will never be uh perfect, and we were already trying to maintain backwards compatibility even pre-1.0 because in practice it was a problem for people if we broke it.

So uh at some point uh we just said, yeah, it's time to do 1.0 now.

Yeah, but I I don't recall a lot of changes or discussions around changing that policy.

I could be forgetting something, but memory is we we pretty much always uh had that intent of giving a couple releases before changing something.

Because I would what would I say?

I'd say it's a different world now from say 1.5, 1.6, 1.7, where things were still changing quite a lot, and you know, for good reasons.

And Django grew with that.

But you know, comparing that to say 1.9, 1.10, 1.11 into 2.0, it it's it's it's very easy to update now.

Um, and is part of that, do you think is just the maturity of Django, or do you think

I mean, I think backwards compatibility for a widely used library in Python is always tricky because nothing is really private.

And uh, as much as you might try to maintain backwards compatibil compatibility, it's very hard to know what people are relying on.

And I think in the 1.0 through 1.7, 1.8 era, Django was changing very quickly and there were major new components being added, and I think that just made it hard to uh to always maintain the the backwards compatibility that we intended to.

And I would guess that if it's if it's significantly better now, it has a lot to do with just the core parts not needing to change as much.

Yeah, they don't change.

Okay, cool.

Because there aren't actually that many testing Django talks.

Um and one of the things you said in there, which I thought which just stood out to me was write less view code and maybe don't even test it with unit tests, just do with it with uh functional or integration tests.

I wonder, does that still hold up for you?

Do you still stand by that?

Because I I love the line you said something like anytime I see a view of more than 10 or 12 lines, it's sort of a smell test because I feel that way.

Um so I'm just curious, you know, eight years later, does that sort of general advice still hold for you, or how have your views changed?

Uh pun intended.

Um I think uh that yeah, that's still basically what I believe.

I mean, these days I don't it's been a number of years since I've really uh written a Django view, to be honest.

Uh so uh in practice it's not that particular question isn't an issue for me, but I still certainly believe in uh pulling pulling business business logic out into testable components and isolating it from IO uh in order to make things testable.

And that's essentially what that advice boils down to, right?

And I'm uh for me, I'm thinking of doing a in-depth Django testing course at some point, so I'm trying to review to the extent there are best practices because that's one of them use PyTest.

But it seems kind of all over the map to me, honestly, aside from I don't I don't see any pure canonical way to test in Django, but yeah perhaps I'll find it.

Uh it it certainly highlighted some of the some of the things in early Django's testing frameworks that maybe weren't ideal, uh like the the emphasis on the test client and on writing lots and lots of kind of end-to-end functional tests.

And uh, you know, it's easy to say that that doesn't scale, but seeing in practice how it doesn't scale is another thing entirely.

So I have to ask, so you can't you you came to Instagram, it was built on Django.

Yes.

And then you know, you know, and for me this is you know, super interesting topic because I've never worked on anything like that kind of scale.

But I've worked on some big sites, and people say, Can you scale Django?

I'm like, Yeah, of course you can scale Django, but then there's a level above that.

So I guess you know what what can can you sort of talk us through the the evolution of um to the extent you can for Instagram and how they hit those scaling problems and that you know and the approaches that are needed to take and things because it it's it's something that I think most of us will never see.

Um I gave a talk in 2016 at Django Under the Hood in Amsterdam, uh, where I really kind of go through a lot of that history in in detail.

So it'd be an interesting thing to look up if you're curious about sort of the the scaling challenges as they came.

My memory uh from the research I did for that talk is uh that as you'd expect, I think the the ORM was one of the first.

Well, there were some some kind of funny minor things that hit earlier on.

Like turns out sending an email to the site maintainers every time there's a 404 uh that didn't scale particularly well.

That uh became evident early on.

Uh but but yeah, as far as bit sort of big components, I think the ORM was the first one.

There was a uh there was a need, I think, after about a year already to be able to shard the database.

And uh at that point it was still pre-multidabase, I think.

Uh I'm not sure.

Multi-database support might have happened around that time, but I don't Instagram certainly didn't have access to it.

And uh and there was no real support for uh for sharding uh databases in the ORM.

So Mike Krieger, the the this uh founder of Instagram, basically sort of wrote a a sharded ORM, kind of a very bare bones sharded ORM, and moved from the Django ORM to that.

Uh, and then later from that to Tau, which is a uh graph database built at Facebook, uh, which is what Instagram uses for the primary data store now.

Here's an anecdote, I'm not sure where I heard it, but I believe it to be true that in the early days uh there was a Postgres issue with the space allocated to users was so big that they couldn't figure it out.

And I think Frank um at uh Revsys, there was some hack at the time but helped them with, but just these crazy scaling challenges when I think it was just you know the two founders and pushing the limits of software at the time.

One of the funny stories that uh definitely people told when I first arrived at Instagram was uh in the early days uh once Instagram sort of really kind of made it mainstream, and I think Justin Bieber was one of the first big Instagram users.

And uh because of the way that the sharding scheme was done, essentially like a user and all of their media were sharded together, and uh which makes sense for sort of access patterns, but uh it caused a bit of a problem when you have sort of very popular users like Justin Bieber would make a post, and then everybody you know goes to like and comment on that post, and all of a sudden you have a ton of database activity all concentrated on one shard.

Uh, and that apparently caused this was before my time at Instagram, but apparently that caused a lot of problems and sort of memorizing Justin Bieber's user ID was uh a requirement for uh Instagram operations for a period of time there because sort of every time he post made a popular post, it was sort of an operations emergency that everyone had all hands on deck kind of thing.

But those things are in a lot better shape now.

Whereas the the average user, you well, not you or me, but like just small small people, you'd fetch them on demand because they're not many fetches.

Uh so it they're still using Python, but they're not using so much Django.

Can can you talk about where the stack is today in terms of is there any Django left in there and how much is Python as opposed to other backend languages?

I mean, it still runs through the the Django request handling stack and the middleware and views and URL routing, all of that is still Django would look familiar to any Django developer.

Just very big, lots of views, lots of URLs, lots of code.

Um, there's a lot of, I mean, obviously the ORM we don't use anymore.

The admin is very much based on the ORM, so that's not in use anymore.

I mean, there's a lot of components where Django provides kind of pluggable back ends like sessions and authentication.

And uh for all of that stuff, we've essentially rewritten the entire system, but we're actually still using the the uh sort of Django API for those things because it's pluggable.

So I would call that a success story.

I mean, in that we've been able to very smoothly migrate away from the components that uh that no longer worked at our scale, and in many cases without even having to change the client code uh touching those subsystems because Django provides this ability to swap out the uh swap out the back end.

So yeah, and you know the Django core request support that we still use, we're we're 100% async now, so we've had to uh I mean already a few years ago we kind of forked and modified a bunch of that to support uh async or concurrent handling of multiple requests using async.io.

Uh so there are have been some changes, but

Um but talking just about that that core line, there's only like a 160 lines in base handler, right?

There's nothing there.

So there's no the point I made in the talk was that there's there's no room for it to be slow and clunky.

I mean that Django hex sometimes gets described as slow and but when it's not doing anything, it can't, you know, it can't be.

It's still whole doing that same thing that all the reb frameworks are doing.

So that's kind of that's really nice.

I was wondering, would would Instagram at the scale they're at need needed to have moved away from Python to a you know a faster language, quote unquote?

Uh Python is not particularly efficient in how it uses uh CPU, and there would be more efficient languages, and that would certainly save on server costs, but uh there's also a lot of code that would have to be rewritten, and that uh also has a cost.

So there's certainly always a debate about the pros and cons there.

Uh so far the conclusion has been that we'd rather work with Python and try to improve Python and make it more efficient uh rather than trying to rewrite uh all of the existing code in a different language.

Okay, that's cool.

So can you what might that mean?

I mean,

So one of the projects we've talked about publicly is something uh we call strict modules.

So, like one problem that we have with Python is the fact that uh when you import a module that can execute arbitrary code, which can do anything, it can go out and talk to the network, it can uh end up being very slow, it can fail, and uh when the code base gets very large, and there's really so you can have all these side effects of import, and sometimes one side effect of importing one module can be a prerequisite for some side effect

of importing another module, so they have to be imported in a certain order, and that situation can arise even accidentally over time, but yet the order that modules are imported isn't really fixed or determined anywhere, it's kind of an emergent property of the entire system and all the imports and all the modules, so you end up with this very kind of fragile state where somebody can remove or add an import in some module in one part of the system, and that has a cascading effect of changing the order of imports, and then something can fail just

because two modules are being imported in a different order.

So uh, and another side effect of that, or another consequence of that, is that if you make a change to a Python module and you have all of these modules loaded and running in your web server, say you're a developer working on the code base, you make a change, there's really no safe way to say, let's just reload the one module and sort of hot patch this one change that's been made.

There's there's nothing you can do that's safe other than start from scratch, start the entire process over again, and begin the entire process of importing and loading all those modules.

And you know, when your code base gets very large, that takes 30 seconds, 40 seconds, 50 seconds, a minute, uh, which is a long time for a developer to wait for making a simple change to one view before they can test that view in their browser and see the effect of their change.

So something that we're trying to do with strict modules is do some static analysis to restrict what is allowed at the top level of a module.

To say, you know, at the top level of a module, you can define classes, you can define functions, you can use decorators as long as they don't reach out and have external side effects.

Basically, like you can do all the things that we normally associate with Python at the top level of a module, but they have to be contained.

The only effect they can have is within that module.

Uh, and it once you can guarantee that that's the case, that unlocks a whole lot of possibilities because now you can just safely uh re-import modules, it doesn't matter what order modules are imported in, all of these problems go away.

So that's one one kind of example of a project we're working on more in the developer efficiency side of things.

I when you you talk about the static analysis now, I mean, how does that tie into the kind of thing you see with the um type inting that is coming in and um like the Pylance um thing that Microsoft have got that doing the static analysis there and MyPy and the related tools?

Is it the

So it's but you're using you're leveraging those text.

Yeah, for sure.

It's uh I mean so strict modules in particular doesn't make too much use of of uh of type hints, but it's definitely all in the same uh kind of area, which I think is a really uh productive and growing area right now of figuring out how we can uh do more advanced analysis and make stronger assertions about what Python code does and doesn't do without actually having to run the code to find out.

Um and so type hinting is one example of that that um we've been very excited about.

We we started adding type hints to our code base, I think, already in 2017, maybe 20, yeah, I think 2017 already.

Uh so at this point, our server code base is basically 100% typed.

Everything has type annotations, and we found that to be a big productivity boost for developers and makes the code safer.

Uh, and so we're looking at you know how to further leverage uh type ins and and static analysis to make Python both faster and and more efficient for developers.

Because we had a discussion about whether we would bring type ins, type anything into Django itself, and we were the conclusion that the technical board came to were well not right now, because it's difficult and you know the the technology is still nascent, and uh we ha.

So if you could give us some advice, like what would you say to you know, if how is it feasible?

Are we gonna be able to get Django there?

What should we be looking at?

Can can we start small and grow in?

Well well, just as your sort of you know yeah.

Uh I think it's an improvement to the language, but I I I think that the decision the technical board made on that was a reasonable decision, probably the right decision.

It is very early days still.

I mean, uh at a place like Instagram, we can kind of uh go all in on something that's still a little rough around the edges, and um our developers are kind of willing to put up with some rough edges in order to get the benefits of typing.

But for a framework like Django that is so widely used, uh I think it does make sense to wait until the uh the technology has matured a little bit and some of the rough edges of usability are are smoothed out.

Um so I think I think that was a good choice.

I think you know uh it's always a challenge to bring static typing to code that wasn't developed with static types in mind uh because patterns are used that just uh don't lend themselves to static typing.

So I think you know it's been fine for us at Instagram because you know we don't use the ORM.

Uh there's parts of Django that are particularly challenging for static typing that we don't uh use anyway.

So for us it's been pretty easy to uh just have some stub files that cover the APIs that we use, and uh that pretty much covers our needs as far as typing Django itself.

It will be challenging to do a good job of typing all of Django, especially given the backwards compatibility constraints.

The other thing is when we're typing Instagram code that was written in a very sort of dynamic way that doesn't lend itself to static types.

You know, if it's internal code, we can just rewrite it or you know change the pattern.

And for Django, that's much more challenging because of the backwards compatibility constraints.

So uh I think it would be great if someday, yeah.

We can just you know go through the entire code base and change all the call sites and make sure they all do the right thing and just ship that as a diff.

And you know, you can't ship diffs to all of the code that uses Django in the world.

There's always this, I always have this mystery of when someone like you, so a Django core developer, goes to a large company, it's a little bit of a black hole in terms of like what do they do there?

Like, how does it work?

Do you come in and they say, what do you want to work on?

Or do they say you have to do this?

Like, how how does that evolve, right?

When you I mean you're focusing on you know eking out performance in Python and and and advantages for developers.

But yeah, I'm just curious what your story was like and if there's any sort of standard paths when you know a well-known figure goes into a large company and then you know isn't doing open source work all the time.

Uh Facebook and Instagram have a very kind of engineer-driven culture.

So, you know, there's a a pretty wide latitude for an engineer coming in to pick the teams and the projects that are interesting to them.

And uh the culture is sort of you you decide what you want to do, and then you have to show that the things you did had some benefit rather than somebody telling you what to do.

Uh, so there's a lot of opportunity in that.

It can also be a little bit uh scary.

Uh, but uh but yeah, so that was my experience coming in, was that there was a lot of freedom to to say, well, this is this is what I'm seeing here, this is what I think is important for us to work on, and so I'm gonna go work on that.

Um, but I I'll link to that post.

I mean, that was it was pretty amazing that even at the size of Instagram's code base back then, a small team was able to do it relatively quickly.

That was one of the first projects I worked on.

It was already uh kind of on it was kind of it had just been started when I joined, so it was kind of something that was already happening that was easy to jump in on, and I thought it was important for us to do.

So uh I worked quite a bit on that, on that project.

Um, and yeah, I mean it it felt like it took us a while, but I think it was under a year, which is impressive given the amount of code that we were talking about.

So when I joined in 2016, we were on Django 1.3, which was already quite old at that point.

And uh we upgraded to 1.8.

The biggest motivation for that was to allow the Python 3 upgrade because Django 1.3 didn't support Python 3.

Uh so we we did the 1.8 upgrade, uh, we did Python 3, and you know, the problem at this point is just given the uh amount of Django code that we use, um, you know, there's some big deprecations that happened in 1.9, 1.10, things like um string views went away, string view names and URL configurations went away, and there's a bunch of new restrictions around like uh the import of models files.

Anyway, some things that would be a fair amount of work for us to adapt to, and there just really isn't a lot of corresponding um benefit.

Uh, because the I mean, you know, I don't think Django's development should be oriented around the needs of a company like Instagram.

It's that's not the 90 or even 99% use case.

And so the things that are being done in Django by and large are things that we either already have our own solutions for or just aren't going to be relevant to what we're doing.

So I think at this point, given the forks we've already had to make for things like async, the most likely path forward is that we end up pulling in the remaining Django code that we use and just kind of continuing to iterate on it as kind of our own internal thing, uh, rather than probably ever doing another Django upgrade.

I guess as you say, it's not even the 99% that you know five nines out, it's like the one in a one in a million.

Um but it's also kind of sad.

It's like, oh, you know, it's here.

Yep.

Yeah.

I don't even really have a clear sense of what's in the recent Django versions, because I'm not getting to use them, obviously.

So that is a little sad.

But anyway, I'm still having fun working on Python, so it's all good.

You'll have to start a side project.

I wanted to um start ask you then.

So if you're not getting to contribute to Django and work on Django anymore, what about other things, open source things?

Are you are you able to do open source work or

One thing was uh we had kind of a silly issue with uh the way that our development process works via the local development server.

We actually run USGI, which would run in production and also for local development, but it runs as root just because of kind of legacy of how the operational systems developed.

And so it was if if you enabled PYC file creation, it would sprinkle all these PYC files owned by root through the code tree.

That obviously doesn't really work.

So at some point years ago, before I even got there, someone just turned off uh PYC file creation for local development.

And at some point after I got there, I was looking at this problem of how long it takes to start up the server in local development.

When someone makes a change, how long it takes to restart it.

And I was like, well, we have no PYC files.

So not only are we re-importing all these modules every time someone makes a change, we're actually like recompiling them from scratch because we don't even have PYC files, and that's costing us like 25% time, which is really silly.

But then there's there wasn't really a great solution to that.

So I just wrote a feature for Python where you can set an environment variable and say, put all my PYC files over in this other directory somewhere else, and just kind of have a parallel directory tree elsewhere of all your PYC files, uh, which works great for us.

Now we can do that and we get the benefits of PYC file caching.

So that's just one small example of like uh and then yeah, I contributed that upstream.

I think it came out in Python 3.9, maybe 3.8.

I think 3.8 came out in.

So so yeah, a few things here and there with Python, but not as much as I would like.

And do they do they give you time?

I mean, like um, I don't think they'd still do it anymore, but Google used to have this 20% time or whatever that I don't as I said, I don't think it's still there.

But do they did do you get that kind of thing, or is it only if it's project related?

You don't have to

Um it's kind of like I mentioned earlier, like anything you work on here, it's like you can do what you want, you just have to at some point be able to say, hey, hey, hey, I did something useful this app.

So it's kind of up to you how to how to allocate your time.

Okay.

That's cool.

You know, in Django, there's like I like there's this project called Django Waffle that's really great, but Instagram is a whole different beast.

Um, are there things what things crop up, you know, flipping features?

Like I think I've seen Instagram's been playing around with removing the like button in Canada or a whole bunch of countries.

Like, what does it even take to roll out something like that?

That's like very far away from the things that I deal with on a daily basis.

But certainly feature flagging in general is uh is critical.

Um, and you know, not even not even sort of for sort of big experimental features, but for almost anything uh that gets rolled out, gets rolled out behind a flag so it can be turned off.

And one, I mean, in some ways, uh that's not that's not just about sort of experimentation, it's also just a necessity of working at this scale because um the changes to the code base are coming so quickly that there's there's really no possibility of maintaining a long-running branch for a new feature.

Uh so you can't have this development style where you sort of branch and work on your feature for a long time until it's just perfect, and then like a month and a half later you merge a big branch back in.

There's just no way that would work.

It'd be a huge

So it really forces you into this uh sort of continuous integration development style where everything has to land in the same trunk branch uh that's constantly being rolled out to production in fairly small increments.

And so that means if you're developing any significantly sized feature change, it forces you to figure out how I can do this incrementally and backwards compatibly and have things in a fully working state at every point along the way, and you know, gate the feature off until it's really ready to be rolled out and all that stuff.

So so yeah, it's a very big, big part of what we do.

It's migrating the database store and how uh when you take the field out or when you put it in, or yeah, yeah.

There's no such thing as like, okay, I'm gonna synchronize this code rollout with this database migration, and everything will roll out together.

Like every change you make to the data store.

I mean, first of all, it's a big deal to roll out any uh change to the data schema, but any change you make has to be backwards and forwards compatible.

You always have to have things work if it's new code running against the old schema, or if it's old code running against the new schema, all that has to work.

So um and there's certain that certainly has a a cost in terms of attacks on development speed, but it's just it's the only optional.

You know, I mean, because I I know of like small sites that take an hour, you know, because they have an optimizer done this, that, and the other thing.

I mean, how is it not take months for a little com committed Instagram?

Um the bad news is that I have mentioned earlier about sort of how early Django really pushed you into this path of writing a lot of kind of end-to-end HTTP tests using the test client.

And there's a lot of that legacy still in the code base.

So the bad news is that the good news is we have lots of tests, the bad news is a lot of them are slow tests.

When you have a lot of slow tests, then you have a very slow test suite.

Uh so really, I mean, our our only solution to that is that we uh throw hardware at it.

So when we I I I literally think that nobody in the world has any idea how long it would take to run the Instagram server test suite and and sequentially one test at a time, because I don't think I think it's been six years since anyone has done that.

Like when we actually run the test suite, it runs uh sharded out to a bunch of different machines.

Um so like there's like a bunch of machines all running tests in parallel and then sending the results back.

And with that kind of a system, you know, we can effectively run the test suite for fifty in 15 or 20 minutes, which is uh good enough.

Um but yeah, enough to go

Yeah, yeah, exactly.

So that's

I like your point though on hardware though, like because that reminds me at um the only thing at scale I worked on is Quizlet, which was a top hundred site at the time, and we were on Joint and we were had the whole executive team come in and this, that, and the other thing.

After a full day, they're like, Yeah, you just need more hardware.

You know, it's just like the people who wrote it were just like uh hardware.

Sometimes throwing uh computing power at the problem is the most efficient option.

Oh, there's something in a way beautiful, like you know, from a business perspective, if you can just throw money and solve an engineering problem, that's sort of great, actually.

It's like you'd take that.

I mean, maybe it's a bit more than that.

Uh we just said Oddbird is still going strong.

But yeah, it's still going still going well, still using a lot of Django.

I think they're up to six or seven people now.

So it's uh if you need a a Django application, uh Django site uh or application built and you're looking to contract it out.

I mean Oddbird does everything from the back end to the UI design and kind of figuring out what the user needs are, so it's a great option.

So we're on video.

Uh you're not in California, you're remote.

I I think you've been the whole time you've been remote at Instagram, is that right?

That's right.

How has that been?

But I'm just broadly curious how you know remote learning the last odd number of years and then today where everyone's out of the office.

Uh I I never really had a a big issue with it, uh, even when I was one of the few people remote.

I did travel uh out to the office in California pretty frequently, which I think is important.

There's really no substitute for getting to know people uh in person at some point, and so just to sort of have those relationships with the team, and I think it was important to make those trips out.

But on a daily basis, you know, we have good video conferencing.

Uh pretty much everybody's already using, you know, the funny thing is, like the the Instagram or Facebook offices are so big that half the time people are meeting over VC anyway, just because they're in different buildings and they don't want to walk across to the other buildings.

So uh really everybody's already using kind of these remote working tools on a daily basis anyway.

So I didn't feel that out of place using all the same tools.

I I felt like I was able to keep up.

And now, of course, yeah, it's the world that everyone lives in.

So right.

I was actually, I think, on the same block as the Instagram founders.

I was on 2nd Street, um, you know, around the same time quit uh 2010.

But it was it's you know, it's even if I'm sitting right next to someone, I would send them a you know a chat message because it's rude to poke someone with headphones on in the middle of something, right?

It's like, you know, unless it's lunch, like I don't want to get poked on a question, just like you know, toss it in my chat thing and I'll so now here we are, you know, extending that out.

But for you know, yeah, it makes sense, right?

It's it takes however many fifth ten, fifteen minutes to get into a flow state, anyways.

And if you're constantly being interrupted, even if you're right there, yeah, it's rude to poke a coworker.

Yep, right.

It's really interesting chat and love um love to get the back story and then you know the story of Instagram and how they grow up and you know that's super.

Thanks for coming.

Thank you.

It's been a pleasure.

Were you gonna say something, Carlton, or did I talk over you?

Bye bye.

Thanks, guys.