← Back to Show Notes

Transcript: Ethical Ads - David Fischer

Will Vincent

Hello and welcome to another episode of Django Chat, a weekly podcast on the Django Web Framework.

I'm Will Vincent, joined by Carlton Gibson.

Hi, Carlton.

Carlton Gibson

Hello, Will.

Long time nice.

Yeah.

Will Vincent

And this week we are joined by David Fisher from Read the Docs.

Hello, David.

David Fischer

Hello.

How are you?

Carlton Gibson

Marvus.

Thanks for coming on, David.

Will Vincent

Yeah, thank you for coming on.

There is a whole bunch of things I want to ask you about, but maybe we'll just start with your origin story.

How did you get into programming, Python Django, and then we'll go from there.

David Fischer

Oh, all right.

Um, so I ha I have a pretty like standard origin story, you know.

I sort of studied math in college and loved programming before that, wanted to get into it, sort of had a programming heavy undergraduate, and went right into like a job at a you know pretty big tech company right out of college.

So that was a while ago.

Uh but now I sort of decided, you know, I'm done working for these big companies.

I don't I, you know, I'm not like super negative about them or anything like that, but I I wanted to get my hands a little dirtier, wanted to be a little bit closer maybe to both the business side and the code side.

And so I started working for more on the startup, startup size companies.

And and read the docs actually a couple years back, about not quite three years ago, was a pretty natural fit uh for me as they were sort of building out their advertising platform.

Will Vincent

Got it.

And uh when did Python fit into your programming journey?

David Fischer

Oh man, uh I took like one class that touched on Python in college, and it actually helped me get my first job, which is sort of a random thing.

You know, I I did Python in one class, and then this job application was like people who know Python.

I was like, that's me.

I know Python, you know, as much as an undergraduate who doesn't know Python knows Python.

Um but that it sort of helped me get my first job, didn't do Python at that job for years, and then maybe four or five years into that job, uh they were like, do some Python.

So that was that was how it ended up working.

I was already working on some web stuff and I ended up picking up Django.

I sort of like looked at a couple of the alternatives at the time, and Django, this is in the 096 days, was the that it it ended up being a great fit.

So that was how I got into Django.

It was a long time ago.

Will Vincent

Carlton, what version was it for you?

Carlton Gibson

I don't know, about one 1.1, one something around the 1.0 times.

I because I remember going to a conference and there was this uh everyone was like Django, Django, Django.

So it must have been around the 1.0.

Um around that time.

Will Vincent

So um maybe for listeners, what is um maybe Read the Docs?

What's the quick story on that if they're not familiar?

They I'm sure most user uh listeners have seen Read the Docs, but maybe they don't know the story of Read the Docs.

David Fischer

Yeah, it it's sort of an interesting story, and I'm not the main sort of protagonist in this story at all.

Sort of uh Eric Holsher and Anthony Johnson are sort of the main the main folks in this story.

I I came much later.

Although I was an early user of Read the Docs.

I think I started using Read the Docs in like 2011, you know, had an account.

You know, I I sort of looked back in my password manager, it's like created date.

I think it was 25 or something.

Exactly.

David Fischer

Yeah, exactly.

That's how you figure that out.

Um but it I think it started at like Django Dash in 2010, which was like a one-weekend sprint project, and and it was basically like the automation around Sphinx.

That was that was exactly what it is.

So, how do you do continuous documentation not building off of every commit to master or you know, something like that?

And so Eric Holsher, I think at the time, was probably one of the main people behind it.

There was a couple other people at that time who ended up not sort of transitioning, not launching it as a company.

And it just sort of took off in the Django community and in the wider Python community.

People started using it, and Eric had a day job, you know, working a regular job like a regular person.

And he would get these pages or calls being like, hey, read the docs is down, and he's like, Well, you know, I got I got my regular job, guys, you know, like so um eventually just it was so big, so many people relied on it.

I think Eric and Anthony decided to launch a company around it, and and basically it's funded through a combination of advertising on open source documentation.

They sell sort of ad-free commercial documentation that has a few additional features, so sort of companies can buy it.

And then there's um there's also sort of uh like what what's called read the docs gold, but it's basically like regular people can pay a little bit of money and you get a you get some extra perks on read the docs, like ad-free on read the docs, and you can sometimes designate a project ad-free.

There's a there's a few different things you can do.

So that's sort of where it all comes from.

And uh, you know, I I sort of looked at this and I Eric had produced sort of this guide on how their advertising was different from other people, you know, it was sort of no advertising without without tracking people, so pretty much the opposite of what most advertising does.

Um and and I sort of just emailed Eric out of the blue.

I'd met him before at a conference, like you know, probably seven years before.

I'm sure that he had no idea who I was, and I I barely remembered who he was at that time.

But I emailed him to ask him a few questions about that, and and he was basically like, You you should you should work our advertising.

And then I was like, I yes, I should.

So that was that was how I got the job at Read the Docs.

I sort of did a few projects as a consultant for them and then transitioned over to full time.

Carlton Gibson

Before we go off into the ads uh story, which it's there's much more to talk about there.

Can I just wind back a bit a little bit?

In the early days, was it like Eric just like hosting this on his own server?

Like, because I think like it it grew up quite big quite quickly, right?

So where's the cost?

Great question.

Carlton Gibson

It's all right if you've just got your blog be at your hosting when you're hosting.

David Fischer

And uh this is this is where a number of the problems came from, you know.

Um so it was it was when I stepped in here, I was like, man, this is hosted insanely cheaply.

You know, they're doing you know 50x or even more, maybe 100x the traffic of some of the other properties I've worked on before.

And they have a budget, like a monthly, you know, infrastructure budget that's like the same as other places I've worked that did, you know, 2% of the traffic that read the docs did.

So it it was it was built to run insanely cheaply.

Now, it it has some advantages.

Read the docs is almost entirely hosting static content that is already built, you know, just straight HTML, CSS, JavaScript that's built through something, uh, you know, built through a a builder.

So yes, you have to run builders, that's kind of expensive.

But the but the hosting and the serving is relatively inexpensive.

So there are some advantages there.

But yeah, read the docs just to, you know, our our stats are pretty public, we're up front about them.

It did something like in August, 45 million page views uh in last month.

So a lot.

David Fischer

It does a lot of traffic is is the answer.

And that's just the open source side of the hosting.

That's not the commercial hosting.

And we we have some privacy protection in there to like, you know, we don't we don't send anything to analytics if somebody has do not track marked on their browser, things like that.

So it doesn't count any of that.

You run an ad blocker, you don't get counted.

This is 45 million page views, discounting all of that.

So it does, it does an absolutely massive amount of traffic.

Carlton Gibson

And tech people are quite likely to run it.

David Fischer

So it would go down all the time because it was hosted on a shoestring budget.

Eventually, Eric sort of moved it over to AWS and had like load balancers, you know, like things that you would do if you wanted something to stay up.

Yeah, crazy stuff.

Yeah, crazy stuff.

David Fischer

And and this worked pretty well on still what was at the time a relatively small budget.

But they they launched sort of some crowdfunding campaigns.

Some of these brought in like real amounts of money, you know.

They they did like a big crowdfunding push, and you got they got like a real amount of money.

It was like $30,000, which sounds like a lot until you realize that's like a few months of infrastructure budget.

But like it was all one-time donations.

And so then like the month after they got $30,000, the next month it was like, oh, we brought in $1,000, which is you know, that that's gonna be underwater on on the infrastructure budget.

So they realized, what do we do here?

And the answer was mostly advertising.

Will Vincent

Well, I remember that post.

Maybe if I find it, I'll put it in the links about you know, talking basically kind of what you said and the you know, not joyously jumping into ads, but basically being like, we need to cover the costs.

And I remember thinking it was really well written and really sort of laid out the dynamic for a lot of people, which is yeah, it's hard, it's hard to charge and can't lose money on something

David Fischer

that's yeah, that's a side project, essentially, or or it was at one time, you know.

Yeah, you you're absolutely right.

And and there is sort of the reality of the situation that you know the budget of Read the Docs is a rounding error to somebody like Google.

It might even be a rounding error to somebody like like GitHub, even pre-Microsoft acquisition.

So like they could just launch it and if it loses money, whatever, no big deal.

But um, you know, for uh for us it's like real money.

You know, commercial hosting does not bring in as much money as advertising for us right now.

Will Vincent

Wow, that's interesting because Yeah.

Yeah.

David Fischer

Huh.

And and you know, like we're we don't have venture capital backing us.

There's no sort of like money banks, daddy, daddy money bags behind us or anything.

It's like, oh, we bring in a little bit more money, that means we can hire one more person.

So it's all sort of bootstrapped.

There's no there's no venture capital at all.

Right.

Will Vincent

Well, I think it's in some ways that forced discipline is the best thing you can have.

I mean it's unpleasant in some ways, but I mean when I back ten years ago, I was working at a company called Quizlet, which was a top hundred website by traffic with what was it?

Two and a half engineers, not a big budget.

And I think in many ways my tenure there, which was about three years, what we the main thing we did is we didn't harm it.

We just 'cause we I spent so much time trying to recruit people.

You know, we kept it free.

We kept it up and you know those constraints were a good thing.

They were frustrating.

Right?

Will Vincent

Because you always have your long list and you're like man but it really does uh focus priorities and I think can also lead you away from sometimes if you have 20 engineers it's like well they gotta be doing something.

And even though maybe that's not what your business needs.

David Fischer

Yeah you're absolutely right you know so yeah constraints sometimes are are sort of that's how invention happens.

Will Vincent

For sure.

Well I uh I love that your so your code is open source.

You're the ad server you know client server uh server and client are both up there.

And so I was actually in prepping for this interview going through back to the first commit because I love seeing how uh people build things.

And I wonder if I'll put the link in for people but essentially it's I love actually how simple the project is even today.

It's essentially a single um ad server app within Django because it's quite easy to I would say bloat out a Django project.

And you've been very constrained but I wonder if you recall you know building you know starting from the kind of that process right like what did you start with and start with to the extent you can recall like the you know the changes over time because there's a big difference between prototype to first stage production to you know the scale that that you are at now.

David Fischer

Well you're probably looking at a commit that's not that old.

Will Vincent

I think it was end of 2018

David Fischer

maybe yeah yeah that so basically when we when I first when I started at Read the Docs the the advertising was basically a Django app that was closed source but got built into Read the Docs at compile time.

So there's sort of these private extensions in Read the Docs that are in a private repo.

The read the docs the main read the docs repo is all public but we have a couple private extensions repos you know one is for the things that commercial hosting gives you but some other things we have some other just closed source that are closed source for for a variety of reasons and advertising was just one of those it was just in a different repo it was just one Django app and it was closed source.

And so when when if you looked at the first commit of the ad server it's probably just mostly taking a bunch of stuff from this private read the docs repo and bringing it into here.

But yeah a bunch of things were sort of renamed.

Will Vincent

It was still pretty iterative um yeah it was I saw I think yeah that I think the you know the first commit was just you know first commit nothing and then the second commit was import um add server.

But then I mean I could see you you you started with basic auth and at some point you added uh Django all auth.

You know kind of all the to me standard steps that um you know not every Django developer gets to do because you often you just parachute into existing projects and you don't have that flexibility and you don't do it a lot from scratch.

So I always love seeing actually how like a production site is done iteratively because that's not an experience many people have.

David Fischer

Yeah it it's actually sort of a crazy experience because I I got dropped into this project and we were basically like we're gonna break our ad server out from read the docs because previously it was just a a Django app in read the docs and you know it does a very large amount of traffic something like you know 30 million ads a month.

Most of those are not paid ads but you know still we're we're talking about yeah 30 million API requests a month it it's kind of expensive.

And and then there might be additional requests on top of that.

So like when we broke out the ad server we had to build something that on day one is is going to handle you know 30 requests a second sustained 24 hours a day.

Will Vincent

No pressure.

David Fischer

No pressure you know and and yes the first time we tried to stand it up it absolutely fell so

Carlton Gibson

okay what does 30 requests a second um sustained look like in terms of infrastructure in terms of the you know what are you running what's you know how many workers are you running how many you know is it using a threaded model or pre-fork you're using garnicorn you're using UWISGI work like how do you how do you solve that much

David Fischer

so it it is pre-fork it's we are using Gunnarcorn uh and it is I think we're looking at four it

's either four or six workers I know I've tweaked this setting so I don't remember what it what it is I could check for you but it's either four or six or eight workers per per sort of instance and we're running I think we're at currently six instances.

Okay.

David Fischer

So that that's about where it is and that that handles it fairly well and we we're right now we're we're hosted on Azure uh we we started out we were we were prototyping on Heroku but now for production we're on Azure that's where read the docs is so we decided we'd just be on the same infrastructure.

It is set up slightly differently than how read the docs is set up read the docs is also on Azure but it's basically just using like base VMs and what are called scaling sets where you can just sort of scale the number of identical VMs that you have.

So that that's how that that's how read the docs is set up.

So read the docs actually auto scales.

Carlton Gibson

But that's super interesting because you know a lot of people you know you you you'd have no idea what it takes to run a site at bigger scale right you you you you build you build your little thing locally okay fine you put up a worker you run it out and how do I how do I plan in advance for if I want to grow you know to that kind of traffic that well you need to think okay you're gonna need half a dozen servers you're gonna need you know this kind of infrastructure that's yeah you know so it's really nice that you can come on and share that kind of information with

David Fischer

and it's much e

asier to build something that will handle that kind of infrastructure than it was 10 years ago.

Ten years ago?

Because it would have been much harder.

Carlton Gibson

Yeah no I mean the the this is the this is the cloud thing right it's I if I need six servers I just get six servers.

It's not

David Fischer

yeah click a few buttons in the AWS dashboard or whatever you know the Azure dashboard just drag the slider to the right and you know my bill also scales linear linearly

Carlton Gibson

okay so you're running the ad server as a s as a kind of um a a micro a a massive service uh on the side

David Fischer

yeah yeah yeah it um I mean the the big reason why we wanted to push it out of read the docs is that we had sort of this vision which just sort of started to happen a couple mont

hs ago of basically taking the ads that we serve for read the docs and making it so that we could help other projects you know other tech projects other sort of similar places like read the docs that need to earn some money how do we help them run ads and we did we didn't want them hitting sort of readthecs.org API endpoints so we sort of said hey we'll break this ad server out it'll be sort of its own thing yes it's part of read the docs yes read the docs is like the primary user of this service but you know we wanted to break it out so

it was separate infrastructure all that kind of stuff

Carlton Gibson

right and so this is where it gets really exciting because it's ethical ads right yeah so I so I'm I'm putting up a site and I'm thinking to myself oh I need to make some money but I can't bring myself to put the Facebook tracking pixels in and the the Google tracking pixels in because I just can't bear to be part of the massive surveillance capitalism world that we live in and now and there's an alternative.

So tell us about the ethical side of it and like why I might choose ethically

David Fischer

right so you know that this is exactly what I emailed Eric about it was probably about three years ago now almost exactly and I you know I had some questions about this.

You know how does it work?

How you know how does it work relative to something because I I had some familiarity with my last job I I wasn't in the marketing department in my last job I was the head of development but um you know the the marketing team would come to me all the time they needed help and I was sort of the liaison with the marketing department.

I helped the marketing department anytime they needed any tech stuff.

And I can remember sort of like helping them set up advertising and I was basically horrified at everything that they were doing.

You know I I have a bit of a background in in security some of that extends into privacy and and basically I was horrified you know sort of standard procedure in this world is take your customer list upload it to Facebook and tell them Facebook I would like to advertise to people similar to these people that is like that is standard procedure in in advertising you know for anything really not just SaaS companies that's sort of anything.

So that that's sort of like I was horrified at that and so I I talked to Eric about this what what do we do differently and and there's a few different things that we do.

You know one we basically as much as possible don't set cookies.

There are some cookies that are like borderline unavoidable like like the uh Cloudflare cookie if you want so if you want your site to be protected by Cloudflare they sort of set a cookie and some things are sort of unavoidable but as much as possible we try basically none of that um and and and there's a few other aspects we we try to like align ourselves with the site owners not sort of against the advertisers but like advertisers are constantly pushing you to put more tracking and since

with read the docs we were sort of the publisher and we heard very much from our from our users like read the docs regular visitors they don't want tracking they don't want cookies even when I started at read the docs I would probably get an email a week about something privacy or security related you're running Google Analytics we hate that you're you know whatever something like this so I would get all these emails so we were basically like no cookies try to align with the site owners don't run any resources nothing at all

from the advertiser so not just scripts you can't you have to take the images from the advertiser and host them yourselves otherwise they'll cookie your users um yeah so just all these sorts of things you you start to realize like all the I mean cookies by themselves are there's nothing wrong with cookies but like you can do a lot of bad things with cookies if you if you really want to so that's yeah all of this is it's just it it it's a shady industry there's good pla

yers in this industry there's bad players in this industry but like we we try to be one of the good guys there you

Carlton Gibson

are there's Connels from one

Will Vincent

next and you I mean it's read the doc so you're dealing with people who understand some of the implications of

David Fischer

it um yeah you know

Will Vincent

not all developers care about their privacy but enough do that it's a meaningful differentiator for you.

David Fischer

We heard from them that they do so you know a lot.

I would get literally an email a week you know something related to this you know either you know turn off Google Analytics or you know respect do not track or you know something.

So we took a lot of the steps there around do not track I I don't know if you're familiar with it it's sort of like a pseudo standard um it it I it's not a real standard or rather there are real standards around it but like there's no there's no teeth around it.

You can say yeah I support do not track I tell you that you're being tracked boom supported sounds like

Will Vincent

something uh corporations would create

David Fischer

absolutely as law but um you know the EFF sort of has their own idea of what of what do not track stands for and and there's they sort of if you if you subscribe to that it there's sort of real things that you're supposed to do you know keep server logs no more than 10 days if they contain an IP address you know things things like this.

Read the docs actually we did not originally support this but we've moved and now now we're in compliance.

With the EFF so yeah with with this sort of pseudo standard that the EFF has put out for do not track both on the read the doc side and on the advertising side.

Will Vincent

Yeah well I mean we can I sorry go ahead Carlton

Carlton Gibson

good well what what I'd like to ask is a kind of if I'm a site developer and can I can I ask you for your um expertise here I want to do some I I don't want to put Google Analytics on my site and I'm not I'm picking Google Analytics is the big one because of I'm I'm concerned about these privacy issues and I don't want to I just don't want to be part of that.

But I would like some analytics I'd like to be able to s you know like and I can grep my logs I guess and get some idea of how busy my website is but what would you recommend if I'm a small um small website developer how what can I put on that enables me to do some analytics but ethically

David Fischer

it's it's hard I'm gonna be a hundred percent honest with you it's very hard you know I uh there's this sort of newer startup called plausible analytics and they they sort of that that's exactly what they bill themselves as and and

you know some people will say well that's all marketing but it it partially is and it partially isn't they're they're doing some good stuff there so I don't want to I don't want to speak negatively about them they're do they're doing really good things and having more alternatives to Google Analytics is a good thing.

Not cooking users is a good thing.

People who say uh you know oh but just just grep the server logs those people in my opinion have not run a real business like they they don't really understand what they're talking about.

You get so little from that compared with what you can get from from JavaScript based analytics.

You know even if a number of users are are blocking it you still get just so much you get things like the time on the page you get whether they scrolled or not you can you can attach actions to specific things like did somebody click this button you know that maybe that button doesn't trigger another page view or something like that.

You can get all this additional data you also it's much easier to filter out bots.

You know read the docs traffic is like half bots you know some of those are malicious but most of them are just not they're just search indexes indexing read the docs and so you know we would have so little data we we actually do use Google Analytics on read the docs still there we're we're always sort of evaluating a few alternatives here.

We've thought about maybe running it server side where you actually hit a read the docs endpoint and we basically strip out a bunch of stuff and then send it off to Google Analytics from the server side.

And you you could do things like drop IP addresses, anonymize user agents you'd not have any cookies for users so you know you wouldn't have any Google Analytics cookies for users so Google wouldn't be able to tell hey this is exactly this person.

So these are advantages but you know on a site like Read the Docs you're talking about 45 million additional server-side API requests that then have to hit Google while you wait for a response right so that there's there's real

Carlton Gibson

dramatically care about privacy.

David Fischer

Yeah and you could host your own solution but you know we're we're talking about a a real time we're talking about a service that's 45 million requests a month like standing up something that's going to run analytics on that is hard.

It is it will be expensive it will cost many hundreds of dollars a month just in infrastructure and we we tried it a little bit and many of these solutions like fell over at that scale.

So there's there's drawbacks it's hard.

Will Vincent

It feels like to me just broadly with advertising there's even more than usual there's just this divide between if you have a little bit of money you can not see any ads and if you don't you're just gonna get bombarded.

I mean like YouTube right like the last two years YouTube went from almost no ads to now two ads on the start ads every three minutes and um you know probably like a lot of people like I don't watch regular TV really ever and when I do it's for a live sporting and it's just awful because of the ads, right?

Everything is a streaming service and which ties back to so YouTube that they have this premium thing and they have a three month trial and I've been using it and it's just like I'm I'm almost tempted to spend you know twelve bucks a month just to not have the ads.

So I guess which you know which is to speak if there was some global like Chrome ad thing where I just never saw an ad on a chrome site um you know that would be appealing but that's because I have some capital and I value that time.

And I guess totally but I don't like that about the world.

I feel that's that's that shouldn't shouldn't be the case.

I mean that's like capitalism to the whatever power

David Fischer

put yourself though into the advertisers mindset.

Like the people who are willing to pay $10 a month or $12 a month or whatever it is to Nazi ads unfortunately those are also the most valuable people to advertise to that there are.

So you can't charge a market rate for this you have to actually charge way over a market rate for something like this because like the people who will pay $12 a month to Nazi ads are worth more than $12 a month in advertising.

Will Vincent

That's a good point.

David Fischer

Yeah it's it's it's a bad thing.

Like I I don't know exactly like what the solution here is.

Carlton Gibson

I've had on my list I've had on my list to talk about a solution I've had on my list to try out this pie hole thing where you you set up a Raspberry Pi to on your local network and you you set it up as the DNS resolver for your whole local for your router so that all traffic goes via it and it's got a blacklist and it just won't load any ads or anything and I've hear I hear glowing reviews.

I haven't had the you know day and a half

Will Vincent

this is your man of the people uh site Carlton might be

Carlton Gibson

this is a a local solution it may not be you know it doesn't solve the you know

David Fischer

I know a lot of people who run them so you know they they work really well you will run into some issues where like some sites are just gonna not work and you're gonna have to basically somebody's gonna send you a link you're gonna see oh this doesn't resolve at all you're gonna have to log into your your pie hole and fix something and check it out again but like you

will not see ads essentially at all anywhere.

And again like the advertising companies are probably unhappy about this but like again those are the people who are the most valuable to advertise to yeah we at at Read the Docs we've sort of like decided you know ad blockers there's nothing you can really do about it.

People are going to ad block you can try to sort of maybe urge them not to you could try to diversify your revenue stream and and that's essentially the tactic that that we've taken you know you can think about this the other way too is so many people are blocking ads and and yes as a result of that the ad industry has gotten they've had to get more intrusive, more ads, bigger ads, etc.

So like that that's sort of one response to this.

But the the the other side of this is like people are essentially boycotting advertising.

Like it you know it it's a very wide boycott.

A lot of people they hate it.

And for a few different reasons.

I think tracking is part of that problem.

I think just more intrusive ads is part of that problem.

Although let me tell you I remember what ads were like on the internet 15 years ago they were terrible.

You know pop under ads, pop up ads like all those things you know interstitials.

Absolutely those are still there.

David Fischer

But like you know pop under ads are are literally the bane of everything.

I still get probably an email a week from at at the read the docs advertising email to be like you're losing money by not running like the worst ads the internet possibly has.

And and they're certainly right.

Like could we make more money by selling out our user base?

Yes.

True.

Carlton Gibson

Yeah we could but you always have to remember you always have to remember that at the end your heart will be weighed against a feather you know oh yeah.

David Fischer

Yeah so we we don't do that but we you know I get an email a week probably about it.

Will Vincent

It's also that that short term thing where you can always if you do any sort of analytics it's always going to show that you should do it because you can't measure the long term you can't measure the subjective brand quality.

I mean even for a a the pup uh someone who has ads on their site like I was at Quizlet top hundred website we had a new Google rep every six months you know another 23 year old and every time I was like is there actually something useful you can tell us and you know I don't blame the pr the rep, but all they could say is a bigger ad more prominently up there.

And the problem is once you put it on there you get addicted to the revenue from it and you can't take it off and then your site looks like Facebook looks like now.

Yeah.

So it's a it's um it's I I admire sites that don't have it.

I mean I know I I understand how it happens.

David Fischer

Yeah I mean you know if we put a second ad on read the docs for example like that would probably double the revenue.

Right, right.

David Fischer

Like it and and the revenue is good but like we could probably double it by just putting a second ad on there.

But we don't want to do that you know like the there are drawbacks to this and and you've hit it on the head like you got to think long term versus short term.

Will Vincent

But I think users can't articulate that right like sometimes people will tell me with my personal site they love the design and really what they're saying is they love no ads.

Yeah like I don't fool myself I'm like is it the design I think it's that it's just there's no ads on it or there's very small ads on it.

But Carlton you were gonna you had a point.

Carlton Gibson

Well I was just gonna I I was gonna perhaps segue back to the um the Django setup of the ads server.

Perhaps we can go through the third party apps that you

David Fischer

sure yeah I think I I I went through them to refamiliarize myself with them uh for this because you know like some of them you don't you don't work with like every single day so you don't even remember you're like what was that for but uh yeah

Carlton Gibson

okay so here's one I've sit I see on the list we've got in the show notes here is uh you've got Django rate limit that you're using.

Yeah so that's a brilliant app and and we should talk about that because I think people Django doesn't come with rate limiting built in into it.

I know DRF has got rate limiting on the API views but you know tell us about Django rate limit.

David Fischer

Yeah we we actually use it for probably something different than what a lot of people use it for.

A lot of people probably use it for things like rate limiting logins or something like that.

We actually use it for rate limiting advertising you know um you can't there's there's sort of a maximum amount of ads that anybody can click on in any sort of amount of time there's a maximum amount of ads that you know a real person could view in a reasonable amount of time and we sort of use it for those kinds of features.

So it it's for us it's kind of a security feature or a you know an ad fraud feature.

Ad fraud is real like you know I I spend more than my fair share of time dealing with it.

And and you know like it's one of the easy things when you're just Google or Facebook you just you just handle it at huge scale but for somebody like us it's it's hard.

And you know especially when we were both when all the advertising was run on read the docs and you know we we have sort of an incentive to report things correctly but when we now have sort of these third party publishers who were running ads on their site that that's only started at you know I think I mentioned a couple months ago and they sort of are getting paid out on this they have different incentives and their incentives don't necessarily like align with ours and we have to make sure that that things are legitimate.

Yeah okay you know we're we're we we s we basically have advertising for developers developers know very well how to automate clicking on ads or viewing ads or whatever you know and and

Carlton Gibson

well yeah I've just you know I've just learned async IO so now I can click on that ad you know concurrently

David Fischer

absolutely hundreds of times a second

Will Vincent

well that's how you yeah I've been I've you know I've used some version of that to like we've had to log in to like go to the local beach here and

I'm it's like I I almost wanted to just set up a thing you know to slam the site.

Will Vincent

I didn't though too busy with kids but it's like it's right there for you you know you can script kitty it and just it works.

David Fischer

I think I think uh what was it Django rate limit I think made the it made like the newsletter the Django newsletter pretty recently oh yeah yeah I think so I think it was on there pretty recently but yeah so we use it actually for something a little differently than that uh all auth sort of has its own rate limiting sort of built in so we we use that for authentication but that that's not as big of and we we use that both on our ad server and on read the docs all auth that is but yeah rate limit is specifically for sort of man

ually rate limiting advertising as a as an ad fraud feature.

Will Vincent

Can I ask so there's a number of so you're an international platform and there's a number of packages around you know country specific things.

I guess broadly can you talk about the challenges of moving beyond just being US based and a global supportive Django project because it's that's a whole nother thing.

David Fischer

Got it yeah I it's it's hard but and and I I actually almost don't want to talk about it too much because I don't think that read the docs spends like an appropriate amount of effort there.

You'd be surprised but something like 92% of all documentation on read the docs is just English.

Is it

Will Vincent

so how what percentage is the US?

David Fischer

Oh you mean traffic wise so traffic is totally

Will Vincent

there yeah there's internationalization of language and then there's traffic

David Fischer

yeah so so traffic wise it's about a quarter North America US and Canada so like 21 22% uh US is sort of the US percentage but like in terms of language like written spoken language it's 92% English and virtually all of the remainder is Chinese so everything else is a rounding error sub 1%.

But like

Carlton Gibson

the support for um multi-language um sites right from my I can if if my docs are translated I can serve them in English and in German and in French but the the the the the the effort to do that is just monumental.

We do it on Django but we have like literally a whole team of people who you know and volunteers who do the translation on each language and it's just it's it's the amount of effort you for a solo you know for a smaller project like a you know my Django filter there's no way I could translate the docs for Django filter.

It's just correct it couldn't happen.

David Fischer

Yeah and and this is actually an area uh so Anthony this is an area of interest of him one of my coworkers at Read the Docs he really he like this is an area that's definitely one of his interests and he he has a few ideas here like integrating with some of these third party translation services like TransFX or something like that where you can maybe you know Sphinx supports something similar to what Django supports where you you have these sort of dot po files that you can upload to a service and hopefully get a translation for them and then serve

them.

It is a little tricky to set up a multi-language project on on read the docs but it is possible we have a few projects doing it um probably the biggest one is GoDot which is like a it's a a game engine a C game engine they they're a huge project on Read the Docs totally not in the Python community until you start looking at how does ReadTheDocs make money?

What are our biggest traffic projects I would have not been familiar with them at all because you know it's not in the Python community.

It's not something that I work with on a daily basis but they have translations for probably a dozen languages for their documentation.

Okay.

They are also a very they are very expensive to host because their documentation builds take like 20 minutes each and so it's like oh we committed to the main repository now we have to build documentation for 20 languages.

Carlton Gibson

Well that's that that's one thing I wanted to ask is um how clever is the caching on the um on the docs rebuild?

Like uh do you do kind of checksums up front on the docs folder or that kind of thing to say hey no do you know what whilst there's been a commit nothing's changed we don't have to rebuild

David Fischer

we used to and we removed most of them and the reason why is because it's actually extremely hard to do it correctly because a lot of times Sphinx projects will use some auto API or something like this that's referencing code.

So it's actually building, it's building docs from code directly.

And so it might be a change that isn't in the docs directory, but because of a change in the code directory, it affects the output of the docs.

So we we just decided at some point this is too hard, we'll just waste resources and have our builds be correct.

Carlton Gibson

So you actually you're in danger of using more energy trying to guess whether you should rebuild than whether you would, you know, actually rebuild.

David Fischer

Yeah, it it ended up being just sort of like a problem that we decided was was too hard to solve.

And and interestingly enough, I think a lot of the CI services went the same route.

So we we sort of model a lot of what we do off of some of these CI services like Travis or or Circle CI.

And that's that's what they're doing.

You know, they're they're not trying to get clever and say, oh, well, we think the tests are gonna pass because you didn't change anything over here.

They just rerun it.

Well, okay.

Carlton Gibson

Okay, interesting.

Will Vincent

I really want to ask about Stripe, but maybe that just it works brilliantly for you, and the new API has been no big deal switching over.

David Fischer

Um we're actually not using most of the new APIs.

We're actually using it to pay out publishers.

It's sort of a little bit different.

Connect size or not.

David Fischer

So not connect in the marketplace.

Yeah.

So basically publishers can sign up for a Stripe account.

We don't get their bank account information, but we can basically transfer money to them.

So that that's what we're using it for.

Probably most of our payouts are still PayPal, though.

And we don't have like an a fully automated solution there yet.

This is one of the things when you stand up your own advertising network over like a couple months, lots of things are not automated yet.

Will Vincent

Well, even um Carbon Ads is uh PayPal.

I thought it would be a little more advanced, but it's not I think a lot of the new Stripe stuff too is around um subscription and um blanking on the European law, but there's all sorts of things around that, which is so it's less so for a sort of one-offs.

Though Stripe is finally adding, I think I think they just added in like tax support because for a normal like someone my size, if I added Stripe, it's like to collect tax per state per country is impossible to do.

Will Vincent

Even if I tried to, which I have, it's just like well, I guess it's a no-go.

Um but Stripe is slowly rolling up all these things around analytics, you know, around tax jar and all these other third party things you can sub in to do taxes appropriately.

David Fischer

But we're actually not using it for subscriptions, which I think is like a lot of what some of the new stuff is.

Yeah, that causes a lot of people.

David Fischer

Yeah, we we use it mostly for you know sending invoices to advertisers and paying out publishers.

Those are sort of our main things.

And the the Stripe Connect stuff has worked fairly well, although there's sort of like a beta for kind like if you want if you have your balance in Stripe in US dollars and you want to pay somebody whose balance is not in US dollars, or actually even just not in the US, it's a problem.

Um, Stripe has beta support for it, so we are like applied to join the beta.

So right now we can only pay out publishers via Stripe in the US, but maybe that'll change in a month.

Everybody else has to use PayPal or something else.

Or they have to give us their bank information.

Right.

Which is yeah, as much as possible we want to not do that.

Will Vincent

Well, that's like my mom there was something not to pick on my mother, but as an example of a, you know, she wanted to write a check as opposed to enter her credit card.

And I was like, you know, your check has your routing and your account number on it, right?

Like you know that and your address.

Yeah.

David Fischer

Checks are way less secure.

Will Vincent

Oh my god.

They're amazingly insecure.

It's just, oh my god, handing that to a stranger when the things I could do if someone gave me a check.

Um but anyways, so uh while DjangoCon uh US has been in San Diego the last two years, will be again next year.

Can you just briefly talk about um you wrote the bid for that, right?

Or you've I wrote the

bid for that.

David Fischer

So um I, you know, I'll combine this with sort of the San Diego Python stuff.

So you know we have a couple, we have a there's a couple people who are maybe bigger, they have a bigger sort of maybe name in the Python community than I do.

I really focus my efforts locally.

I'm not big on social media, like I I have like very little social media presence.

But um Trey Hunter, who's one of the other sort of San Diego Python organizers, and he was basically like, You, you should, David, you should write the bid for DjangoCon US to come to San Diego.

I was like, all right, fine, Trey, I'll do this.

And so I I wrote it.

It was pretty convenient actually because at the time I was working in this office in downtown San Diego, and actually the San Diego Tourism Authority, which is like this quasi-government, well, actually, they are governmental, they're part of the local government.

They were in the same building, so I just sort of stopped by their office and was like, I have this conference, you know, it's gonna br it's it's this big, it's gonna probably bring in this many dollars to San Diego.

I need help writing a bid.

I wrote the whole bid, but like they pointed me in the right direction so much.

You know, Django, Django Con was basically like, you know, here's here's like our target budgets for for hotels, here's like our target budgets for some other things.

And I I brought that to the San Diego Tourism Authority, and they're like, this is perfect.

These whole hotels of this area, like you can't even talk to them.

Uh, you know, they helped me so much because I would have spent so much time talking to the wrong hotels that are just you know double the budget and stuff like that.

So yeah, it helped a lot.

I ended up writing the bid.

We I think there were a couple other bids, but we got accepted, and I'm sure it's probably more expensive than some of the other places they've hosted, but you know, it's certainly not Bay Area prices or anything like that.

No, for sure.

Will Vincent

I mean, San Diego's a lovely place to visit.

I mean, I'm often I've only recently come to appreciate all the work that DEFNA and you, Jeff Triplett and others there do to organize these conferences because I mean DjangoCon Europe just happened and it's so much work to do a conference, and it's um lot you know, very much in the you know, not not publicly seen.

So it's it's interesting just to hear about what it takes to

put that on.

David Fischer

Yeah, the the bid is relatively minor compared to like the operations and all the other things.

So really like Jeff Triplet and and I don't want to just call that one person, you know, that whole team is really you know, they they they're what make Django Khan a success.

DjangoCon US anyway.

Um, so I it my my contribution is like so minor by comparison.

But yeah, so San Diego Python, getting back to that, yeah, uh, you know, I'm I'm sort of one of the co-organizers.

I sort of took over as probably like the main organizer in maybe 2012.

The person who was the organizer before moved to the Bay Area, which is a common problem in San Diego for for organizers.

You know, they just sort of like make a name for themselves or you know, take the next step in their career, and the logical next step is move to the Bay Area, make way more money.

Um, so I I was the person who I'm I'm never leaving San Diego.

I love it here, it's great.

So I was I was sort of the person who ran it by default because I was not moving away.

Uh, but now we have this great team, you know, like that that's one of the big one of my big successes in my opinion, is that now there's a bunch of other organizers and when I need a month off or like something else like that, Django or uh San Diego Python continues without me.

My my my wife and I had a daughter four years ago, and I sort of didn't do any San Diego Python stuff for a year, and it worked.

Like people still went and there were still talks, and you know, that's sort of a that's sort of the other the other organizers really made that happen.

So I'd say that's my biggest success is that the group will continue when I'm gone.

Will Vincent

Well, I mean, in any organization.

I mean he here in um Boston there's there's a Django Boston group, which I think is the third largest by members.

Um after

David Fischer

we modeled ours after that one.

Will Vincent

Oh, okay.

Yeah, yeah.

So it's um uh uh John Baldwin and I'm I'm sorry, I'm blanking on the other.

Um, but it's you know, it's it's one or two people who do, you know, all the work and it's quite a lot of work.

Um but it's such a great uh attribute for the community.

Um and even I guess even as big as Boston is, which is a pretty big developer community, you know, we've had we used to have eighty, you know, fifty to eighty people.

Um the Python meetings of course are like three, four hundred.

But um it's a great

David Fischer

that's way bigger than us, but yeah we're doing okay.

Will Vincent

Well that but I would say, you know, again, uh mindful of time, but you know, in Boston web is not that big a thing.

It's so much more data science and or even hardware stuff.

There's not as much web stuff, whereas when I was in the Bay Area, it was very much, you know, consumer web kind of things.

There's much more of a you know, I don't know, PhD level programming, and the web is sort of this thing you sprinkle on top to just deploy it to customers.

David Fischer

Sure.

Will Vincent

Carlton, what's the what's the scene on the coast of Spain?

Carlton Gibson

Well, where I am, there's there's me.

Um there's a few little local web dev shops.

Um there's not much here.

They're down in Barcelona, there's a good amount.

There's a uh Python Barcelona meetup there, which uh you know is good.

I'm really bad because I've got so many children, I just don't go.

I'm like Yeah, no, I could I could take out this massive chunk of my life to go down to Barcelona and then hang out till quite late at night and then drive home.

It'd be totally uh I can't do that, so I don't go down very often.

Um but they're a great crowd, and the you know, it's active and they still keep going out.

I think they've got a an an online round table tomorrow discussing, you know, the changes in the the remote working and all the rest, because they brought in new laws in ch in Spain yet like yesterday about remote working and how that's gonna affect everything.

So it's you know, it's a really active community.

I mean Barcelona's yeah, it's it's quite a good tech scene there.

Will Vincent

I would love to ask you more questions, but I think we're we're close to time.

Um we're gonna have links to the the code.

I definitely recommend people take a look at the ethical ad server, especially if you look at the models.py file.

That alone is one of the rare readable production level code snippets I've seen.

Um well commented and everything else.

So I definitely recommend that.

Um ethical ads.io, right?

That's the site if people want to have ethical ads on their site.

Yeah.

Will Vincent

Um anything else as we head out you want to promote or shout out?

David Fischer

Uh, you know, no, just I I think San Diego Python is sort of like that is one of my biggest successes, I feel.

So I'm I'm I'm happy that we talked about that.

I'm happy we were able to fit that in.

Read the docs is fantastic.

I love them too.

And and you know, I'm really happy.

I mostly at Read the Docs work on the advertising side.

So um, you know, I I work sometimes on like the security and privacy stuff on read the docs as well.

Uh, probably my big six my biggest success there, and maybe I'll shout out to Cloudflare.

Thank you for that.

But like all the stuff where we now have HTTPS on the thousands of custom domains for docs on ReadThe Docs, that's all like courtesy of Cloudflare.

That would cost us like thousands of dollars a month.

It would probably that alone would double our infrastructure budget if we were paying like retail price for that.

So I'll thank them for that.

Will Vincent

I mean, I use Cloudflare.

I feel like there should be a whole course on Django plus Cloudflare since it's so powerful.

But yeah.

David Fischer

Read the docs is actually doing some really cool stuff there, but maybe there's maybe there's not time for that.

They have this, we have this whole thing where like when there's a new docs build, we're we're purging from the cache just those docs and all those sorts of things.

You know, when we rolled that out, like docs, especially if you were browsing them in like Asia or Europe, they got so much faster, especially Asia.

Because it's far from our main data center.

Will Vincent

Well, maybe maybe we'll have to have you on again.

I mean, yeah, just before this, I was manually purging some pages on Learn Django that I updated, and I'm like, yeah, I know I need to automate this, but I just can't be bothered.

Yeah.

Will Vincent

So I can't even imagine at the scale that you you all are at.

David Fischer

Oh yeah, and yeah, I I won't go into it because I know we're we're at time.

But yeah, it's super cool.

I didn't work on that, so I I I don't want to take credit for that.

That was mostly Eric Culture.

Carlton Gibson

But the the long and short of it is you don't go through the ad through the UI though clicking purge, perch, purge.

Absolutely not.

Will Vincent

Copy paste each URL.

David Fischer

Oh, absolutely not.

No.

Will Vincent

Well, thank you so much for coming on, David.

Um, we'll have links to everything in the show notes.

Um, really appreciate it.

I've been and glad uh Jeff Jeff connected us.

Yeah, Jeff Triplett, because um I was saying uh I was complaining about ads, and he was like, You should use ethical ads, and I was like, what's that?

And then he's like, and there's someone there you can email.

I was like, oh, okay.

So here we are.

David Fischer

Yeah.

Advertising, it's a crazy business.

Lots of lots of bad stuff, but you know, we're trying to do we're trying to do good there as much as one can.

Will Vincent

Uh so as ever, we are at DjangoChat.com, chat Django on Twitter, and we'll see you all uh the next episode.

Bye bye.

Join us next time.

Carlton Gibson

Bye bye.

Thank you.