Transcript: Django Security - Markus Holtermann

Hi, and welcome to another episode of Django Chat, a weekly podcast on the Django web framework.

I'm Carlton Gibson, joined as ever by Will Vincent.

Hello, Will.

Hi, Carlton.

Hi, Will.

And with us this week, we've got Marcus Holderman, who's a member of the Django Technical Board

and on the Django security team and on the Django Ops team.

He's just a stalwart of the community.

Hello, Marcus.

Hi, Carlton.

i will hi everybody hello thanks for coming on marcus so i guess look let's kick off look what

how how perhaps you could tell us a little bit about yourself marcus like how did you get into

program how did you find django you know how long you've been with the community these kind of you

know i've been writing code for oh two decades now i'm not even sure it's been it's been quite a

well um and then got into django um 2010 ish um as part of the german ubuntu users.de community

that's a django project that started even before the days of django 1.0 or something it's like

9 0.90 or something like the very very early stages of django that's when that project was

started and it's a support forum for the german-speaking ubuntu community and yeah features

a bulletin board and news articles and planets and all these kinds of things is that still going

it's still going it's still maintained by a few folks on all it's unfortunately not open source

it's a kind of a thing that we since ever talk about yes we want to make this open source

But because it has so much legacy, it's also one of the things where there's a lot of technical depth in there that the folks who work on it these days want to get out of it before they actually open source it.

Were you going to say cruft?

Is that the word that was coming to mind?

Yeah, probably.

Maybe that's the kinder of the words that start with CR.

Yes, exactly.

but that's it that's that's just any old code base right any or like so that's that's what um

10 years old 15 years old coming you know that's an old code base yes it's i think they started

writing that and i wasn't on the original team but they started writing that 2006 i believe

i believe and it's one of the it's a project where then as a side effect like workzeug flask ginger

two other projects

came out of.

It's kind of in the

early stages of the Python

web part, but I guess

the Django-ish

style web

development part. Yeah, that's kind of cool.

And that got you into Django?

And that eventually got me into Django.

And yeah, there's

Apollo 13 is on that team as

well. Back in the day

was under the theme as well and then he

eventually kind of like always

pushed me a bit to maybe contribute to

Django because at that point he was a

core contributor as well

and yeah that eventually

got me into

contributing to Django after

Andrew Godwin

merged his migration patch

in 2014

I believe, 13, 14, 14

1.7 right? 1.7

exactly and

yeah then

our contributed

bug fixes there

every other day or so

because you're

like so when tickets come in

on the migrations I always wince a little bit

because they're quite hard and I'm always

quite glad if Marcus comments or you know

a couple of other people but Marcus is one of my core

you know

go-to people on the migrations framework

yeah maybe still

I unfortunately

didn't have the time for the last few years to actually

do a lot of things on the migration

framework.

But it seems

there's a bunch of other contributors these days

that actually have picked up

behind me and behind Andrew

and others that contributed on a

regular basis and, like, squash

all these bugs these days.

Yeah. So,

go on, Will.

I was going to say, I'm going to put a link

to all the various teams on the

Django Project

um site because i think you have the record for being on like all of them right so there's and

again and this is also in flux you're on the ops team you're i guess you're not technically a

releaser you're on the security team you're on the technical board though not the advisory board

so i guess you're and then you're on the technical team so you're missing two but all the other

boards and then that all is in kind of flux because now there's the new yeah technical so

perhaps um what is the future going to look like in terms of the organization right because like

the term django core even yes tossed around what do you make of all this for a casual listener so

how should they understand so about a month ago or something beginning first quarter first half

of march or something um we passed the django enhancement proposal number 10 which essentially

states that we want to

change how Django

is being developed or

how the Django community

works and how

decisions are made and

who has

I don't want to say the right to do something

but who can

eventually if there's a

discussion that doesn't come to a solution

where people can't agree on

one of two things

who can make the eventual

or final decision to go

okay let's go with this way

or another one

and

yeah I guess that's

one of the things that's

been around where we've

been trying to do that for quite a while

but never really got

around that now

James Bennett finally managed to

actually write down the step

and

well at that point the Django

Core team

voted on that

and it was accepted

with quite a large number

of votes

and yeah so then the

future of the Django project is gonna

look different

I guess

we're trying to open up the

decision making processes

and make those more

visible

yeah because the other piece was

the so DepTan

And then the core voted on it.

And then the Django Software Foundation board, which I'm on, most recent meeting, we approved it.

So going forward, I think the idea is that there'll be the technical board and then the one I'm on, which is sort of the non-technical things as kind of the arbiters of disputes or decisions that arise.

Because the next step is an election that Frank Wiles, who's the president, is going to be configuring, setting up the election.

of technical board members yes um frankly i'm not entirely sure about the exact process

from the uh from the top of my head because it's a fairly lengthy document but yeah we need to yeah

well i mean i'm i'm privy to uh it is a long document um and james has spent a ton of work

on it basically the reason why uh frank is going to take that on is because we don't want someone

on the technical board to be in charge of the forum for voting but it's just going to be a

google form with the proper permissions yeah and seems like that will be a good idea happening

so but i think part but part of that is as you were saying is public versus private discussion

right because that's sort of a big thing about so what are some examples of why that matters

for the django community i think it's it's django is an open source project and we should

be able to make decisions in a public forum and and make it visible to anybody who wants to

contribute to understand the reasoning of how the people who made a decision came to that decision

and seems to be just in the nature of how to do those things and that making those decisions in

public seems to be the right choice i think the other part as well is that the term django core

is something that you get and you never lose even if you're not as active i mean because for example

you're incredibly active and there are members who have the title who are not as active so i

believe one of the intents is to i think that's going to be a legacy thing that can be bestowed

but but to have the technical team reflect the current active members as opposed to people who

were maybe more active in the past well the the um the term django core or rather django

commit or django core team member has been or is a term for somebody who had been given commit

access in the past and it was always kind of a yeah once you have it you don't lose it unless

you mess up like big time but i don't think anybody ever did or you hand it or give it up

by yourself um but then these even yeah as you said if you then don't actively participate in

the whole development of that then you still seem to be on this list of people who aren't in there

and you can still take credit for that position.

And sure, when you have contributed in the past

a significant amount of work,

and that was usually the thing

that kind of led somebody to become a core member,

then using that position

or using that title that you had in the past

kind of made sense and seems fine to me.

But over the last years,

it's been clear that there's not really this big thing in Django

that somebody could do to become a core committer or core member.

And the things that in the past indicated somebody becoming commit access

are just not really there anymore.

So finding a way to become a core committer

wasn't just something that still existed.

There wasn't a route to become one.

So, dropping this whole concept and acknowledging people who had this position in the past, but then essentially dropping this whole position seemed kind of like a better or good way forward.

Well, part of that is that, because the term core was around before there were Django Fellows, because Tim Graham and now Carlton and Marius, in terms of releases, at least, they do much of that work for the community.

Yes.

So that's part of the...

Exactly.

So in the past, with commit access, you were able to commit to the GitHub repository directly.

And some people, a small number, I think like three of three people,

had release access, which meant they had the possibility

to put packages in PyPI.

And with the new change of the DEP-10,

this also changed in a way that you don't need to be,

We will have a merger role, which is the people who have commit access to the Django repository.

And we have a releaser role, which are the people who have access to PyPI and can put packages on PyPI.

People can be in both, but they don't have to be.

So if you, for example, don't have the time to constantly review or time to merge patches,

and specifically review is not necessarily up to the mergers they should yeah they should be doing

like a senate like a check on style and and so on and do a like general review on the pr that

it makes that whatever is committed or whatever they commit makes sense but um the discussion

if the approach is taken in a pr should be taken by the community and by people who are going to

be involved in this in this change yeah because you could imagine like you know a number of people

reviewing the pr and then the merger is just the person who clicks the button at the end it's it's

kind of the yeah you know i always describe the fellow role as like the janitors and that's kind

of what we do is we just you know just keep it tidy i mean we do do reviewing and all that yeah

i mean you do more than just being the janitor um you do far more than just being the janitor

for the Django project but yeah it seems to be a significant role to to what you're doing well

and I think part of this in general is the reason why that like for me on the on the board and you

and the technical board why we spend time on this is because it's an open source project basically

everyone's a volunteer and we wanted the system to be in place so people can contribute but not

one person has access to everything for example and so when people do leave there's a process as

you said for new people to come in to be a handoff of of knowledge and power i mean even for example

so we added github sponsors to uh to the jenga repo and that's something i worked a lot on

so i have access to part of the jenga repo but i couldn't fill out the form so i had to get

marius who's the fellow because he has you know more access so these layers of permission

are important just in case yeah you know you don't want someone to have the keys to everything

and you want to have a bit of structure, but not so much that it's prohibitive to get anything done.

is how i think about this is i mean the things particularly with access to like pi pi you know

that's that's sensitive right you you can't you can't have a package being uploaded as django to

pi pi if it isn't django yes there's too many people depend on it being django for for that

to happen exactly there's opening up the whole thing um opening up the whole contribution process

to Django and allowing more or less arbitrary people to be voted on

to be mergers or releases comes with a side note that with a perspective

and me taking this with a perspective of a more security conscious person

right now, comes with the problem that we need to trust that person

to not put in some backdoor or whatever in that release package

that somebody needs to inspect and find.

And so, yeah, the selection process or the voting process

on who becomes them will be open according to the things

that are stated in the DAP.

But it's definitely also going to be something where the community

needs to look out for that not some random malicious person is going to like hijack django

because that's not going to be a good uh but i was i was reading the the depth the yesterday or

day before some for some reason i needed to refresh my memory on it but um the the the

qualification to be on so the technical board have to um uh okay the nomination of mergers and

releases for these important roles but to be a candidate for the technical board so the technical

board will be elected by the dsf members so already to be in the dsf you have to be known

by the community right anyone can join but you have to be known by the community well you can

you have to you have to be nominated and then approved by the yeah but then to be a candidate

for the i think you can self-nominate but there's and actually i'm linking i wrote a blog post just

recently about what the board actually does there's i think 180 or so individual members

so sorry continue carlton but that's like the first step of yeah but like the to be a tech so

but not anyone can be on the technical board you have to have a show like a recent like within the

last couple of years a shown um history of contributing and reviewing prs or working on

track or the you know github or being being in the group so to to be a candidate for the technical

board is actually quite tough it's it's not like any any person who isn't known to the community

and known to be a contributor could could even stand for the technical board i think so i think

the term the wording in the depth is is actually very um conscious of the debt of the possible

dangers if it were more open so i'm quite relaxed about it in that yeah it definitely

james definitely put a lot of thought into that part and and the other people who contributed in

the in the writing of the dab i mean james did a major part of that but as again the discussion

on the depth was uh held in public on the on the depth itself on the pr and it was over a year in

discussion right it was it took a long time i don't even know it's taking quite a while

but that's cool so maybe we can talk about some of these hats that you wear because i i hope that

there's more of an understanding of the you know the many many roles that you you play in the

community so let's pick security team so you and carlton are both in the security team and i think

what's the process for a normal security release and then there was that the github sql injection

injection one recently which i believe you did the commit for that you you all turned around in two

days a month or two ago um right right so that it was first discovered on github so what's the

normal process and then i thought that that one was particularly impressive where maybe it was

three days but the it was discovered the commit was done and it was released yeah and so the usual

i thought that was cool thank you um um the usual process for security thing is usually that

well somebody reports either on hacker one on a security mailing list or security reporting list

security at jangleproject.com

reports a security issue

or a suspected security issue

rather be conscious and

think this might be an issue

report it and then we go actually that's not

but we'd much rather have

that than somebody just

putting something into the open

like oh shit this is something

that we should have

fixed now

so you usually report something in a private way

and then people on the security team

are going to review that

um is it an issue is it like do we need to fix it like now now or do we have time to to rethink the

whole approach we have there um we then usually talk to the reporter and try to figure out who's

going to write the patch who's going to review the patch like we try we love for reporters to

also contribute to patch and then the people on the security team reviewing it but if they feel

like yeah now it's i don't really feel like writing the patch then somebody on the team

is going to write the pageant we then hand out the patch or the proposed patch to the reporter

and ask them to apply that and see that this actually fixes their bug in their well probably

production environment that they have there um this maybe goes back and forth once we are done

with that, we pre-notify a set of organizations, companies, of an upcoming security release.

We also pre-notify on the Django announce mailing list that there will be a security

release with a given date and time.

And then at that point, well, usually Carlton or Marius are going to commit the patches

to the Django repository and issue the corresponding releases

and announce the release.

Now, yeah, go ahead.

I was just going to say about that commit,

because it's like potentially sensitive,

and on that morning of the release, and so we've got a time,

and it's like we've got the patches ready,

and they say we're going to have to, you know,

we just dropped 1.11 from extended support,

but at the time we were supporting Django 1.11, Django 2.2,

Django 3.0 and you have to merge those patches onto onto the branches and then within a short

period get the release out and the announcement out because once you've merged them those patches

are public and so that it's kind of like you have to get all the pieces lined up and it's a little

bit tense you know it's it's much harder than a normal release which you can you know you take

your time on yeah carry on Marcus you were you were saying so we've yeah this is um I can

definitely understand that like issuing a security release is a whole different story just from the

mental uh capacity that's involved there because you can't like a usual release you can go like

oh yeah we broke whatever in there we're just gonna it just don't install the new version

like pin something to before and then like take the time to to fix this bug and or revert it and

issue another one but if that happens with a security release you go like oh damn well we

just published a security issue which we don't have a working patched version for and you go

like this is not ideal um so the the other issue that you brought up with the um with this with a

github uh sql engine and it wasn't i think it was like this it was an account hijack it was an

Potential account hijack.

Yeah, it was like the password reset.

Yeah, that's what it was.

Password reset.

Yeah, something like that.

It was like a Unicode character or something, too.

You could shove something in there.

Yes.

You guys actually worked on it.

I'm just sitting here, like, talking over you.

You did the work.

The issue was that if you get a domain with Unicode characters in,

which sort of lowercase comparisons to, you know,

with certain database backends to another domain.

So, you know, it could be google.com,

but the O's are funny Unicode characters.

And then it would look like it was a Google email,

but in fact, it wasn't a Google email.

Yeah, in that case, if the issue itself is known,

then you don't really need to treat it

as a sensitive or private issue anymore

because the moment those security issues are known

and possibly exploits are known,

a bunch of smart people are going to look through

all the court bases out there

and trying to find projects that's occurred

bases that are vulnerable and

at that point this time between

announcement and pre-notification

is just

well kind of wasted time

because we should

instead have that time

people already fixing

their installations. Yes normally

we give seven days but you can't

give seven days when there's a

exploit out in the world.

So yeah

in this case we just dropped the pre-notification

notification we issued the release on may pretty notified by a day yeah something like that i was

just going to say for carlton and i often beat the drum of keep your django version up to date

and even if you're using an lts you should do the minor releases the security releases for this

reason because otherwise you won't get yeah the security updates yeah if you're if you're on a

version that's in the extended support then it's definitely a you know if there's a point release

you definitely want that like yes there is yeah because it's either data loss or it's security

you want that yeah absolutely let's go down the list ops team so you and i have had some

i've been emailing around that maybe we don't need to discuss some of that but what does the

ops team do like what is the ops infrastructure of django itself so you're on there along with

uh florian tobias and um carlton um well we have this website and you probably have heard of it

django-project.com

and that needs to run somewhere

and well we also have

a CI

continuous integration that we use for

testing Django which is

run on django-ci.com

and

hands out when you run

websites you need some servers or something in the

backend that does work

and

yeah we usually kind of

maintain

the software or

Not necessarily software there, but the infrastructure.

We keep Jenkins up to date, usually.

When there's fixes to the Django website, we deploy them.

These kinds of things.

Yeah, well, it's, I think, another sort of unsung role in the broader community,

but quite a bit of work is done.

And so for me on the board, as the treasurer,

what's relevant is there's some costs involved with that.

so i monitor and help pay those rack space has been sort of the major contributor the last few

years who provides the servers they've been very generous with that so just another one of these

things that's happening in the background that makes django happen and um you know you marcus

are very involved with so i want to call that out thank you um i have a long list of things to ask

you about but maybe let's talk about your day job right you work at crate io which is iot scale

database can you tell us what all that means yeah yeah yeah iot scale database explain that that's

not you that's that's the headline on the website yeah uh it's the headline on the website which is

fine um so we built a database for the iot market um so iot is usually time time series database

time series data so something numbers usually or maybe some added labels to that and then a

timestamp and um well you want to like aggregate data from usually or in the the market we are

focusing on is the industrial iot so you have factories that companies have where they equip

their production lines with sensors or the machines in themselves are already equipped

and you want to do analytics on on those machines and you want to understand when something goes

wrong and like this whole in order to do analytics you need to store the data somewhere that and then

aggregate things and turns out when you have a lot of plants or manufacturing lines and machines

and a lot of data and a lot of numbers then you need to have a database that can cope with that

And yeah, with CreateDB, we have a database that's based on Lucene, scales fairly well, like pretty much almost linearly for I don't even know how many nodes, and can cope with ridiculous numbers of records and tables.

And yeah, it's a SQL database after all.

So you still write your SQL statements and it returns the records.

So why not Postgres?

Right.

Obviously there's a lot that Postgres can't do, which is why you work there and you have

this product.

But what is, where does, if I start a Django project that's IoT and I'm using Postgres,

where do I see that, oh, I need something better or bigger.

create like do you have a sense of where that friction comes up so one of the things with

postgres is that it's transactional for example like i'm not even sure if you can run postgres

without transactions i suppose you can't i guess you can't um you probably also don't want to

that's it um create db on the other hand is um eventual consistent and you have

kind of multi-master

you don't really have a master node

in your cluster

you have a single node that holds

at the particular set of time

at the current time holds

an information about the whole cluster

but that's also known to

every other node

but you can write to every node in your cluster

connect to every node

write to every node, read from every node

and it's

more or less self-balancing with the data

if you don't screw up your table creation.

And is the advantage of that, is that ingest speed?

You can pull in the rate at which you can ingest data

is just an order of magnitude higher.

It's orders of magnitude higher.

I don't even know the exact numbers

that we are currently looking at with some of our benchmarkings.

But it's, sure, you can scale your Postgres

ridiculously high

and improve a whole bunch of inserts

but at some point it's going to

you're going to run

into issues and then

for the most part in Crate you

just add two more

nodes to your cluster

the just in quotes because it's

like just in IT is never going to be

a you can simply again

in quotes

but it's like tutorials I always

try to put simply just simply

install jing exactly it goes over well um but yeah for the most part it's you add a node or two or

five and your cluster has more storage space your cluster has more throughput um aggregation

capabilities memory all these things um and so the other question you said eventually consistency so

how long does that take because i've used eventual eventually consistent data stores and quite often

that just means where's my data you do a write and then you do a get to get it back and it's like

not found and you're like hang on i just inserted it and you said it worked where is it um so i mean

how long does that take to run through say if you're you know i mean i guess it depends on the

size of your class you know but as i think it's sub milliseconds or a millisecond area like you're

not gonna wait five minutes for this number to show up somewhere i mean if you're if you're

complete if your entire network is completely overloaded for sure it's going to take time but

then you may want to reconsider your network infrastructure um spin up that extra node you

talked about yeah though so adding more nodes also means more communication right and you so the you

said it was based on lucene which means it well the thing that comes to mind is well elastic search

how does it compare to elastic search because i know a lot of people use that for high ingest

use case yes in the very early stages of creative it was merely a plugin on top of elastic search

that provided the sql interface which over time became a as well not a fork of elastic but

had a bunch of code from elastic that was um used and then we we had our own stuff around that or

melt it into the whole thing

and that made up

CreateDB at that point and then eventually

last year, I guess

I think we dropped the core team

there, dropped Elastic

in that sense that

it's not in the code base anymore

so it's

I'm not involved in the

direct development of the database itself

so I can't

actually tell you that much about the details of the development there and the specifics around

the code um but are you aware of the sort of general like what was it that elastic didn't

scale in a particular way or do you know do you happen to know that maybe you don't i don't know

it's just interesting i'm not entirely sure to be honest okay fair enough fair enough it was just a

you know well and you i think your your master's studies you you did some work on elastic search

right so that's back in 2014 along with alex uh no no oh whoops sorry i got the wrong i was

looking up ah i pulled up wait oh weird i'm sorry i was just quickly pulling up um wait it's on your

i'm on my masters without alexander griebert because i i saw you had a talk on elastic

search and i was just trying to pull it up and it pulls up i did from 2014 you have a whole post on

it so i saw you oh i saw you give a very good talking um on elastic search it was just one

course at uni um where we so so all my all my studies i always with all the projects that i

had to do i always tried to use yet another technology just to like play around with it

and went from C, C++, Java, eventually Python.

And then in the Masters, in that one course,

in this one blog post, I guess you're referring to,

with the school data in Berlin,

we figured, hey, how about we just build a static website

and then throw all the data we have into Elastic

and just query that straight from the front end.

But that's about it, what I did with Elastic there.

The thing I think Hugh Carton referred to was a talk I gave on Heidelberg.

On Heidelberg, yes, probably.

Like to search, no, what was it?

Well, let me summarize what I remember.

I mean, I've got it here.

It's combining Django and Elasticsearch.

You have the slides up, the link to it.

Yeah, so it was this great talk about how you can use the Django ORM

and write that and then how you would sync to Elasticsearch

do your information retrieval your search queries and i remember it being so good because i at the

time i was flirting with this idea that oh we'll just drop the orm if we're going to sync everything

to elastic search and search from elastic search and even we're going to serve the web pages perhaps

from elastic search why not just write to elastic search and not bother with the django orm at all

and you were like no no because you lose the transaction handling you lose the orm you lose

all these and you're exactly right and i was like yeah that's that's that's answered that question

canonically for me um yeah the talk i i looked up the talk it's um on the lookout for your data

from jango con europe in heidelberg you're right there's 2018 but that's a great video folks you

must watch that on youtube yeah link in the show what the um the other talk project that i wanted

that i was saw of yours recently was the the logging article where you you know you talk

through django and and logging at a really deep level but then you just casually link to this

gitlab repo that's amazing where you have all these examples of how to structure

your logs in different applications which we'll we'll link to like you know because i i put a

code open source sometimes and you know you've got a lot of good stuff in here right you have

an airline example a bank example that's and you just sort of casually mention at the end like you

know i would just like make a video of that being like check this out it's a lot of work to make

that um it's been a top this this so the topic there in this talk is about structured logging

and yeah structured i could recite parts of the talk now but i'm not going to do that i'll i

suppose you can put the link to the talk somewhere in the description um or in the links um i i feel

i'm gonna spoil the surprise of this talk but um okay well i just i was really impressed that you

actually have code to back it up which was which i always like like it's not just talking it's like

you gave examples yeah yes the um the parts you mentioned there with airline and so on it's the

reason for that is because of the title and the the whole story of how i try to bring this whole

title of the talk into

why you would want to do structured logging

and with that refers to

a movie. Okay, we'll put the

link up for everyone so we won't

ruin it.

Or maybe we could talk about Django

3.1. The two things I'm excited

about is the async view. So OK 3.0 had

the ASCII handler which enabled

you to embed Django in a

ASCII environment

so that's the foundation step

but 3.1 is the starting where

you can actually write an async def view and okay that's going to be basic and that's going to that's

going to be fledgling as what we'll be able to do in future but it's like yes so now if i've got my

janga project i need to make a few api calls out and they need they could be done in parallel

without even running a different server still with gunicorn still in the whiskey environment

i can write a parallelized view which fetches those rather than having to do those sequentially

and that will be much quicker and that's that's just a quick gain without having to do anything

else so you know this is the real start for yeah we can start to do these extra exciting things and

okay we'll there aren't going to be the async class-based views that yet that we'll probably

get in a few you know when we all learn how to write these views and we all learn what the

patterns are django support will evolve but this is like the for me this is like yeah actually this

is the real async bit this is the bit where we'd like yeah actually most of what you can do and

And then one of the next versions,

we need this three-point async for database level, I guess,

because that's still not there, right?

Yeah, I mean, so yeah, the limitation is if you're still using the ORM,

that still needs to be wrapped in a sync wrapper and executed in a thread.

So a lot of the things you want to do, there'll still be sync.

Yeah.

But, I mean, Andrew Godwin, who wrote the Vue's PR,

we merged it, and he's quickly joking,

oh, right, now the ORM.

It's like, oh, my God.

Right, exactly.

It's going to be exciting, I think.

I mean, I'm excited.

I've been writing for years.

I had to spin out a Node.js on the side

before Python had Async.

I'd have to spin up a JavaScript thing

to do something Async.

and then python's had async for a little while with async io and you know there are other options

but async io being the standard libby option and you think okay well i can just do it with async io

fine brilliant and there are a couple of web frameworks around that and um that we've talked

about tom christie style it as as a go-to there but to not even have to change your web framework

to do the basic thing that's yeah it's more and more civilized all the time that's pretty cool

absolutely and i think that because flask is working on this but it's sort of like totally

different things too so not that it's a competition but i hope to get i guess david lord or someone on

from flask to talk about how they're doing it but it's it's gonna be another reason that django just

marches into the future and you know when people say async yeah you can you can do that though of

course like i mean in andrew's talks to me the interesting thing is the use cases for async

aren't as ubiquitous as maybe we thought they would or the community thought they would be

five years ago yeah um like that to me is sort of the interesting piece i mean you know in the

same way that we have http2 but it's not aside from gaming and chat maybe well perhaps and you

know in in iot cases it's being used but it's not used everywhere the the interesting part about

the part about async is that it's really giving you a lot of benefit and and when you have a lot

of io and frankly most websites and yeah should chat and and all these things they are not

particularly normal websites in that sense because there's a whole part of this media part in there

But regular websites, it's, well, here's my view that fetches a few things from the database and returns a bunch of HTML for the most part, or returns a bunch of stuff in a JSON response as an API.

There's not a lot of, not really a lot of IO happening when you compare it to a bunch of other scenarios.

and yeah you may get a bit of speed up here and there but the occasionally i think that the

mental overhead that you need to apply there in order to wrap your head around async can easily

reduce the maintainability of the of that page then yeah um comes with pros and cons

it's pretty much everything yeah in the pr there was a note andrew put i'm sure i'm sure

it's there. It may not. It must be. It must be. Otherwise, I wouldn't be sure. But there

was a note saying something like, we advise you to keep using sync until you've profiled

and you know that there's a real benefit here. Don't just jump onto the async bandwagon because

you think it's cool. I mean, it is cool, but that's not a reason to build your product

in it.

Yeah, absolutely. It's definitely an interesting approach and interesting way of writing code.

And I mean, the whole front end part in web has approached it for a while now and has been doing as in for quite a while.

But then in the front end, you have direct user interactions where you want to do bubbles here and show things there asynchronously, which you don't really have in a web server back end with like APIs for the most part anyway.

yeah i mean there's there's some um was it live call it live view so there was a the phoenix

project which is in um elixir elixir yeah yeah they had a they've got a thing called live view

and then somebody's ported that to django there's a django live view project which is really

interesting i think they i think they're using channels under the back but what's nice about it

is that um the the front the say a form can do um validation on the server side because it's doing

what looks like local client-side JavaScript validation in the form,

but it's actually pinging the server and getting the response there.

And one of the issues with writing decent JavaScript

has always been, oh, I've got to rewrite my validation logic.

And, you know, it's always been a bit of a pain.

There's the Batavia project from the PyBee project,

which aims at providing Python in the browser

through cross...

I was going to say cross-compiling to JavaScript.

I'm not entirely sure.

I don't entirely remember how they built the whole thing.

But essentially can write Python code

and then they have a Python...

Yeah, what they have is essentially a Python runtime.

So no compiler or something or REPL.

It's just the interpreter that you throw some Python bytecode at.

And the interpreter is written in JavaScript, so it runs in the browser.

And then you run this Python bytecode in the browser.

Yeah, that's kind of cool.

So it's part of the BWare project, right?

Part of BWare, yes.

Yeah, I was going to say, yeah, Russell's, yeah.

I think he referred to that in his keynote at PyCon last year, too.

Probably, yes.

That specific, yeah.

Well, maybe for me, a final question. So this is something I was hoping to do in person at

DjangoCon Europe, but since we're in a virtual situation. So sitting on the board, a lot of what

we do is the janitorial side of Django, paying the bills. But one of the things I want to do is be a

little bit more aggressive around if we had a grant fellowship, if we didn't have to rely on

Kickstarters for major new features, what would be some features that you think would be good for

Django to add, assuming we had unlimited

people and money? Are there a couple

that come to mind between the two of you?

I mean, authentication is a big

one, right? So I guess

ContribAuth could

use an overhaul,

let's call it that, make it a bit more

modern and adjust it to a few

modern security or

authentication processes.

Stuff like

OAuth or

TwoFactor in

in whichever way authentication um i think the thing that's been on the docket for ever

is a new admin there's this well right it's just a million dollars right they did the cost

i don't even remember who who brought this number up i have the feeling this was i think it was

jacob jacob at django under the hood i think um yeah if if if we have a million dollar a million

dollar and a few folks i think you mentioned 10 to 10 people or so that's that spent a year then

this is achievable or something like that um yeah well i just it's i i like the big one on my list

is the auth i'd like to see two-factor auth built in i'd like an easy solution for users to be able

to you know have one-time passwords have you know yeah all the other bits and bobs that come um

other things that's been go ahead no well that's that that's my one sort of battery that like there

are other things that we could add like nice little features that would improve it in a number

of ways but my one big missing battery at this point is to fa of some kind of modern yeah you

know um use your use your smartphone with its bio bio registering security measures and know that

it's you that's logging it.

I think we could do that now.

I mean, it's a big job, but...

It's another thing that's been floating around

in a fairly long time,

and I think kind of was dropped to a degree

once the whole channels,

once Django channels came around,

and absolutely once it was clear that Django itself

is going to have some async capabilities,

is an API for task execution.

So not the execution built into Django,

but an API where you're going like,

I want to send off this task and do a thing,

and then whatever implementation project

or project is out there that wants to implement that

can use and can build on top of that.

So just to pick one,

Celery Django integration would leverage that task

or this task interface that Django could, anyway,

could leverage that task interface that Django uses

and you define whatever this one setting in Django or in Celery

and it, again, quote-unquote, just works.

Yeah, well, there's the rub.

But the idea is that you could replace Celery with whatever, RQ or I don't even know what else is out there.

But you might have an SQS backend, you might have a Redis backend, you might have a, you know.

Well, that might not even be the level of where the integration goes, but it's been floating around like five years ago or something.

I remember having a discussion with that in Cardiff at the DjangoCon.

um so that that was 2015 right well i hope that this is something that i hope to encourage more

in the community because on the dsf side and treasurer if someone said we need this new feature

raise x amount of money there's a number of ways we can get that money um so i don't want that to

be the limiting factor uh i hope to encourage that you know again we don't rely on kickstarters

but it's sort of it's sort of the chicken and egg where i mean because the total budget for

the django software foundation is around 200 000 most of it goes to the fellows then to conference

workshop sponsorship and that's and then teeny tiny amount for the ops and and whatever and

that's it because we don't need more and we're not trying to raise a ton of money but if some

feature was 50 000 you know whether it's getting a grant from google which has provided help for

some of these corporations or you know there's a number of other ways we can um try to raise money

we can raise money in a way that doesn't go against any of the principles of django but

we need to know that we need it um is the the first step so that kind of brings us full circle

because one of the things i'm really excited about the debt 10 which is the change to django's

governance is that the technical board are meant to have a new role of helping to guide the

technical direction of django and to each major release to to sort of have a take a bit of time

solicit ideas compile those ideas into a bit more of a feature roadmap because we've got this time

based release and we'll keep that but the technical board will hopefully be able to lay down some um

look we're aiming to improve contrary both we're aiming to you know introduce a back-end system if

that's what the technical board come to the conclusion with consulting with the community

then we will have that um that list of ideas which are that list of things we want which then you

know the dsf board or the membership can act on yes absolutely um but that also um requires like

ideas from outside or from people using django or from people in the community or so to go like hey

this would be cool and then while we i guess for some obvious reasons we can't do all the things

and we probably shouldn't do all the things,

but the things that are useful for the majority of people

where there's a possibility to build an API in Django

or build the bits and pieces into Django

that then third-party applications can use

and leverage that to build a more sophisticated path

to be used as applications.

I guess that's definitely something where we definitely will need to rely on the community to bring up ideas there.

So one of the things I'd like to do this year is to do a Django survey again, which I think Tim Graham did in 2015, something like that, similar to the Python survey, where in addition to asking people how they use Django, we could have a place to say, you know, what would you like to see in part based on your use case at your company?

because we're sort of getting at is that a lot of the times we're flying blind in terms of Django usage

because we don't have tracking in there.

So we don't really know a company's using Django unless there's something open source

or someone from there tells us by design.

But that does limit sometimes the new features and things that we can add.

Yes, absolutely.

Another was a discussion around integrating some kind of usage tracking in Django, which was, I think, fairly strictly, let's call it shut down by the core team.

I think there's a dApp out there, like a dApp proposal, like four years old, I'm guessing here.

Yeah, I think it was it was shut down. I mean, because we can you can see on PyPI, you can see the number of Django downloads, but the PyPI stats are, there's many ways that those are unreliable, actually.

So I mean, I, I think our take to be conscious of security of privacy and all these things is the right one. But it does mean it takes a little more work to find out what, how people are using it. I mean, literally, I mean, Jeff Triplett and I are trying to, or Jeff Triplett's been working on a list of like the top 500 companies using Django.

And it's a bit of work to find that in part so we know who they are, that we can ask them around features.

We could, you know, try to suggest sponsorships and stuff.

But, like, we have no idea who they are just without doing the work, the manual work.

The DEP I was mentioning is DEP 8, Gathering Django Usage Analytics.

It's difficult.

I mean, commercial companies obviously often put this kind of analytics in their packages.

know you download bs code it's got analytics which goes back to microsoft um that's not necessarily

a problem you opt into all of that and you know about it but when perhaps the message is gonna

have is it like i think the i mean we could message i mean for me uh we could totally read

someone could we could look through django project in general from scratch with someone with fresh

eyes have a docs fellow or team i think there's a lot of ways that we could improve that and

talk about a lot of the things that Django project and team members are doing that we don't

spend the time talking about rather than just doing. But it's capacity as well. Like the,

you know, that there's a group of contributors who are very active and there's new people coming in

and, you know, some come and stay, some come for a bit and leave, some come for a bit, leave,

come back again. You know, everybody's working quite hard on keeping Django moving and it moves

fast you know look at the last few releases they've all been you know just major great releases

yeah and that's and i hope that and the project i want to call out more attention to the people

doing that work because i know that everyone who's on you know our mailing list knows who

one another is but most people don't know um and i not that people do it for that reason but i think

it's important to as to your talks carlton right we need new contributors there's a lot of strength

But there's quite a bit of fragility when it's a half dozen people doing, you know, two of you are on this call are doing a lot of the work.

Fortunately, it's a remote call.

It's a video call.

So if a meteor strikes one location.

But anyways, but I mean, certainly I view my, you know, my role on the board as being a janitorial, a support role for the court, you know, the technical team and trying to enable that.

So, but yeah, I mean, so, okay, that's a good point to wrap up.

So, Markus, thank you for your work on the ops team, on the security team, on the technical board, and everywhere you're around at DjangoCons and on the community as well.

Thanks, Will Carlton, for having me.

Thanks, everybody, for still contributing to Django and keeping this project running.

It's pretty impressive what's happened for the last 16 years now?

15.

15, 16, yeah, something like that, give or take.

yeah yeah wow it's quite a while it's gonna be a few more yet one things yes all right thanks

again all right and everybody we're at chat django on twitter uh django chat.com and links

to everything in the show notes okay join us next time bye