← Back to Show Notes

Transcript: GeoDjango - Anna Kiefer

Carlton Gibson

Hi, and welcome to another episode of Django Chat.

I'm Carlton Gibson and I'm joined as ever by Will Vincent.

Hello, Will.

Will Vincent

Hi, Carlton.

Carlton Gibson

And today we have Anna Keith with us.

And Anna, we're going to talk about um GeoDjango.

So how are you, Anna?

I'm doing well.

How are you both?

Carlton Gibson

I think we're marvelous, but thank you for coming on the show.

So Yeah, happy to be here.

Carlton Gibson

Well, awesome.

Let's kick off.

Perhaps a good way of getting you if you could introduce yourself, tell us about your background, how you got into program, for instance.

Yeah.

Or on how how you discovered Django and such things like that.

Anna Kiefer

Yeah, yeah.

Actually, so I got into software development through a bit of a roundabout way.

I was and not sort of formally, you know, trained as a computer software engineer, but uh I was working at World Wildlife Fund on their renewable energy team and uh was tasked with helping sort of design, build, you know, a simple just website for some of our corporate renewable energy partners.

And that's where I got introduced to software development.

And I really loved that.

Um and so I did a fellowship in San Francisco and you know that gave me a little bit more, of course, more foundational knowledge.

Um and I wanted to combine energy, uh, my sort of uh interest in energy and the power grid uh with um programming and software engineering.

So building web applications, um, you know, sort of track and measure the energy grid and I guess the physical world around us.

Yeah.

Um yeah, and so uh I started as a software engineer at Kavala Analytics, and they're an energy analytics uh startup uh in San Francisco.

And they do a lot of mapping.

So um, you know, mapping feeder or distribution lines, um, substations, transformers, um, solar panels, so mapping all and and uh aggregating all this energy data.

And that's where I started uh learning Django and using Geo Django.

Carlton Gibson

Okay, and so were you what were you using before just out of interest?

Anna Kiefer

Like um well, I don't think we had, we weren't really utilizing any, I think just Postgres, right?

And so doing raw sort of you know, using Psycho PG2 or something.

Um and connecting to Postgres that way and not really, you know, very hacky hacky, I guess, not really using an interface.

Carlton Gibson

But you were you were lucky enough to start with Python then.

Anna Kiefer

Yes, yes.

Carlton Gibson

Fantastic.

Anna Kiefer

Yeah, yeah.

Um and so yeah, when I came on board onto Kavalo, we were or they were using um already using Python and already using uh Django.

I don't think they had started utilizing Geo Django.

Carlton Gibson

Okay, so tell us about GeoDjango.

What is what is GeoDjango?

Anna Kiefer

Yeah, so GeoDjango is Django's sort of answer to geospatial uh mapping.

So it's their geospatial uh web framework.

Uh so it's it's just a contrib module, basically, through Django, um, that handles geographic information.

Um and uh it offers um you know the ability, like Django, uh, to you know have models and classes, but uh in addition, it offers the ability to um handle, manipulate geospatial data.

Transform it.

Carlton Gibson

What so what sort of tri so what what sort of when you say handle geo geo um geographical data, what do what sort of what sort of things am I gonna do with it?

What sort of uses am I gonna have?

Anna Kiefer

Right, yeah.

So I think maybe uh the easiest way to describe this, maybe let's take like you know, type of data you might want to model, right?

So say a road.

Um and so just like a regular Django class or model, uh you might have, you know, a road has a name, a road has um maybe uh a length, a road has uh maybe the state that it's in, all this sort of you know metadata that are just strings.

Uh but a road also has uh a geometry associated with it.

And so GeoDjango allows you to uh specify a geometry field, um, and this can be a point, a line, a polygon, you know, a multi-line, a multi-polygon.

Um I think there's like eight different types, something like that.

Um and you basically specify this.

Uh so a road, you know, road would probably best fit as a line.

Um and uh through that, um through that uh attribute, you're able to do transforms, you're able to uh do other sort of geospatial and distance queries.

Um yeah.

Carlton Gibson

So could I could I could I create a TomTom type app?

Like a um, you know, if I had all the roads in as long as you're

Will Vincent

dating yourself with that reference.

I don't think so.

Carlton Gibson

Yeah, I probably am.

What's a what's a TomTom?

Google Maps.

Carlton Gibson

But yeah, could I do Google Maps?

Like um turn by turn direction, I guess is the thing.

Could I do something like that?

Anna Kiefer

Uh yeah, we did not, you know, we did not build some something at all like that.

Well, you know, we had a map and we had but but yeah, that and other some other fancy you know algorithms and optimizations you could in effect create Google Maps where you enter in, you know, the address of your destination and it geolocates your specific point.

Uh and using Geojo, you could route to that point, uh, you know, the fastest way or something.

Carlton Gibson

But Google Maps is quite pretty, right?

It's got those nice um tiles and it shows me the pictures, but how do I do that with Geojo?

Right.

So I have I can't use can I use Google Maps with it, or do I

Anna Kiefer

yeah, you could use Google Maps API, certainly, um, with a you know, using Dojango uh as the framework, backend framework.

Um, you know, so so what the user sees on the on the front end, right, is just you know the Google Maps, those are like Google Maps tiles.

Of course, Google Maps also uses like all the API layer and all the fancy things.

But um, but you could certainly use, you know, say leaflet um or map box with like different actual tiles uh that are say the Google Maps tiles, and that is just um the uh how do I describe like the I guess what you're what you're seeing, what you're viewing.

Right, on top, yeah, yeah.

Carlton Gibson

Right.

So I've heard of Leaflet.

Will Vincent

Well, I was gonna I was just gonna say, so I've used Leaflet in Mapbox, and I was curious uh for you, Anna, how would you how would you describe the the set the options available?

Because I mean Leaflet is open source.

Mapbox is open source data, though they're VC funded, and then Google is their own thing.

I think because Google, I I'm remembering, I'm dating myself, five, six years ago, Google did one of their periodic we should monetize our API things for Google Maps and charge like hellacious rates, and then everyone got all excited about kind of Mapbox and these open source.

Um so so what's your take of the landscape, right?

Because I know now there's Apple Maps.

Anna Kiefer

Um I guess the interesting device is Map Map is a different is another

Carlton Gibson

one of this one that's

Will Vincent

Mapbox uses that data, right?

I think or used to.

They were a skin on top of so it's I guess the two parts are there's like proprietary data, which like Google and Apple don't share, and then there is OpenStreetMap, which I believe is what powers Mapbox and Leaflet.

Yeah.

And Leaflet is more of a open source, whereas Mapbox is also open source but has raised a ton of money and

Anna Kiefer

right, right.

I think ultimately Mapbox is this is the umbrella over Leaflet, and Leaflet's the Mapbox sort of free lower.

Will Vincent

They hired the Leaflet per uh guy, as I recall.

Um anyway, so how how do you think about this with projects, right?

Because at least I simply break it down into like, yeah, there's Google APIs, but they could charge a lot, and there's open source.

Um, what are the what are the factors you think about like for a personal project?

Like, because you have some really cool ones on your site which will link to um versus an enterprise setting.

Are there any distinctions there that

Anna Kiefer

you would make?

Right.

Um, I would say ease, you know, ease of use, and that's where Leaflet and Mapbox to me have really made it like their documentation is is really great.

Will Vincent

No, they're way better.

Anna Kiefer

Yeah, I mean, we were actually at Kavala, we were using OpenStreetMap and not using Leaflet or you know, or Mapbox.

Um and we found it a little bit um, you know, they're they're a team of I think they're volunteer um, you know, so so it wasn't just

Will Vincent

anyone can do it, like you and I could go in and it's sort of like Wikipedia.

Anna Kiefer

Right, right.

Um, which, you know, in some regards is really is really great.

Um, but I guess they were less full featured than we wanted.

Um and so you know, Leaflet just has so many sort of bells and whistles if you want to create a map that looks really nice.

Um it's got the ability to do choropleths, uh, which are just overlays.

Um it's got you know a really easy way to code um, let's see, you know, satellite, toggle, um, pop-ups, um, you know, all these sort of the things that you know a user would expect to, you know, to to see in it on a map.

Um, yeah.

Will Vincent

Could you say a little more about chloroplasts?

Because I those are I uh if people haven't used like leaf learning mapping, that's a big word.

Um what would how would you explain that further?

I've used it, but I you know it's me

up at first.

Anna Kiefer

Right.

Yeah.

What's an example, right?

Will Vincent

Like I mean, example would be something like I don't know a color scheme for like you have right you've you have predicting disease spread is one of your projects like where it's redder.

Right that would be an example of a chlorophyll.

Anna Kiefer

Exactly yeah so so it's it's a a you know way to to visually you know display data when say points or lines uh don't convey the information you want.

So something that's like disease spread would be really um you know a a good way of sort of maybe um or where where you would use a choroplath.

Then way you know areas that are really um affected by the disease or is a really dark or dense in areas that are not um so

Carlton Gibson

so like a temperature map on a weather forecast right

Will Vincent

yeah yeah I was just thinking like are like a political like a red blue purple like US political kind of map

Carlton Gibson

okay and all of this is built into leaflet and that's leaflet JS.

Okay.

And is that part of Geojle or do you have to pull that pull that in for your front end when you're writing your templates?

Anna Kiefer

Right so that's not part of Geojo exactly.

When you you know you you basically after you you know construct your view um you throw the sort of leaflet um you use the leaflet API to to just pull a map up in the browser um yeah and uh and display your tiles and then you send your GeoJSON um oftentimes from Django to you know display on the front end.

Carlton Gibson

Okay and how do I serialize?

So is that so say I've created a Django model, a Geojle model and I've got some you know right um geographic data in there I've got you know my roads or the the shops in my town where I want to go shopping next week I've I've put that into Geojo for some reason.

How do I send that to the front end?

Do I need to use REST framework or can I just send does does Geojle give me the serialization tools I need just to send GeoJSON

Anna Kiefer

that would be the same way you would send data through uh through Django.

So you know you would use a use I guess views.py it's been a little while since I've used uh Django but yes you use your views.py and then your URLs um uh and then um you use Geojo to you know once you've retrieved your data from whatever database um to serialize that data as GeoJSON um yeah and then

Carlton Gibson

but but the the point the question was um the it includes Geojo gives me the serializers I need to

to turn right okay yeah because sometimes you need to put in a third party like REST framework or I don't know what.

Anna Kiefer

Right, right.

I believe it gives you the serializers you need.

Yep.

Fine.

And along with um you know I think the biggest benefit or the biggest um yeah the biggest benefit I guess that that I liked using GeoDjango for was that you know you didn't need to do these sort of hairy SQL queries, you know, raw queries in your Python.

It provides an interface to PostGIS which is Postgres's you know geographic database to do these perform these queries much simpler.

And so you know a query say taking that roads example say you want to find all of the roads that intersect with a hospital or that you know run in into a hospital or that are within you know a certain distance of your hospital GeoDjango provides a really easy way of filtering and doing these distance queries.

Whereas with you know just with SQL it's you know a little bit more confusing I guess.

Will Vincent

Yeah I mean that's the Django ORM data data from RAW yeah well so so you had a fantastic uh talk on Geojango which we'll link to which is I think how Carlton I both um c came to know of you um but can you recall what was the process of like how would you recommend someone learn Geojo?

Because it is a somewhat advanced use of Django and these other technologies like how would you baby step someone up that curve if you could do it again?

Because I suspect you were kind of maybe thrown in the deep end or did you start more on like the front end JavaScript side before?

Anna Kiefer

What was

Yeah I started actually with Flask as many I think Python users do, you know, go from Flask and then think you know I little need a little bit something you know something with a little bit more I guess bells and whistles and and structure.

Batteries.

Yeah yeah um and then you know I just remember for the first project I did I was banging my head against the wall I was trying to get oh it was um I was trying to get I think it was I did this project where I mapped landfills across the US um because I wanted to I wanted a user to be able to enter their zip code and then see where their trash you know routed to and if that landfill was full um you know or how close to capacity that landfill was.

Anyways and I remember banging my head against the wall um trying to you know figure out you know how to get this data to to do to just display on my tiles.

And I was probably something very minor that I had done wrong and you know all of a sudden the entire US was full you know I made must have made one tweak and all the US was full with landfills you know I mean mapping

Carlton Gibson

something is quite realistic though.

Will Vincent

It's hard and then you also you have the visual component so it's not like just like a little error it's just like you're like you can see the mistake that you made.

Yeah yeah yeah um and oftentimes you know there's the there's simple things like the lat long you know switching those geographic yeah yeah so geographic systems oftentimes I think nearly all the time actually post GIS um certainly they use longitude and then latitude uh to define a point whereas users um are a little bit more familiar with latitude you know and then long

Anna Kiefer

itude following

Will Vincent

right xy makes more sense than y

Anna Kiefer

right right um anyways yeah so so how would I get started with Geojo?

So Geojo does have this great tutorial um on um you know seeding a database with some GeoJSON or with a shapefile um and uh and then it walks you through you know the process of querying that data uh and manipulating it a little bit um what you could also do is just if you don't want to worry you know don't want to deal with uh you know maybe setting up a database connection you could

just um you know import Geo Django and grab some GeoJSON um you know in a one you know in a in a Python file and uh just you know use the methods that way.

That's really not using all of you know the things that Geojo offers, but it would allow you to say, you know, um very simple sort of sort of queries.

Will Vincent

Well because you could pull in like a GeoJSON of like a map of like the US states right or just to like sort of explain how how uh so GeoJSON excuse me.

Right.

Will Vincent

But I mean like if you pulled so if you had like a discrete data set of GeoJSON is that kind of like it would just have like the outlines of the am I getting it right?

Like there was

Anna Kiefer

Yeah so GeoJ

Will Vincent

what would you import right?

Like where like where would you get it and what would it actually look like before you applied chloroglyphs uh chloropleths to it.

Anna Kiefer

Choropleth right right

Will Vincent

pless pleths I'm saying it wrong plast

Anna Kiefer

I think it's choropleths but I always forget if there's one L or two.

Yeah like chloro or choral

Will Vincent

it sounds like very scientific.

Right.

Will Vincent

Anyways yeah I that that sort of didn't really make any sense.

But I guess I was asking uh if you think of so you have your data and then you have it how it's displayed and then you have or how does GeoJSON differ from the skins that we've been talking about like leaf like leaflet and map box.

Right.

Because a lot of times you need to have the GeoJSON yourself or get it from somewhere right before you can apply that skin on top.

But it's different from the coordinates that it would be in the database itself with um in your Django database.

Anna Kiefer

Yeah so so a leaflet or a map box or some other you know sort of front-end web framework like that allows you to display this data.

And so the GeoJSO or the shapefile or whatever you know that is the actual data that you care about that that you're gonna put on the map and you're gonna use map box or leaflet to you know, get it on your map and to display it.

And so GeoJSON is just one form of geographic data.

And this is, you know, GeoDjango and PostGIS allow you to transform, you know, you can turn your GeoJSON into a well-known text or well-known binary.

And um, you know, and use it that way as well.

Um Right.

Will Vincent

Well, I guess I'm trying to think, so like if if you had um like a table of of uh um Geojango data, so rectangular, and then on top of that you could put it get the the GeoJSON or your your shape files, and so then you it could say if I'm within the shape of a US state, you could map and see is did does the data point in my database fit into this particular shape, and if so, apply some logic to it.

I'm just trying to like uh verbally to think about how all the pieces stack together for people who've never used this before.

Because it is a little confusing.

Anna Kiefer

Right, yeah.

So so you would have um you know, you're

Will Vincent

what's the bottom?

Anna Kiefer

Yeah, so the bottom would be just the data, right?

Like coordinate data with whatever opening information.

Will Vincent

So like at at lat long, like disease is 19 out of a scale of 100 or something.

Anna Kiefer

Yeah, yeah.

Yeah.

Okay.

Um would be yeah, just your uh your data, and then you know, you you would put that data, you know, into a data table.

Um and again, you would assign a uh you know geometry field uh to this data.

Uh how would you do that?

That's just done in uh Django classes.

Will Vincent

Oh, right, but that's the one there's a couple options, right, for how you would do that?

Right.

Carlton Gibson

So that's like a model class.

So like that you've got your normal Django model, like I know you did your disease, disease rate, and then you want to say it's at this particular location.

Is that so you could add a point field, would that be?

Anna Kiefer

Yeah, yeah.

So say you have um, you know, let's say illness, instances of illness or something, right?

Um you know, you could have, yeah, you could have, you know, one instance of illness would be a row in your table, and then you would define, you know, as you know, a column that's that's that lat long at that location.

Um and that would just be, yeah, that would just be be a Django point field.

Okay.

Um and then you'd be able to, you know, take a polygon, say, from a different table.

Maybe you have your states table, your US states.

And you know, do a query to combine these two.

So you could say um how many instances of illness are within this particular state.

Okay.

Carlton Gibson

And then you'd show it on the map.

Anna Kiefer

Right, right.

Okay.

And this is where you know Geojo is um makes this all very easy and fast.

Carlton Gibson

Super.

Okay, and a couple of times you've said post GIS, which is Postgres, which is like the Postgres extension that that enables all this geographic stuff, right?

Yeah, yeah.

Anna Kiefer

So so GeoDjango can be set up, I think, with a number of different databases.

Um MySQL, SQLite.

Carlton Gibson

Space Spatialite is is like these the SQLite version.

So that's just a file that and one might think that's the simplest to get started with, but is that the case?

Is that is it simplest to get started with that or simplest with Postgres?

Anna Kiefer

Um I guess so in the sense that you don't need to.

I think that's what Django comes automatically with SQLite.

If I'm like, it uses it as sort of its default.

Um but it's quite easy to get started with Postgres as well.

Will Vincent

Um I think it's that you need to add either Spatial Lite to SQL, which is built into Django, or if you install PostgreSQL, you need to add Post GIS on top of it.

Anna Kiefer

Right.

Carlton Gibson

It's the setup.

What what would you recommend people to do?

Will Vincent

Which of the two?

Carlton Gibson

Yeah.

Or the three.

You know, my

Will Vincent

well, Oracle's not supported, and then MySQL is it's a bit of a issue, I th I believe.

But you're the expert, so you tell us.

Anna Kiefer

I've actually used primarily Postgres and PostGIS.

So I would have to say, you know, use those, but I don't want to make any enemies.

Carlton Gibson

I've heard tell it's the most capable as well.

Anna Kiefer

What is post-GIS?

Yeah.

Carlton Gibson

Yeah.

I've heard I've heard people say that it's the it's the more advanced, it's the

Anna Kiefer

Yeah, it's you know, they've uh they've done so much with geospatial data.

We were, you know, at at my current company were doing some of these um queries by hand using Go, a different, you know, not Python.

Um and uh they're they get very complex very quickly.

Um so using something like Geo Django and you know and PostGIS really uh just make these querying and filters so much easier.

Leave leave the hard work to you know to to people who have already done it.

Um because is

Will Vincent

the idea that sorry, go ahead.

Anna Kiefer

Oh no, I was uh Yeah, I was just gonna say various um you know things that geographic data can be, I wouldn't say finicky, but um more like it can be easy to get um incorrect, particularly with units, um when you're doing filters, um, you know, is it meters or miles?

Um there's something, there's a there's a geography concept, and then there's a geometry concept.

And GeoDjango also uh you know reconciles the two of these.

Um but a geometry concept uh takes the say polygon or line, takes the shape, and models it on a 2D plane, whereas a geography uses a spherical representation.

So that's sort of um you know where you the state comes in, right?

So it's taking um you know these points and actually models them to coordinates.

Um and this concept, you know, you need to make sure to transition between the two at points, um, or else you know you might be you know your units can get all get all messed up.

Carlton Gibson

And so does does Gian Yang do the right thing there?

And um, you know, if you pass a geometry into a geography, it says, no, hang on, you've you've got to convert these first.

Anna Kiefer

Yep, yep.

And you can specify, typically I think geometries are a little more are more common to use, but you can specify that you want to use a geography.

Uh, and then it will it will um handle all of that for you.

You don't need to, you know, be doing transformations and and crazy sort of you know x, y to you know, to your projected coordinates, all that complication.

Carlton Gibson

Like the more I listen to you, the more I think, wow, actually there's just massive masses and masses being wrapped up here.

Masses of maths that you don't have to know, masses of SQL that you don't have to write.

Anna Kiefer

Like, absolutely.

Absolutely.

Yeah.

Will Vincent

So what's the Go equivalent, right?

Because I assume at your current job, the idea is that Go is faster in quotes than Python, and therefore you can't use all these nice things.

Is there an equivalent uh framework for Go spatial data?

Anna Kiefer

There are some.

Um there are some.

I'm trying to think of the one.

They're not they're nowhere near as popular as um something like Geo Django or um,

Will Vincent

so you have to stifle the urge to say, you know, this was solved if we'd been using what I previously used.

Yeah.

Anna Kiefer

In fact, uh we're switching, we're we're switching back.

Will Vincent

Okay.

Anna Kiefer

We're switching back, believe it or not.

Yeah, and part of the reason is that we were doing, we were creating all of these um, you know, all these functions by by hand, basically.

I mean, we're using the the Go GIS libraries, but even with that, we needed some custom functions uh that weren't available or aren't available in Go.

Uh so actually, yeah, we're switching back to using um to using Python, using uh PostGIS, um, which we could have been we're certainly using with Go as well, but um

Carlton Gibson

Right, so the database remains the same.

Anna Kiefer

Yeah, it's it's more the you know, sort of Python geospatial libraries um that are really full featured that we weren't quite getting as much in in Go.

Again, I don't want to make any enemies and say that there aren't great libraries to use, but um,

Carlton Gibson

but it's kind of interesting because you know, on on on the on the chip, Go is faster, you know, it's a faster language, it's more low level, it's you know, it's got higher throughput, but there's developer time as well.

And you know, if you've if your Python is fast enough, the developer time trumps it, right?

Anna Kiefer

Right, right.

Will Vincent

Well, I mean, as we talk about often too here, uh generally in a web stack, the language isn't the big issue.

There's a whole lot of other things that are slowing you down.

Yeah.

Yeah.

Anna Kiefer

And we found the development with, like I said, the development with Go and with creating these uh all these geospatial, you know, functions ourselves to be a lot just it was costly developer time.

Um and we also aren't in sh aren't sure we're doing it right all the time.

Carlton Gibson

Yeah, right.

Anna Kiefer

Okay.

You know, you can write your test for your geographic function, but you know, you're you can make your test pass.

It's the the beauty of test.

You know, you also write the test path, but it still may not be entirely accurate.

So um you know Yeah,

Carlton Gibson

it's so sometimes uh an algorithm matches the test case, but or the test cases,

Anna Kiefer

but then yeah, they're both errors.

Right.

Right.

Will Vincent

Well that's why you want to always want to check, you know, does it do what I think?

And also doesn't do this other thing that it shouldn't do, because sometimes it just always passes.

Right.

Well, so what so day to day, like what so what does it look like?

What are the challenges that you wrestle with?

I mean, I maybe it sounds like maybe classic data science, where a lot of it is it's a little bit of algorithms and testing, but a lot of making the data fit and not be wrong, right?

Because if it's w 0.1% off, it blows everything up.

Is that accurate?

Or how how would you describe like would you describe yourself more in the data science side or more some other area?

Anna Kiefer

Right.

Well, I would say so.

My previous role, I mean, at Kavala, well, it was a a smaller team than I'm on now, um, which was awesome because you got to work totally doing everything, you know, from front end, a lot of mapping and coropleths to the API layer, uh, to, you know, the the data ingestion and and data pipeline, uh, and to to our you know, sort of energy, developing our energy methodologies.

Um and now I am on the data engineering and sort of a little bit more on the data science uh side of things.

Um and you know, I'm no longer doing, I guess, the the front-end mapping, uh, although we certainly have a need for, you know, for for a fully fledger, full-fledged um front-end and map framework, but I'm not currently doing that.

Um so I'm doing more of our data science, productionizing the data science methodologies, uh, to do things like um create a power network using machine learning.

So figuring out where feeder systems and feeder lines are on the grid from you know from from actual data, feeding it into a machine learning model, um, and then validating it and seeing you know how close we were using the actual data.

Will Vincent

And I assume that's I mean that's very Python-based most w which which packages are you uh using day to day?

Which of which of the which Python ones?

Anna Kiefer

Yeah, you know, right now we're not using a whole lot.

Um and I I almost wish we were using something a little bit more like I'm like the I am spending my time now doing a lot of these raw SQL queries in in my Python.

Will Vincent

Maybe you get to do an outer join, right?

There's a discussion like uh online.

Simon Willis and some others were saying, you know, like they've never in their career done an outer join.

Really?

That's asked in every interview, but you've never, you know, in a 20-year career, Simon said maybe once or twice he'd used it.

Anna Kiefer

Um that's crazy.

No, I would yeah, I do well, I mean, they're I wish it were just an outer join, right?

It's like an outer join with uh, you know, various um various other like this, you know, get all the the feeder lines within a certain distance, um, but then mash them to the buildings, you know, and then do a sum of the aggregate on the buildings, you know, so it can get pretty hairy.

Will Vincent

Um is that s is that something where um do people talk about like GraphQL in this context at all?

Um given that you have, you know, Go like non-relational

Anna Kiefer

kind of?

Yeah, yeah.

So uh before I was on the data team, this data engineering team, I've only been on the team about maybe a month or two.

I was in Go land where it was GraphQL.

We were using GRPC and um yeah, and GraphQL.

Will Vincent

Um So postpone the decisions on the relations until later.

Anna Kiefer

Right, yeah.

Will Vincent

I'm just kidding, sorta.

Anna Kiefer

Um Okay.

Will Vincent

I mean it seems like it, especially the the data sets you're dealing with, I mean, it's a sort of it would make sense to maybe use it non-relational in that context.

Anna Kiefer

Right, right.

Um and we actually used Dgraph, which was a sort of a very I think it's small.

I think it's a team of like five, five people.

Um and it's a it's a graph database written in Go.

Uh so we were using that and it's very fast.

Um, but again, it didn't have its syntax is, you know, I guess we're also used to SQL and the SQL syntax.

And this really kind of turns it on its head.

And so we were again, we were spending a lot of developer time just trying to get this syntax working to query the database.

Um I think with with you know, obviously, with practice and more experience, and as the Dgraph team, you know, that that software becomes a little bit more mainstream, we might go back to using it.

Um, but for now, um, you know, given all of the geospatial data that we do deal with, we wanted something like that that we were comfortable in.

Yeah, that being SQL.

Carlton Gibson

It's interesting because like a graph database, uh I've always found them deeply fascinating, but just never really found been able to make it work in the real world compared to the established stuff.

But it's it's like yeah, everything's a graph problem if you look at it in the right angle.

And surely we should be able to solve it this way.

And then I can use these nice little pathfinding algorithms.

Anna Kiefer

Yeah, yeah, yeah.

It's all great, it's all great.

Anna Kiefer

But I was really interested in to use it and was really happy that I got, you know, a couple months' experience using it, even though we're not using it in production now.

But um, yeah, and we kept, you know, it it's it's a whole sort of mind shift because we kept saying, you know, I kept saying, oh yeah, so they're you know, the rows and the table, and then you're like, oh wait, but they're not they're not rows, right?

In fact, it's not a table.

You know, and you're like, what?

Okay, the nodes, and then we kept having to, you know, change our how we how we describe this data, you know, the nodes in the you know, in the graph, right?

And it's just like a a different conceptual way of thinking of data.

Carlton Gibson

Um Right, but it's kind of interesting.

Uh we talk, you know, in software design manuals, they're always talking about matching the the the mental model of your users.

In this case, like you're the user and the database isn't matching your mental model, and so there's a an incongruence and yeah, that's that's kind of cool.

Anna Kiefer

Yeah, yeah, and I think um something, I mean, I think like we're still trying to find how to use graph databases because some of our problems are really more suited to graph databases.

Um like we have something that we kind of call dependency chain, and that is a um a link of um sort of cascading of I guess dependencies, right?

So if you think about a power system, um, you know, power system has a dependency, say, you know, I'm on on uh water, maybe.

So a water plant, let's just use this as an example, and your hospital is downstream, and that has a dependency on both the water plant and the and the power system.

Um and so you know you've got these sort of cascading dependencies, uh, and modeling that in a table could be pretty hairy because you know you've got your okay, dependency one, but many, you know, there's many, it's m a lot of many-to-many relationships going on.

And something like a graph, you know, can model that a lot a lot better.

Will Vincent

Yeah, as soon as you have one many to many, I start going up.

Here we go.

Yeah.

Yeah, yeah.

Anna Kiefer

How many association tables you need, you know.

And that's something we haven't really addressed, I don't think, yet.

At least at my company.

You know, there's technologies that have, but right.

Will Vincent

Well, I mean, the thing about data science, right, is at least from I was just look looking when I was studying this.

This is a couple years ago, there was this Twitter feed, Big Data Borat, which had all these pithy sayings.

And I mean, I remember being struck, I spent almost all my time cleaning the data, and then a teeny amount actually running an algorithm or something I made, and it's just kind of like it felt very um you know janitorial, but I think that's just kind of how it is, right?

Anna Kiefer

You it's toil.

Will Vincent

Like it's hard to clean it and you have to do it the right way, and then uh yeah, I guess you go down.

Yeah, so it's there's some quote about that.

Anna Kiefer

Yeah, yeah.

And in fact, in in both roles, and I think, you know, I we deal with a lot of sort of students, um, you know, who who also deal with geospatial data.

Um and there's while there are certainly uniform, you know, there's GeoJSON, um, there's uniform ways of of handling geospatial data and making sure that you know the data is is the same from one government, you know, or one organization to another, a lot of the format is not the same.

And so we can build an entire data pipeline for you know our US data, right?

So that's ingesting US census data, but we have the need now to ingest Japan's census data, and it's entirely different.

So that data pipeline, you know, we need to change drastically in order to get it to work with you know these files, basically.

Um Yeah.

Carlton Gibson

And I I always get the impression that data pipelines, you know, I'm more of a web jockey, but you know, data pipelines are always like the sort of one-shot scripts that I write for myself to do stuff.

And they're sort of bodged together with duct tape and oh, you know, and I'll just run this here.

Yeah, it'll

Will Vincent

find out what's good to go.

Yeah.

Right?

Yeah, yeah, yeah, yeah.

Anna Kiefer

Yeah, you know, at least at both companies I've been in, we've we don't, you know, it almost seems like once the data's in, right, you do it once almost, you know, and then you don't have to do it again.

We're not talking machine learning models or anything, we're just talking static data.

And so, you know, you have all this, yeah, the duct tape, and you have, you know, it's all shoddily constructed, but then it's in, right?

And you don't really and every time you're doing that, you're in that process, you're going, oh gosh, we need a better way of doing this, we need a better way of doing this.

But then it's in, you know, and you kind of forget about it.

Right.

And then, you know, then that's everything.

Anna Kiefer

Yeah, yeah, right.

Will Vincent

Yeah, well, you don't want to prematurely optimize it, right?

I mean don't just polish your tooling, gotta use it.

Yeah.

Yeah.

Anna Kiefer

Yeah.

And so like right now we are developing, you know, well, knock on wood, but we're trying to develop a, you know, a more flexible, fully featured data pipeline that we can use when we deploy, say, you know, in Japan and China and India, and you know, where it can fit with uh, you know, I I still don't know how this is gonna work.

Just being the one to to you know build the to duct tape the pieces together.

Um I can yeah, I'm I'm questioning how we'll find a solution, but exactly.

That's what makes it fun.

Will Vincent

Yeah, yeah, yeah.

Yeah, well, I I think that's how often do these does this change, right?

Like because some data sets you have to ingest, you know, minute by minute, if it's like a parameter, and then some are I don't know, you know, monthly, yearly.

Yep.

Anna Kiefer

And census data is every 10 years, right?

And so, you know, if you only have to do this, you know, hacky construction once every 10 years, right?

You're like, that's not so bad, you know, we don't want to spend, yeah.

But but exactly, but something like um like pricing, like um energy pricing data is like every, I don't know, every couple minutes, um, or or meter data, like you're saying, is every couple minutes.

Will Vincent

Well, yeah, because there's some um in the Boston area, there's a couple companies that are using Django to uh yeah, like to uh Internet of Things to check on solar meters.

And it's as I recall, I mean, I kept I remember being like, well, why don't the device manufacturers do this?

But basically they need a degree of separation, so it's um yeah, I won't name names, but it's it's interesting, it's a very interesting problem, and it's kind of shocking that it um isn't addressed by a big behemoth that I'm aware of.

It's sort of smaller places that are doing this mod monitoring service for these power grids.

Yeah.

Anna Kiefer

Um yeah, I think it's um, you know, we're still at the forefront of all the sort of smart grid and grid optimization work.

Um and it's gonna be exciting to see uh, you know, how's see how it shapes up because we have this old infrastructure, and then you know, you're trying to glean insights and you know, f figure out basically, you know, how to optimize you know power on the grid.

Um Right.

Will Vincent

Well, and it's I mean Internet of Things is I I would think Carlton like a pretty good use case for you know async Django stuff.

I mean depending on the side of the side of the side.

Well, right, async anything, right?

I mean it's aside from you know a chatbot or something, it's it's that's a real world application.

When I think of async up um, because Django's adding uh async in 3.0, um that's the first thing that comes to mind for me, all these Internet of Things.

We have something just pounding off data, you know, every second or faster.

Carlton Gibson

Yeah, so whenever you whenever you need anything to handle an an event at an unknown interval, then async becomes perfect.

So Internet of Things is perfect, you know.

It doesn't know when you go and open your fridge door, but when it when it gets the fridge door open, it needs to handle it and tell Amazon to send you some new milk or interesting.

Will Vincent

I guess that is it's the unexpected part, yeah, because otherwise you could just plan for it, but it's the spikes that async really shines.

Carlton Gibson

But like go come back to the energy optimization, because that's fascinating.

I sort read a story a while back about Google, and obviously they control the environment in their data centers perfectly, but they use their AI to you know save 60% on the power bill or something.

I I don't know the number, but can we are we gonna be able to do similar with the with the grid?

You talked about the old infrastructure, obviously we can't just rip that up and start again.

But do you think there are savings to be had and efficiency gains?

Anna Kiefer

And yeah, yeah, definitely.

Um and figuring out uh yeah, there's certainly cost savings, there's energy savings, um, and not being you know an energy expert, I don't want to um say too much, but um but there are certainly um like we have very little information other than you know what your utility bill, right?

The average consumer is very little information about what the demand is on their, you know, on the grid at a given time.

You know, maybe they do want to uh you know lessen their energy bill in some ways or you know put solar panels on the roofs, and you know, if you wanted to change your energy provider, you know, basically areas have pretty much one energy provider and that's it, right?

Like we've got PGE, yeah, utility, and we can't, you know, we can't change it.

And you know, I'm sure lots of people would like to.

Um, but I think giving consumers a little bit more um insight into, you know, their sort of energy uh and electricity, whether that's through something like the Internet of Things and the smart grid, you know, that's gonna be really revolutionizing the industry.

Right.

Will Vincent

Well it's that and it's that two-parter of you know, you could mail someone an automated monthly packet saying, here's your usage and how it's changed, but then you need that second part, like here's a non-PGE option.

Right.

Um well and in and in the solar context, you have an interesting thing where you have with people who are net um contributors to the grid, and actually it sort of messes things up for the utilities if in a good way, like if uh there's too many people giving stuff because electricity actually has to kind of flow like water, like in the same way that LA create all the farmland because they're stealing too much.

Anyways, um what uh is there anything we haven't covered that you want to address in the podcast?

We're sort of coming up near the end.

Um or anything you want to plug?

Anna Kiefer

Right, right.

Um I don't think so.

I mean, I mean I would say yeah, I've I've absolutely been very happy using Geojo, and for anybody who hasn't, play around with it.

And uh they say that people who like maps really like maps.

Uh and Geojango certainly makes that um, you know, makes at least made me sort of love maps and geospatial data and doing queries and sort of making all of these data discoveries.

Will Vincent

And you're moving back to Python world a little bit at least.

Anna Kiefer

Exactly, exactly.

Yeah, and I uh I think that's about it.

I did write a some notes.

I'm trying to see.

Oh yeah, one thing that I just on the sort of the future of uh of geospatial data, if I could say, so one thing I haven't handled much is uh, you know, I've handled mainly 2D geometries, uh, but handling um 3D geometries um I think would be really interesting.

So, you know, my current company, we have uh, you know, say we we might map a flood, right?

Um and you know, a flood has a depth.

Um and actually, you know, being able to map this depth, uh, you know, we might glean other insights that we certainly might not from 2D data.

So, you know, volume of water.

Carlton Gibson

Flood prediction as well, you know, like if you on this volume of water will this this housing estate will be under threat and it needs this kind of barriers to protect it, that kind of thing.

Exactly.

Anna Kiefer

In fact, my current company, one concern, does exactly that.

So you have really just hit the head, yeah.

Carlton Gibson

Right, fine, super.

Yeah, yeah.

Anna Kiefer

So um and I know PostGIS does have the ability to to handle, I think, 3D data.

Um, you know, but I haven't played around with that too much.

Um and but I would be very curious too.

Carlton Gibson

Okay, but that's on the horizon that's available to that's so that's kind of cool.

Anna Kiefer

Right, yeah, yeah.

Absolutely.

Will Vincent

Yeah, I mean I have to play around with this a little bit.

Um it's been uh I haven't used Django when I'm doing mapping.

Well, I haven't done anywhere near the functionality we can, so I'm sort of excited to do that again.

Yeah.

Um well, so we will link to you have a personal site.

Um and yeah, thank you so much for coming on.

Thank you also for your talk last year.

I thought that was a great idea.

Anna Kiefer

Yes, absolutely.

Will Vincent

Yeah, and it's funny that there's a you know specific tutorial because uh there's not that many tutorials within Django itself.

People just think of the polls tutorial, but Dojango is really its own kind of world within Django.

Anna Kiefer

Yeah, yeah.

I think they use spatial light in the tutorial and they use a shapefile.

So I think they have you sort of seed your database using the shapefile and then create a you know a simple model uh to play around with the data.

Will Vincent

Yeah, then you need then you need a way to display it.

Anna Kiefer

Yeah, I'm not, you know, I'm not remembering what map mapping framework they use, but

Carlton Gibson

all right, Anna.

Well thank thank you so much for coming on and telling us about GeoDjango and your the work they've been doing and the discussion around all of that has been super.

Anna Kiefer

Yeah, awesome.

Awesome.

Thank you for having me.

This has been great, you know.

I wasn't uh sure if I'd be um, you know, have not having a taking a little break from Geojango, so I'm glad it went okay.

No, it's super simple.

Will Vincent

No, I think that's the best.

That's the best when you you have a little perspective on it, right?

Because when you're in the reads, you don't really think think about it in the same way as when you're a little bit far apart from it.

Anna Kiefer

So Right, right.

Totally.

Will Vincent

All right.

Thanks again.

Carlton Gibson

Join us next time.

Bye bye.

Anna Kiefer

See you guys.

Bye.