Transcript: Search & GeoDjango - Paolo Melchiorre

Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework.

I'm Carlton Gibson, joined as ever by Will Vincent.

Hello, Will, how are you doing?

I'm great. Hi, Carlton.

Hello. And today we've got a special guest, Paolo Melchiorre.

How are you doing, Paolo?

Hi, I'm fine. Thanks for having me.

No, thanks for coming on. Thanks for coming on, super.

So Paolo, tell us and the listeners who you are and how you got into Django and programming and, you know, what's your backstory?

So I'm Paolo Macchiore, I'm from Italy, a small town close to Rome.

And I started programming in high school many years ago.

I started with Pascal language and then I haven't any internet connection.

So I started studying myself HTML, and for the dissertation, the final exam, I made a website.

I was the first one in my school at the time.

And then I decided that web developing is my life, and I studied computer science at university,

and I was very interested in Linux and databases and all this cool stuff.

So I made a dissertation on free software and about licensing free software developing.

Then I started working for a company with Python. I studied only the classical language in the

university, like C, Java, similar one. But then when I started working with this small company,

We worked with Plon. It's a CMS based on Zoop. And so I started using Python 2.4 in this location.

And it was very good because I started also being involved in Python community. I attended my first

conference in Naples. Plon conference was the international one in 2007. And so I made

internet website for a few years with Plone. Unfortunately, it was based on the ZoDB,

the Zo Object Database. It's an evolution of the Pickle from Python,

one of the first NoSQL database in the history. That's kind of cool, right? I mean,

Pickle's nice because you get some objects, you've got a few things in memory, you go,

go, I need to keep those.

I'm just going to pickle them.

And then you get them back

and they're exactly how they were before.

And there's none of this, you know,

serializing to SQL.

But pickles are always dangerous, right?

I always think of pickles,

I think danger, danger, danger for security.

Well, you know.

Potentially.

Yeah.

Well, it's a bit difficult in that time

to interact with a classical website

or other technology

because there is only this Zodibit,

this object database

and no one know anything about it.

so you're forced to translate data in relational database.

And for this motivation, I started using Django.

At one point, I changed my company

and we started exchanging data in industrial software.

So I picked Django

because it was one of the first framework database

that worked well with Postgres.

So it was Django 1.0, I remember.

And another thing strange was that Plone

was not ready for WSGI.

You can deploy Plone on WSGI at the time.

And now I think they updated all the infrastructure.

But then I started using Django in a WSGI environment

and with relational database.

And I was, I've, I fall in love with admin and with battery

includes, uh, possibility for Django.

And then I found also that you can use very easily geo

Django and post case for, uh, for GA GAS.

So, and then I'm still here using Django after I think a lot of versions.

So this is my, my trip from starting it to now.

So you found it and you start with it and here we are.

Yeah.

Yeah.

Yeah.

So I wanted to ask you about Geo django because I saw you gave a great talk at,

um, DjangoCon Europe in Copenhagen on Geo django.

So.

Yeah.

Yeah.

And we made this, uh, uh, we rebuilt this website for our company in France.

Uh, they sell all around the world, um, buildings and house luxury

house, uh, close to the beach.

So the name of the company was Merredmer, my French is not good, so I don't know if the pronunciation is okay.

You've got to be careful how you say that, right?

Yeah.

And so we used PostGIS and Leaflet to show them up all the things you can sell and you can buy all around the world.

and we made everything with postgres from the geographic location to the full text search in the

in the description of these houses or around the world and was very interesting because we

we did everything in the same environment we didn't use any external engines

to search for location or search for information we had also this visited

search in the website you can see at the same time the location and how many of every option you can

you can find clicking on different filters and so i i wrote this the talk and i spoke at copenhagen

last year was very interesting and i received also a lot of requests of information and how we did

this uh this platform uh because a lot of people found that the tutorial in geo jungle documentation

was a bit complex and me too i found that it was very it presented a very very complex case study

and then you can instead adding a point field on your model and starting using

leaflet or starting using the open layer widget in the admin and it's very very easy

So you just add like the single field.

Yeah.

So we added also other fields for boundaries and for states, other things, but if you need

only a location for your object in the map, it's very easy to add.

You can add point field.

Of course you have to use a JS ready database, like SQLite, Postgres, so also MySQL and Oracle,

but it's it's very easy to use you after migrating things you can start storing

location and like other tags or number you can you very store there okay i'll ask you for the

show notes if you've got a recommended tutorial or something that like it's a little bit less

heavyweight than the django geo django tutorial okay perfect i think geo django we've had

anna keifer on to talk about it but i think for both of us carlton i don't really know much about

that area of django i mean that and the orm are the two areas where i'm pretty ignorant

but of the two i think geo django is would be a little more fun to dive into properly

at some point what i liked about your django con talk as well perla was the the how you showed

using leaflet js to because it's like okay so i put i put a lat long in my database what am i

going to do with it ah well actually there's some really good ui frameworks that you can just plug

it into and it's quite simple and you're up and running and that's you know you get to a certain

standard quite easily yeah the the shortest part i think you can use is leaflet can understand

geo json out of the box and you can also install django rest framework with the

yeah um it's it's plug-in you can uh

you can expose geo jangle serializers exactly you can expose this your location data in geo json

format and leaflet can understand directly and you all your i don't know your front-end developer in

your team can understand it without any other addition or configuration it's very very easy to

use would you is it still leaflet its own thing i thought it i thought the creator went to mapbox

and because i've used mapbox a little bit is it yeah one in the same now or are they still

separate no i read also that the creator worked now for my mapbox but leaflet is still

an active, developed library, and there are several things.

I think Mapbox found the development of these libraries,

I think.

But we use also Mapbox for personalized tiles.

You can personalize still graphics and colors of your map

If you need to, you can also use other types from other services.

Um, we also use the, this, uh, this stack for, um, the last project I used,

uh, uh, post this NGO Django was for mobile application in Android.

Uh, we send location to this mobile pro mobile application.

They sent to us a feedback with another location.

So you can use the web, mobile phone,

and you can store everything in your database,

in your Postgres with no problem at all.

Yeah, I mean, it's one of the use cases that I've had

is that you get the location data from the phone

and then you need to still persist on the back end,

geojangles, there's your option.

You can also calculate live distances

and all other things.

Yeah, bounding boxes, intersections.

Yeah. Yeah. I've done a bunch with Mapbox, but I remember it, it, and maybe Leaflet came on the

scene around, I think it was 2012 when I was in San Francisco, when Google maps, they severely,

they tried to charge for their API. And so like the company I was at, we went from free to

tens of thousands of dollars and everyone around us and everyone said no. And so,

cause Leaflet and Mapbox are both based on open street map, which is sort of as I believe like

wikipedia for geolocation um yeah yeah so anyways that was sort of the cambrian event for at least

in startup world everyone said i mean because nobody talks about using google maps with

chia django that i'm familiar with almost everyone uses leaflet or mapbox

these days yeah yeah it's true we we use some services from google that are better they're like

like address searching yeah yeah because they have a better function for this but also with

open street map there is a nominatum is a service you can use if you don't need a so very careful

location in in a city it's it's good enough to search for addresses and but one one thing i i

like of this solution is that you don't rely on external services so suddenly

for example like in Google Maps they start charging or closing or limiting

their services you have to change you you don't have all your data in your

database instead with Postgres and Postgres you store everything on your

and then maybe you can ask for help

or for other additional information externally

like for addresses similar.

So you're a big Postgres user, right, Paolo?

I mean, not to knock the other databases,

but you're a big fan of Postgres.

You've been using it a long time.

Yeah, I started realizing a few years ago

that it's the enterprise database for open source environment.

So at University, we use very old and heavy database,

like from IBM, or Oracle, and similar.

And at the same time, that period,

everyone uses MySQL, because WordPress and other PHP

framework workspace in MySQL.

Then when I started using with Django,

I found that there was still in that time,

very interesting and powerful feature.

And I started also attending the Postgres Day

here in Italy,

speaking about how we use the Postgres

from the point of view of a user.

with Django and with, uh, uh, with Python and, and every time I tried to update my,

um, my Postgres, uh, searching for new, new feature to use and also full-text search was

one of them. And so I think it's, it's few years ahead from other databases.

So I saw you on Twitter the other week, someone was, um, there was a thing going

around about what's your tech opinion that you know gets swords pointed at you and yours was

uh that if you're running postgres you don't need elastic search which i liked

yeah i i i started using full text search with django i found it from the release in django 2.0

I think they really was at the search fields that you can add on.

And yeah, when I found it, this feature was shocked.

I realized that there is other things to use instead of external services.

Unfortunately, I forced to use external services for doing a search in mobile

application and other, other web services.

Um, I want to clarify, I don't have anything against these services because

more of them are also open source and based on Lucene, uh, it's open

source from the Apache foundation.

But I'm, I'm an engineer.

I want to solve problem.

I don't want to install other soft software from other person and spend

my day trying to connect solution and to find the best configuration.

I want to solve things right in code.

So using Django or RAM and relying on the Postgres full text search

functionality, I was able to, uh, concentrate my, my effort in doing the

best search experience I can do and without having to have more than one

instances of the external engines to solve the configuration and the stack in

the middle from Django or RAM and the languages to query in Elastic I was

frustrated after two or three projects using this external services i think other people can

do a better job than mine using that but when i found that i was i was able to search directly

in the database in the postgres i started using it i and then i started also asking myself why

in the Django project website they are using Elasticsearch so after studying a bit I proposed to

the main list to update it and there was people against and people pros about this this update

but at the end also the fellow of the time Tim said that he he also had troubles updating the

version of the Elasticsearch and the libraries to communicate with Elasticsearch and so I proposed

a pull request in that time was 2017 and after a long reviews and updates finally we started using

Django and Postgres full text search internally and now it's three years it's working so

i'm i'm happy and you're busy doing the same thing again now right so as we speak there's a pull

request open to update um the version of django that django project.com is running and update the

um yeah version 12 i saw there was a whole marcus and tobias and yeah yeah so i mean you know there's

a there's a bit of an ops thing is that we have to get all the various pieces into place in order

do that but that's super work yeah yeah because um as i i shown on my talk django updated every

every year with a lot of features also postgres do the same things so every year there is new

languages they had you can search for text in new languages also russian or other different

type of languages in Postgres, and they added also web search

syntaxes and other feature searching allies, or they

They speed up their indexes.

And so if you want to use all these new features, you have to update frequently your Postgres

version and also Django version.

So I started proposing to update the Django version, the Django project.

And if we, at the end, we will be able to update it.

I want to propose new feature and to the Django search, because in this year I asked the every

time question to user also in conference, also last year in Copenhagen and local meetup

and for feedback about the search function in Django project.

And I received a lot of requests for improving the functionality or to change something to

be more user-friendly.

The second pull request I did in the past for the search functionality was the multilingual

functionality, because with Elastic, only English worked for a few years.

And then with Postgres, we were able to add more than 20 languages in the web search.

So I think also to attract more people in the Django ecosystem,

the first thing they found searching for Django was the documentation and the search functionality.

So having a better working search function is a main goal to attract people.

I'm using that every day, right?

So I need something quickly jump over to Django project.com quickly type

in the search bar, you know, I need, I need, so just before we were recording

now, I was looking at the same site cookie setting, so quickly type it in up.

It comes, you know, if the quick, the better that search works, the better.

Yeah.

Yeah.

It's true.

I use every day too, because I use also for searching about full-text search,

because sometimes I, I forgot about all the attributes you can use or it's

impossible to remember everything.

And I prefer understanding how things work

and then reading the details in the documentation

because it's very convenient.

I think a lot of people find out about features in databases

or in Postgres in particular from the Django docs, right?

So you find out about HTML fields or JSON fields

or array fields from reading the Django docs.

Yeah, yeah.

Well, because Postgres had full-text search for,

was it 10 years?

It was quite a while before, um, it made its way into Django itself.

Uh, I was, I remember we'll link to your talks and you have a great article.

I gave a talk, was it last year on search?

And basically I was just looking at your information nonstop all

last year for inspiration.

Um, so it's really excellent, but I, I remember.

Cause I have a little bit less programming experience than both of you.

And I came to Django from some other frameworks.

the two of the things that jumped out at me is that there wasn't a built-in signup author

registration. I had to do it myself and that search was also lacking because a lot, a lot of

other frameworks have some something or they have elastic bundled in. So I agree that ways that we

can make it easier and have an out of the box solution. I mean, it's kind of the whole point

of that talk and the tutorial I have was showing like the layers of search and you can quite easily

do basic search but then of course it gets much more complex so i think that's a broader thing

that's good to have accessible to django people or people coming to django who are used to just

having built-in search and whatever framework they're coming from but i love that you're pushing

the boundaries with postgres itself because we had on the founder of listen notes which is a site

that does searching through podcasts and he's using django and he's also using elastic because

he was also familiar with elastic so the fact that you're an expert but you're also not just

porting over you know existing expertise you're put you're still learning and taking advantage of

the new functionalities being added that's not always the case but that's obviously extremely

important to have someone doing that and talking about it rather than the confines of an enterprise

system where they maybe can't do that yeah motivation because i i try to promote uh django

postgres full-text search is also because elastic is very famous and they do a very great job in

marketing and a lot of projects are using elastic so they don't need other people speaking

pro this this solution and not saying anything against this solution but i'm only saying that

if you need a easy and quick solution

to implement a full-text search

and you already have Django and Postgres,

you don't need anything in advance.

That's the thing.

You've already got it running.

You're already running Postgres.

So why take on the extra difficulty,

that extra complexity,

that extra ops thing?

Use that first.

There are, I think, scenarios

when you need external search engines.

For example, if you have a centralized search functionality with Django application and, I don't know, Java application and NoSQL database, a relational database at the same time, if you need a central point to search on, maybe you can start thinking to a dedicated search engine.

and it's convenient in this case

but I work in medium to large projects in my company

and usually a Postgres is enough for everything

so if you have more than one Postgres replicated

you can manage millions of records

and there is a lot of space

and if you need elastic maybe you you have a very big project otherwise it is not the example that

um the list of notes founder gave us was high ingest rates because he's ingesting updates to

podcasts and they come in like you know there's a lot of podcasts so if you're trying to index all

of those the updates are a lot and that was his the reason he cited for needing elastic search was

that he needed to handle that interest rate but a lot of sites aren't like that well i saw that

apple i think just in the last month or two added the ability to search like within podcasts um which

so he's got a fantastic service and hopefully he'll he's got enough of a running start that

he can keep that going or maybe they'll just they'll just buy him but yeah he's at a crazy

scale i wanted to ask about so um uh hosted solutions um so algolia i'm just curious if

you've played around with algolia because that's something i've used a bunch and seen it develop

i'm just curious if you have any personal experience with with that option

no i i never used algolia i used for past project for a startup you use the sword

but like Elasticsearch is based on Lucene

and I tried also Sphinx for another project

but in every project

there was dedicated instances of these services

in one case there is an external company

that managed these external search engines

and we was forced to do all the search on this search engine and all the writes in the database

on our databases services and was very frustrating because there was a synchronizing problem

and people have this

delay in searching for information and accessing this information because maybe they modify

a detail on a project of a product and the writing in the database was very fast but then

they were forced to wait for the synchronization to the external search engine and so

the final user experience was very very frustrating because they are asking where is my

data and we were forced to say you have to wait because the external search engine is very slow

or there is connection problem it's kind of eventually consistent yeah where did my data go

it's funny also my wife usually fine for you know shopping online and every time they complain

because they found the last one

shoes

in the market when they click

when she click on

found that in the detail page

the shoes is gone

yeah right

and I explained to her

it's because they don't synchronize

the list

of products and the detail

page so

it's a very common problem

yeah and from the perspective of your crafting

your own django application that's a difficult problem you want to postpone for as long as you

can it's making sure that on every write to your database your search index is updated if you can

that problem yeah you you know forever put that off as long as you possibly can because it's a

big one well and and the issue with search of course is that uh an average user is used to

google so anything on a your average website that isn't google they're going to say well this is

crap which is a pretty um pretty high bar you're also a django girls coach and um there's been a

change and a leadership i believe we're going to have one of the new founders on quite soon but i

just wanted to ask you about maybe you could tell people what is django girls and in your involvement

with that community yes of course i for the first time i i was a coach in the django girls

workshop in 2017 in the euro python and it was a very interesting experience

Django girl workshop was a one-day workshop for four girls organized all

around the world and they accept girls that want to learn how to write Python

and Django and working in web environments and you can propose

yourself like a coach and during these days you can coach one to four girls and

and helping them to following the steps

in the Django Girls tutorial.

It's an online tutorial.

It's very easy to use.

You can also use on yourself.

And usually I suggest also to people

that want to following something more longer

than the Django tutorial.

And you spend the day answering for question

about why Python is not installing good in Windows machine

or I have a very old Linux machine and it's not enough

or a similar question.

Why your static files aren't loading, that kind of thing.

And in my first coach experience,

there is two girls, very skilled, they already use Linux.

So I waited all the day asking if they need some help.

But at the end, I was not able to help them because they are already very skilled.

So I updated the tutorial to Python 3.6 in the meantime to spend the day, but also we

organize it in Rome and in Italy, they are very active.

They organize in, you know, city all around Italy, but also all around the world.

day there is every month workshop unfortunately during this period of pandemia it's difficult to

organize in-person workshop so i i saw people organizing also online but i i hope that after

this period was gone we can start again organizing jungle girls workshop days and in conference or

during the year because it's a very good yeah that's an awesome brother that's that reminds me

there's i i need to be a i need to be a coach i've thought about and i haven't done it and

certainly once things are normalized because here in boston there's a very active python community

which has regular meetups and even workshops and that would probably be a perfect

place to you know have a have a number of people who are because the part of the problem with

Django, you know, cause I teach, I teach it for a living is you have to know a whole bunch of

other things before Django makes sense. So I like, I spent a lot of time with, okay, you have to

install it. Have you installed Python? Do you understand web development? You know, a little

bit about HTML, CSS. There is that issue of what's the bare minimum you need before you can

dive into Django that. So if you're at a Python conference, presumably they have a little bit of

Python, right? But if I just walked into a school, I can't just probably start teaching Django.

yeah and this location is good because there is girls from every environment people from

advertising people from at school and you found out they and this understood django from

very different backgrounds and yeah i i learned a lot trying to explain to this girl out django

Why Django work in that way and not in other, in other ways.

So, uh, and also you can meet other coach and have, uh, uh, a very good

reflection in your, your city or your region with other very skilled and, uh,

and people that care about the community and also involving girls in community.

I think it's very good.

Yeah, no, for sure.

Well, I'm thinking there's also a Django meetup in Boston, which I think is one of the biggest

ones in the world that I don't think we've done a Django girls there.

We should do that.

But yeah, the questions you get from people really solidify or makes you think, right?

Because people come in, maybe they don't know anything, or maybe they come from an MVC setup

and they find controllers confusing or, um, yeah, it really makes you 360 on what is Django

and how is it different?

But I think that's something actually I, you know, this pandemic will probably continue

for a bit i want to do some videos to explain just like how do the pieces fit together like

because carlton has a fantastic talk on middleware but i think middleware is something that your

average jingo person doesn't really understand at all um yeah and yeah but it could be you know i

could just show them carlton's talk but you know it could be simply explained you know peeling back

the onion but you know explaining explaining the interaction of of use templates models and urls

that's basically the django story if you can internalize those four things um but that's

kind of the django way i guess so anyways yeah people should volunteer and you learn a lot you

meet great people as you say yeah i hope the next django con europe we will have a django girl

workshop too maybe well they they just they just announced they're gonna try to do it in i think

it's june um they put it yeah so it's the week of june i think yeah so i hope that happens um i

still have my plane ticket that i haven't gotten refunded for so i'm ready to i'm ready to go yeah

the last jungle con european copenhagen was very good i met also carton there and also the sprint

days was very very interesting i meet so many talented programming python programmers and

it's a very good occasion to speak with other people exchange ideas and

i i think we would be able to do a game yeah i miss it i miss it the sprints at copenhagen was

where i um realized we needed a workshop for new contributors because like i was there trying to

to help people and I was running around different tables I must have you know spoken to 40 50 people

but there's only one of me was so yeah I wasn't able to do that so in um DjangoCon US San Diego

afterwards I did a workshop we had like four tables all together or working and it was brilliant

because people were helping each other so Django Porto was meant to be like the second running of

that workshop but um it obviously didn't happen this year so next year next year but Carlton

you're working with i mean from google we got a season of docs um grant yes so there's work being

done on the contributing section yeah yeah so there's a season of docs project that's going on

that'll be running till march um with gabby precious who's um from nigeria so she's working

on on that now and hopefully that will make the contributing guide more accessible because

you know starting as a fellow i was asking you know you know why don't more people contribute

and the big answer i heard a lot of times was hey the contributing guide is really quite

intimidating like it's it's you know several thousand words long and it's got all the

information you need in there but perhaps it hasn't got a pathway which guides you into that

first contribution as well as or as smoothly as we could so that's what we're hoping to

improve over the next few months there yeah well so we're coming up on time a little bit is

is there anything we haven't asked you about or any projects or things you'd like to talk about

yeah i want to say i think it's interesting uh in the in the company where i work we use only

django i want to say that this year the nobel ps prize winner was the world food program

and we did work in django for the world food program so django django is involved also in

in in these fields and i'm very happy to to say to other people to to make they understand that

open source and free software can can help the humanity in general in various form also

writing code it's a way to to help other people and it's it's a good way to to be in human beings

Super. Yeah. So Django is your Nobel prize winning web framework for this year.

Yeah. Paolo, thanks for coming on. That's super. What a great, what a great chat. Thank you so

much. Thank you all in this particular day to speak with me and for inviting me. It's a very

good experience. Super. Great. Thank you. With Django chat. Join us next time. Bye. Bye. Bye.