Transcript: Search & GeoDjango - Paolo Melchiorre

I'm Carlton Gibson, joined as ever by Will Vinson.

Hello, Will, how are you doing?

Hi, Carlton.

And today we've got a special guest, uh Paolo Melchiore.

How are you doing, Paolo?

Thanks for having me.

Thanks for coming on super.

So, Paolo, tell the tell us and the listeners um who you are and how you got into Django and programming.

And you know, what's your backstory?

And uh I started programming in high school uh many years ago.

I started with Pascal language, and then uh I haven't any internet connection, so I started studying myself HTML and for the dissertation, the final exam, I I made a website.

I was the first one in my school at the time, and then I decided that web developing is my my life, and I studied computer science at university, and I was very interested in Linux and databases and all this cool stuff.

So I made a dissertation on free software and about licensing uh free software developing, and then I started working for a company uh with Python.

I studied only the classical language in university like C, Java, uh similar one.

But then when I started working with this small company, uh we worked with um Plone.

This is a CMS based on ZOOP, and also I started using Python 2.4 in this in this occasion.

Uh was very good because I started also uh being involved in the buying tone community.

I attended my first conference in Naples.

Uh Plone conference was the international one in 2007.

And so I made internet website for a few years with Plone.

Uh, unfortunately, it it was based on the ZoDB, the Zoe object database.

Okay.

Uh one of the first NoSQL database in the history.

That's kind of cool, right?

I'm just gonna pickle them, and then you get them back, and they're exactly how they were before.

And there's none of this, you know, serializing to the other.

I always think of pickles, I think, danger, danger, danger for security.

Oh, you know, potentially.

So you have forced to translate data in relational database.

And for this motivation, I started using Django.

Uh at one point I changed my company and we started uh exchanging data in industrial uh software.

Uh so I picked Django because it was uh one of the first uh uh framework databases that worked well with uh uh Postgres, so was Django 1.0, I remember.

And another thing strange was that uh Plone was not ready for WSGI.

You can deploy uh Plone on Wasgi at the time, and now I think they updated uh all the infrastructure, but then I started using Django in a Waste GI um environment and uh with the relational database, and I was I've I fell in love with admin and with battery includes uh possibility for Django, and then I found also that you can use very easily Geo Django and Postgres

for uh for GI GIS.

So and then I'm still here using Django after I think a lot of versions.

So this is my my trip from starting it to now.

Yeah, yeah.

So I wanted to ask you about Geo Django because you I saw you gave a great talk at um DjangoCon Europe in Copenhagen on Geo Django.

And we made this uh uh we rebuilt this website for a company in France.

Uh they sell uh all around the world um buildings and house luxury house uh close to the beach.

So the name of the company was Mered Mer.

My friends is not good, so I don't know if the pronunciation is okay.

xt search in the um the description of these houses or around the world and was very interesting because we we did everything in the same environment.

We didn't use any external engines uh to search for location or search for information.

We had also this visited um search and the website.

You can see at the same time the location and how many of every option you can you can find clicking on different filters.

Uh so I I wrote this uh the talk and I spoke at Copenhagen last year was very interesting, and I received also a lot of request of information and how we did this uh this platform uh because a lot of people found that the tutorial in Geo Django documentation was a bit complex.

Uh me too, I found that it was very it present a very uh very complex case study.

And then uh you you can instead uh adding a point field on your model and starting using uh leaflet or starting using the open layer widget in the admin, and it's very very easy.

Uh so you just add like the single field, yeah.

But if you need only a location for your object in the map, it's very easy to add.

You can add point feed.

Of course, you have to use uh uh JIS uh read database like SQLite, Postgres, or also MySQL and Oracle, but it's it's very easy to use.

You uh after migrating things, you can start uh storing uh location and like other text or number you can you really store there.

Okay, perfect.

I mean that and the ORM are the two areas where I'm pretty ignorant.

But of the two, I think Dio Django is would be a little more fun to dive into properly at some point.

What am I gonna do with it?

It's very very easy to to use.

I thought it I thought the creator went to Mapbox and because I've used Mapbox a little bit.

Is it one and the same now or are they still separate?

and uh colors of your map uh uh if you need or you can also use other tiles from other services um we also use the this uh this stack for um the last project I used the uh Postgres and Geojango was for mobile application in Android uh we send location to this mobile pro uh mobile application they send to us uh feedback with another location so

you can use a web a mobile phone and you can store everything in your in your database in your Postgres with no problem at all.

co when Google Maps they severely they tried to charge for their API and so like the company I was at we went from free to tens of thousands of dollars and everyone around us and everyone said no.

And so because Leaflet and Maplox are both based on open street map which is sort of as I believe like Wikipedia for geolocation.

ough to to search for uh addresses um but one one thing i i like uh of this solution is that you don't rely on external services so suddenly for example like in google maps they start uh charging or closing or limiting their services uh you have to change uh you and you don't have all your data in your database instead with Postgres and P

ostgres you store everything on your own and then maybe you can ask for for help or for other additional information externally like uh for addresses uh similar so

ordPress and other PHP framework was based in MySQL then when I started using with Django I I found that there there was uh still in that time um very interesting and powerful uh feature and I started also attending the uh Postgres day here in Italy and uh speaking about how we used the Postgres from the point of view of

um of a user with Django and with uh uh with python and and every time I I try to update my um my Postgres uh searching for new features to use and also full text search was one of them and so I think it's it's a few years ahead from other databases.

Which I liked

I forced to use external services for doing search in mobile application and other other web services um I want to clarify I I don't have anything uh against these services because uh more of them are also open source and based on Lucene uh it's open source uh from the Apache Foundation but I'm I'm an engineer I want to solve problem I don't want to install other

soft software from other person and uh spend my day uh trying to connect solution and uh to find the best configuration I want to solve things writing code so using the Django RM and uh relying on the Postgres full text search functionality I was able to uh concentrate my my effort in doing the best search experience I can

do and uh without having to have more than one uh instances of the external engines uh to solve the configuration and the um uh stacked in the middle from Django RAM and the languages to query in Elastic I was frustrated uh after two or three projects using these uh external services I think other

people can do a better job than mine using that but uh when I found that I was I was able to search directly in the database in the Postgres uh I started using hit I uh and then I starting also asking myself why in the Django project website they are using uh Elasticsearch so uh after studying a bit I proposed to uh the main list to update it and there

was people against and people pros about this uh this update uh but at the end also uh the fellow of the time team said that he he also had troubles updating the version of the Elasticsearch and the libraries to communicate with Elaxis Search and so I proposed a pull request the time was 200 2017 and after a long

uh reviews and updates finally uh we we start using uh um Django and Postgres for text search internally and now it's three years it's it's working so I'm I'm happy.

So every year there is new languages they had.

You can search for text um in new languages, uh also Russian or other uh different types of languages in Postgres, and they had also web search syntaxes and other feature searching alights, or they speed up uh their indexes, and so if you want to use all these new features, you

have to update frequently your Postgres version and also Django version.

Um so I started proposing to update the Django version in the Django project, and if we at the end we will be able to update it, I I want to propose new new features and to the Django uh search because in this year I asked the every time question to user also in conference also last year in Copenhagen and local meetup um for

feedback about the search function in Django project.

And I received a lot of um requests for improving the uh the functionality or to um to change something uh to be more user-friendly.

Uh the second pull request I did in the past for the uh search functionality was the multilingual functionality because uh at with Elastic the only only English worked for a few years, and then with Postgres we were able to add more than 20 languages in the web search.

So I think also to attract more people in the Django ecosystem, the first thing they they they found searching for Django was the the documentation and the search functionality.

So having a better working uh search function is uh our main I don't know, main goal to attract.

So I need something quickly jump over to Django Project.com, quickly type in the search bar.

You know, I need I need so just before we're recording now, I'd be looking up the same site cookie setting.

So quickly type it in, up it comes.

You know, if the quick the better that search works, the better.

I use every day too because uh I use also for searching about full text search because sometimes I I forgot about all the attributes you can use, or it's impossible to remember uh everything, and uh I prefer understanding how things things work and then uh reading the details in the documentation because it's it's very convenient.

So you find out about install fields or JSON fields or array fields from reading the Django docs.

Yeah, yeah.

It was quite a while before um it made its way into Django itself.

Uh I was I remember we're we'll link to your talks and you have a great article.

I gave a talk, was it last year on search, and basically I was just looking at your information nonstop all last year for inspiration.

Um, so it's really excellent.

But I I remember, because I have a little bit less programming experience than both of you, when I came to Django from some other frameworks, the two of the things that jumped out at me is that there wasn't a built-in sign-up author registration.

I had to do it myself, and that search was also lacking uh because a lot of other frameworks have some something or they have um elastic bundled in.

So I agree that ways that we can make it easier and have an out-of-the-box solution.

I mean it's kind of the whole point of that talk and a tutorial I have was showing like the layers of search, and you can quite easily do basic search, uh, but then of course it gets much more complex.

So I think that's a broader thing that's good to have accessible to Django people or people coming to Django who are used to just having built-in search in whatever framework they're coming from.

But I love that you're pushing the boundaries with Postgres itself, because we had on the founder of Listen Notes, which is a site that does uh searching through podcasts, and he's using Django, and he's also using Elastic because he was also familiar with Elastic.

So the fact that you're an expert, but you're also not just porting over, you know, existing expertise, you're push you're still learning and taking advantage of the new functionality that's being added.

That's not always the case, but that's obviously extremely important to have someone doing that and talking about it rather than the confines of an enterprise system where they maybe can't do that.

So they don't need other people uh speaking if uh pro this this solution.

I'm not saying anything against this solution, but I'm only saying that if you need a easy and quick solution to implement a full text search and you are already have Django and Postgres, you don't need anything in advance.

So it's

So why take on the extra difficulty, that extra complexity, that extra ops thing?

Use that first.

Uh for example, if you have uh centralized uh search functionality with Django uh application and I don't know, Java application and uh uh NoSQL database, uh a relational database at the same time.

If you need uh central point to search on, maybe you can think start thinking to a dedicated search engine and it's uh it's convenient in this case, but I work in medium to large project uh in my company, and um usually a Postgres uh is is enough for everything.

So if you have more than one Postgres replicated, you can uh manage millions of records, and there is uh a lot of space, and uh um if you need elastic, it maybe you you have a very big project, otherwise it is not

So if you're trying to index all of those, the updates are a lot, and that was his the reason he cited for needing elastic search was that he needed to handle that ingest rate.

But a lot of sites aren't like that.

But yeah, he's at a crazy scale.

I wanted to ask about so um uh hosted solutions.

Um, so Algolia.

I'm just curious if you've played around with Algolia because that's something I've used a bunch and seen it develop.

I'm just curious if you have any personal experience with with that option.

I use for past project uh for a startup, you use the sour.

Uh but like Elasticsearch is based on Lucine, and I use uh I tried uh uh also Sphinx for another project, but in every in every project there was uh dedicated instances of these services.

In one case is there is uh an external company that managed these uh external uh search engines and we was forced to do all the search on this search engine and all the writes in the database on our databases services and was very frustrating because there was uh synchronizing problem.

And people have this uh delay in uh searching for information and accessing this information because maybe they modify um um a detail of a project of a product, and the the writing in the database was very fast, but then they was forced to wait for the synchronization to the external search engine, and so

the the final user experience was very very frustrating because they are asking where is my data, and we forced them to say you have to wait because the external search engine is very slow or there is a connection problem, and so

I I where did my data go?

Also, my wife usually uh fine for the shopping online.

Uh every time they complain because they found the last one uh shoes in the market when they click when she clicked on uh found that in the detail page the the shoes is it's gone.

Yeah, right.

There's not anymore.

So uh it's a very a very common problem.

It's making sure that on every right to your database, your in your search index is updated.

If you can avoid that problem, yeah, you you know, forever.

Put that off as long as you possibly can because it's a big one.

So anything on a your average website that isn't Google, they're gonna say, well, this is crap.

Um pretty high bar.

You you're also a Django Girls coach, and um there's been a change in the leadership.

I believe we're gonna have one of the new founders on quite soon.

Um but I just wanted to ask you about maybe you could tell people what is Django Girls and your involvement with that community.

Uh for the first time I um I was a coach in the Django Girls um workshop in uh 2017 in the EuroPython, uh and it was a very interesting experience.

Django Girl Workshop was uh one-day workshop for for girls organized all around the world, and they accept girls that want to learn uh how to write Python in Django and working with in web environments, and you can propose yourself like a coach, and during these days, uh you can coach one to four girls and helping uh them to um f

ollowing the steps in the Django Girls tutorial is an online tutorial, it's very easy to use.

You can also use um on yourself, and usually I suggest also to people that want to um um following something more longer than the Django tutorial, and you spend the day um answering for question about uh why Python is not installing good in Windows machine, or I have a very old uh Linux machine, it's not it's not enough, or similar question, it's on it's very wh

So I I waited all day uh asking if they need some help, but at the end I was not able to helping them because they are already very very skilled.

So I I updated the tutorial to Python 3.6 in the meantime to spend the day, but also we organized in Rome and uh um in in Italy.

Uh they are very uh active, they organize in in all city all around Italy, but also all around the world they there is every month's workshop.

Unfortunately, during this period of pandemia, it's difficult to organize in-person workshops.

So I I saw people organizing also online, but I I hope that after this period was gone, we can start again organizing uh Jungle Girls uh workshop days and in conference or during the year because it's a very good yeah,

I've thought about it and I haven't done it.

And certainly once things are normalized, because here in Boston, there's a very active Python community which has regular meetups and and even workshops, and that would probably be a perfect place to you know have a have a number of people who are because the part of the problem with Django, you know, because I teach I teach it for a living, is you have to know a whole bunch of other things before Django makes sense.

So I like I spend a lot of time with okay, you have to install it.

Have you installed Python?

Do you understand web development?

Do you know a little bit about HTML CSS?

There is that issue of what's the bare minimum you need before you can dive into Django.

That so if you're at a Python conference, presumably they have a little bit of Python, right?

But if I just walked into a school, I I can't just probably start teaching Django.

And yeah, uh I I learned a lot trying to explain to this girl out Django uh why Django is work in that way and not in other in other way.

So uh and also you can meet other coach and have uh uh a very good relation in in your city or your region with other very skilled and uh and people that care about the community and also involving girls in community, I think it's a very good thing.

Well, and I'm thinking there's also a Django meetup in Boston, which I think is one of the biggest ones in the world that I don't think we've done a Django girls there.

We should do that.

Because people come in, maybe they don't know anything, or maybe they come from an MVC setup and they find controllers confusing, or um, yeah, it really makes you 360 on what is Django and how is it different.

I think that's something actually I you know, this pandemic will probably continue for a bit.

I want to do some videos to explain just like how do the pieces fit together.

Like, because Carlton has a fantastic talk on middleware, but I think middleware is something that your average Django person doesn't really understand at all.

Um but it could be you know, well, I could just show them Carlton's talk.

But you know, it could be simply explained, you know, peeling back the onion, but it's you know, explaining explaining the interaction of of views, templates, models, and URLs.

That's basically the Django story.

If if you can internalize those four things, um, but that's kind of the Django way, I guess.

So, anyways, yeah, people should volunteer.

It and you learn a lot, you meet great people, as you say.

I hope the next Django Con Europe uh we we'll have a Django workshop too.

Maybe well

Um they put a date.

Yeah.

Um I still have my plane ticket that I haven't gotten refunded for, so I'm ready to I'm ready to go.

Yeah.

I I met also Carton there.

Uh also the sprint days was very very interesting.

I met so many uh talented programming Python programmers, and uh it's a very good uh occasion to to speak with other people, exchange ideas, and I I think we we'd be able to do again.

I miss it.

The sprints of Copenhagen was where I um realized we needed a workshop for new contributors because I I was there trying to help people and I was running around different tables.

I must have you know sp spoken to 40-50 people, but there's only one of me was so yeah, I wasn't able to do that.

So in um Django Con US San Diego afterwards, I I did a workshop where we had like four tables all together all working, and it was brilliant because people were helping each other.

So Jango Porto was meant to be like the second running of that workshop, but um it obviously didn't happen this year.

So next year, next year.

So there's work being done on the contributing section.

So there's a season of docs project that's going on that'll be running till March with Gabby Precious, who's um from Nigeria, so she's working on a on that now, and hopefully that will make the contributing guide more accessible because you know, starting as a fellow, I was asking, you know, well, you know, why don't more people contribute?

And the big answer I heard a lot of times was hey, the contributing guide is really quite intimidating.

Like it's it's you know several thousand words long and it's got all the information you need in there, but perhaps it hasn't got a pathway which guides you into that first contribution as well as or as smoothly as we could.

So that's what we're hoping to improve over the next few months there.

Yeah.

Is is there anything we haven't asked you about or any projects or things you'd like to talk about?

Uh in the in the company where I work, we use only Django.

I want to say that this year the Nobel Peace Prize winner was the World Food Program.

And we did work in Django for the World Food Program.

So Django, Django uh is involved also in in this field, and I'm very happy to uh to say to other people to uh to make they understand that open source and free software can uh can help the humanity in general uh in various forms also writing code is a way to to help other people and it's it's a good way to to be in uh human beings.

So super.

That's super what a great what a great chat.

Thank you so much.

It's a very good experience.

Super thank you.

Bye bye, bye bye bye.