Transcript: Search

I'm Carlton Gibson.

Hi, Will.

So that is a conundrum that there's two stages, I would say, to the journey of search and Django.

If you're like me, my first major Django project, I stumbled through Django, and then at a certain point, this was a school rating search.

So I had 120,000 K through 12 schools in the US, and at some point I wanted a search bar.

And I was like, okay, well, Django will help me out here.

No.

How do I implement basic search?

It's not that it's hard, it's that it requires a pretty deep understanding of Django, which we'll get into.

You have to understand forms and query sets and passing things around.

And then once you get past that hurdle, the second phase is search is actually really, really, really hard.

And everyone expects it to be really good because they use Google or DuckDuckGo all day long.

So Django happens to have now built-in support if you use Postgres for full text search, which we'll get into.

So we'll get into all that, um, starting with the very basics and then advanced options.

But basically, search sounds easy until you've tried it, and then you go hold the phone.

So what's the first step?

Like, you know, you've you've you've you've created a form and your HTML, so you've got search and a button that says search next to it, and you you're maybe the magnifying glass.

The first thing is you need some data.

So you need to we'll start at first principles.

You need to spin up your project, make an app, then you need some data, um, populating the admin or a script to load it in.

So get your models.

And then I would say the way I like to teach this is if you think, think about what it is.

It's a form and it's a a view of some kind.

Basically, it's kind of a list view.

So if you start by saying, okay, I'm just gonna list view, let's say there's 10 10 rows, 10 rows in my database, list all 10.

Okay, now I want to filter them because basically search is a filter.

You're saying of all the things in my database, how do I filter them based on the user query?

So we'll get to the user query, but first with filters, and actually, um, so I would say you, you know, the basic way is use a list view, and then you can use the filter method.

You can use contains, iContains, which is um case insensitive, all get excluded, and you can do some filtering on your list.

And actually, you maintain the Django filter package, right, Carlton?

You want to give that a plug?

So Django filter is great for this exact um use case.

You've got a um a form which then submits something probably via get request.

So query perhaps you in the URL, right?

And you want to first of all validate that those are a saying, and then you want to take a query set and you want to filter it by whatever fields you allowed to, you you're allowed to filter.

So um let's say you've got an address book with some some names, just one single name, like so Adam Johnson, Will Vince and Carl Gibson, Tom Christie.

You know, these are names that you've got in there, and you just want something that matches it.

So you have a car, like a character car filter, a character filter for the filters on strings, and you set it to filter on that name, on that name field, and anything that matches will be returned.

And that's a nice way of filtering down from your address book of 300 to four matches.

So I'm take it slightly different, more abstract, because I was thinking in terms of if I show code to people, but podcasts, I can't show code.

So the first step is you set up your basic form, right?

And then the key thing I think that's really hard to understand is you can use the name field and you can set an input.

Um, which so this is the this is the query itself.

You can call it whatever you want.

Generally, you would call it uh queue.

Yeah.

And then in your, if you're using a list view, you need to modify the query set.

So you can do that just as a field query set, or you can do get query set and have a little more options, and you pass in the queue as the query that you then filter on.

Um I have a post on this which we'll link to, but so you your description was great and correct, but in terms of like the logistics of like, well, how do I get the query?

That's a aha moment for people to see how it's passed from the form to the view uh to be used.

And Django filter does this under the hood, creates a Django form, and then you you you validate the form so that you know it you know it isn't some crazy value that might do harm to your to your database, it's at least going to look like a character field.

And then you get the form the data out, so that comes in in the request.get, you put it into a Django form and you validate it like you should do with all user input ever, you get it out of the form, and then you pass that value to the query set to your filter method of the query set.

Whereas if you passed it through a form field, which is an integer form field, it'll give you back an integer.

So it casts the or a date field, it will give you back a date from a string.

You know, it if you need those kind of um that kind of casting, if you're not dealing just with plain strings, the form can do that transformation for you and can clean that step up.

So that's worth always, always, always, all input from all users at all times through either a Django form or a REST framework serializer or something to sanitize it.

So we'll I'll link to those.

Um but the good news is if you use a Django form, you get most of that out of the box.

So and

Yeah, nice.

Now what?

So you can do these basic filters, um, as I mentioned, you know, and specify on say the name field and the title field, whatever you want.

And you can also chain them together so you can and them.

But sometimes, and definitely you will want to do ORs.

So maybe search by the name of the author or the title.

And for that you need a Q object.

Do you want to take a stab at Q objects?

Because that's a pretty cool, I would say, intermediate advanced um feature object that you use all the time once you know it's there.

A Q you it will when you call filter and you pass in um, I don't know, name equals Adam, right?

And that's that's short for name underscore underscore exact equals Adam because there's an implicit lookup there.

The lookup is exact or contains or i contains any of these lookups.

And the default one is exact.

So if you don't specify one, the the ORM puts that in under for you.

And you pass those in as keyword arguments into the filter method or the exclude method of your query set.

A queue object just lets you create, just take that pair, that lookup, and create it as a queue object, which you can then pass into the the filter method or the exclude method in place of those keyword arguments.

So it takes cues.

Um if you pass in just a keyword argument, it creates, it turns it into a queue under the hood and then processes it after.

But the great thing about queues is they can they can uh you can apply Boolean logic to them.

So they can be AND or ORD or combined in whichever way.

So they're really handy.

It's worth checking out the docs.

We should link to those in the show notes.

Yep, we will.

So uh this is the two ways you can send data.

So a get would uh so both so a post uh bundles the form data, encodes it, and sends it directly to the server.

A get bundles it but puts it into a string that is in the destination URL.

So again, if you do a Google search, go look at the URL and you will see it will be google.

Google.com q equals and a string that matches the query that you made.

So in general, for a search, the basic rule of thumb is if something's gonna touch the database and update it, you should do a post.

Or if it's something that's secure credit card information, we should do a post.

But if it's just a search query, a lot of the times you can get away with just using a get.

Yeah.

So say you do a search and you've you've spent, you know, the Django issue issue tracker is a um a classic example.

There's you know, 1200 open accepted tickets, and there's 48,000 ways of filtering those tickets, and you find you get you, you know, you you learn to filter by component and then you know needs docs, and you get it down, and there's four tickets, and you think, yeah, now I can take on those four tickets, and they're all related.

And I've and then you can just get that URL and you can bookmark it, and you can go back to that search.

And if you want to, it's there.

Whereas to go back and have to rebuild your search from scratch is wow.

But that applies to any.

So I guess But you've

But you're doing exactly the same thing.

You've you're taking the URL with the query parameters already in and you're sharing it with somebody else.

Um and I to me it seems like you would always use a get in um but next time I have this debate with them, maybe I'll share it.

Okay, so what's the next stage?

So it'd be nice to go beyond a query to maybe an entire document, a book, an email, and you can do that with what's called full text search.

So this is a search we're used to.

This is what Google does where you can search what's the technical term is a in Postgres is a document and that can just refer to any kind of body of knowledge.

So what are the options for this?

So there's a couple we'll get into how that works but standard solutions, you've probably heard of Elastic Elasticsearch.

You could use Solar.

Or you could also use there's a third party app, Django Haystack, which actually I used for one of my one of my companies back in the day that connects that's a really nice interface and connects via a driver to solar Elasticsearch or even Whoosh which is built in Python so you can do it on SQLite.

So you can try out cool search without having to get into Postgres SQL and installing that.

So yeah so full tech search so what is it?

Actually before we get what is it it's been in Postgres um since 2008 and in Django since version 1.10 so 2016 um Mark Tamlin led that charge.

Were you a Django fellow in 2016?

Oh what were you involved with that?

Yeah yeah and that was super um you know that so they were uh there were good times like because um Tom Christie had the DRF um Kickstarter the rest framework Kickstarter that really pushed that forward and then there was the Contrib Postgre um Contrib Postgres Kickstarter about the same time.

You know Django was

Because Andrew was talking about some well and I guess it ties into the DSF and

So I mean you know realistically I think we can go um you know we'll get async views in and then the question is well where's the money come for say the ORM rewrite?

That's that's big and there are back end there are async backends and there is the appetite for it but it's a big job and that's gonna need that's gonna need um support somehow.

So it'd be a Kickstarter Mozilla grant whether some one of the big corporations can come in you know I don't know.

ch so for example if someone typed in ran you can say well that matches to run um whereas if you're just doing a straight query there's no intelligence there you can do accent different languages JSON support so all the things you would expect from a decent search which really comes down to relevance um full text search gets you there and it's a really deep and interesting field and the docs on the Postgres site um chapter 12 which we'll link to are fantastic and I'm giving a talk at DjangoCon on this so I'll get into this a

little bit but um

Totally based on what I've read and spoken to people about it is very very performant.

Uh so I would believe that there are cases where Elastic and these other ones are better and they're it might depend on the structure of the data but it is very performant and a lot of folks that I know are setting up using this module it's all built in they don't have to rely on external service and they're very happy and these are large sites.

But that is that would be a good question.

So if you can avoid spinning up that extra dependency you've just got so much more capacity exactly

Yeah I think we're gonna have I we'd like to have someone from I mean there's a number of uh prominent Django people who I think work for Elastic we could have them on and ask them that question yeah it's a good question I would definitely start with someone Postgres search um and then you know see where you go from there.

So so the uh the package in Django has a number of fields that make some things easier.

We'll link to this this is in the docs.

So it is a search vector so you can query against more than one field in your database.

Search vector is a search query so you can add stemming and stop words, search rank so you can get into ranking um and you can I guess the last big one is there's also search vector field where for performance you can add that but um you need to do a manual trigger.

So with all these things it really depends on is the data static or is it dynamic?

So how much is it changing?

With indexing um Postgres has a gist which is faster for dynamic data versus a gin for static data.

And same thing with um just in general with search the question is how fast is your data changing because you don't ever want to actually do a query against the data you want to do a separate process that pre-processes everything and creates the indexes but do you do need to do that every hour, every day every minute trade-offs uh trade-offs they're kind of fun to do but that's why it's hard to give a blanket statement on best way to do it.

It depends on the data and it depends on your needs.

I mean what's interesting is you get like it's a massively complicated topic

the search is tough.

It just really is so you know

Because I mean this is the other factor that at some point um just doing a row scan on your database to see what what can um the see the records which contain the text you're looking for there's it's it's not going to be good enough but it's not gonna be fast enough either.

You need it to be pre-indexed so that you get a result quickly.

You know Google found two million results in 0.04 seconds or whatever.

So we'll link them in the notes but um Paolo uh Malchiori, sorry if I said that wrong has a full text search in Django talk he's given um at I think EuroPython.

Marcus Holderman last year at DjangoCon Europe gave on the lookout for your data a talk.

I know there are some talks by on Django and Elastic in particular that are worth um looking at that we'll link to so it's a deep topic um you know this podcast and my tutorial and the talk I really want to get people up to speed where they can start doing full text search instead of just saying we'll just airdrop you in with a fully configured Django project and of course you know how Forms and views works.

You know I I want to be able to help people ramp up and see how you can progress.

Yeah until you've got thousands of records like many thousands of records a an I contains will nine times out of ten get you everything you need.

Yeah that's a good I wanted

o.

I give an example.

But yeah, it's better to use a as with everything in Django.

If you can rely on the work of millions of smart people, you should do it.

So, all right, that's the highlights for the talk.

If you have any feedback, you can find us at Chat Django on Twitter.

We're at DjangoChat.com site, and we have a newsletter which you're welcome to sign up for if you want regular updates.

So that's all for now.

We'll see you next time.

Bye bye.

Bye.