Luckily, it provides an Atom feed of those updates. It diffs the crates.io index each minute to know about new crates updates. But, what about new crates? We want to be notified about new crates and be able to send those to Meilisearch.ĭocs.rs is the official website to compute and store the documentation of all the Rust packages hosted by crates.io. Meilisearch now understands that data and gives us instant, relevant and typo tolerant responses. ![]() The Meili dashboard interface showing raw documents Now that we can retrieve useful data, I create an async crawler that downloads, extracts, retrieves the Cargo.toml, and upload the essential data to Meilisearch. Again, the Rust crates.io team was here for us, I talked to Pietro Albini, and he pointed me to the not rate-limited servers that deliver packages content. So, I used the crates.io-index repository to initialize our newly created Meili search engine but needed more data first, like, the description, keywords, and categories of each of those crates. A commit is done each time a crate is updated to a new version, or a version is yanked. Fortunately, this index is available on GitHub in the form of several subfolders with the names and versions of the packages, containing something like 32 000 files. To make Meilisearch exhibit those crates, we needed to find all the currently available packages on crates.io. The Meilisearch server for this demo currently run on a pod in this cluster. So, I asked myself: Why not use our new instant search engine and make it useful to my beloved community? It would give us lots of feedback and probably some pull requests during the process.Īt Meili we manage an internal Kubernetes cluster, this is useful to host demos for clients. Meili is typo-tolerant and supports a whole lot of other features Our search engine uses completely different algorithms it is based on prefix search and is typo tolerant. I decided to run our instant search engine and test its relevancy overtime against these existing solutions. To be honest, I prefer its color design, that's why I took it for our search demo. Kornel Lesinski built lib.rs an alternative to crates.io and uses Tantivy to power its search bar. ![]() Sean Griffin is part of the crates.io team and maintains the current search engine of it along with the whole website. The Meili instant, relevant, and typo-tolerant search engineĬrates.io is the official website that stores Rust community crates (packages), and it's the place where the cargo package manager uploads, updates, and downloads those. Today, I am about to guide you in the depths of crates.io and how I made an alternative search bar using our instant search engine: Meilisearch.
0 Comments
Leave a Reply. |