When people talk about search engines, they usually focus on ranking.

That is what I expected too. Ranking feels like the core problem. You look at a set of pages and try to decide which ones are most relevant. There are a lot of ideas around this, from link analysis to machine learning to newer semantic approaches.

After spending time working on Slick, I started to see the problem differently.

Ranking only really matters if the data you are working with is good. Getting that data turned out to be harder than I expected.

What I expected to be difficult

Before building anything, I assumed most of the work would be in ranking.

Things like:

• figuring out relevance

• combining different signals

• improving result quality over time

All of that is important. But it also assumes that you already have a solid set of pages to rank.

That assumption does not hold up very well in practice.

Crawling is where the real work starts

Before you can rank anything, you need to collect pages. That means crawling.

At a basic level, crawling sounds simple. You start with some URLs, follow links, and store what you find.

In practice, it becomes a series of decisions:

• how do you discover new pages

• which pages should you crawl first

• how often should you revisit them

• what do you do when a site blocks or slows you down

• how do you avoid spending time on low-value pages

There is also a more practical constraint: speed. As a smaller search engine, I can only crawl so fast. Bandwidth, rate limits, and infrastructure all limit how quickly new pages can be discovered and updated. That means the index is always a bit behind and never fully complete.

Crawling is not just about collecting as much data as possible. It is about deciding what is worth including at all.

Most of what you crawl is not very useful

One thing that stood out quickly was how much low-quality content shows up when you start crawling at scale.

For example:

• pages created mainly for SEO

• duplicate pages with small changes

• very thin pages with little real information

• automatically generated content

If you are not careful, this kind of content fills up your index.

Once it is in your index, it becomes a ranking problem, even though it probably should not have been included in the first place.

Ranking cannot fix a bad index

It is tempting to think you can solve everything with better ranking.

In reality, if your index is full of low-quality or repetitive pages, ranking improvements do not help much. You are still choosing between results that are not very good.

On the other hand, when the index is cleaner, even simpler ranking methods can produce decent results.

This is when it started to feel like the main problem is not just ordering results, but deciding what should be considered at all.

Crawling is really about prioritization

No search engine can crawl the entire web perfectly, especially a smaller one.

That means you are always making tradeoffs:

• do you go deeper into a site or discover new ones

• do you focus on popular pages or less-known ones

• do you aim for freshness or coverage

These choices shape what your search engine knows.

Anything you do not crawl, or decide not to keep, will never appear in results.

What I started doing differently

Working on Slick changed how I approach the problem.

Instead of focusing only on ranking, I started paying more attention to:

• being more selective about what gets indexed

• filtering out low-quality pages earlier in the pipeline

• prioritizing crawl budget toward pages that are more likely to be useful

• accepting that I cannot crawl everything, and designing around that constraint

Even with these changes, the index is still not perfect. There are gaps, outdated pages, and things that slip through. But being more intentional about what gets included has helped improve overall result quality.

Why this matters more for smaller search engines

Large search engines can rely on scale. They can crawl a huge portion of the web and use complex systems to sort it out later.

Smaller search engines do not have that option.

Because of that, the crawl strategy matters more. You cannot rely on indexing everything and fixing it later.

You have to make better decisions earlier in the process.

Closing Thought

I started out thinking ranking was the main challenge in search.

Now it feels more like the challenge is working within limits. You can only crawl so fast, you cannot index everything, and the data you have is always incomplete.

Crawling determines what your search engine knows, but it is also constrained by time and resources. Ranking still matters, but it is operating on an imperfect and constantly changing index.

The goal is not a perfect search engine, but one that makes reasonable decisions given those constraints.