1. How search engines work

1️⃣ Crawling

✔ Essence

Search engines continuously visit web pages through crawlers (such as Googlebot) to build a "candidate page pool".

✔ Priority of robots.txt

👉 Distinguish crawl controls from indexing directives:

Mechanism	Purpose
robots.txt	Controls crawling; blocking crawling does not guarantee exclusion from the index
meta robots	Indexing and presentation directives in an HTML page
X-Robots-Tag	Indexing and presentation directives in HTTP headers, including for non-HTML resources

There is no fixed precedence between meta tags and headers; Google applies the more restrictive applicable rule when they conflict. Crawlers must be allowed to fetch a page to read its noindex directive.

✔ robots.txt common misunderstandings

❌ Misunderstanding:

Disallow = Page will not be included

✅Actual:

• Disallow = Do not crawl
• But it may still be indexed (discovered through external links)

👉 Want not to include it at all:

<meta name="robots" content="noindex" />

✔ Crawling frequency (Crawl Budget)

👉 Every website has a “crawl budget”

Influencing factors:

• Website authority (Authority)
• update frequency
• Number of pages
• Server response speed

✔ How to observe?

Using Google Search Console:

Path:

• Settings → Crawl stats

You can see:

• Crawl times per day
• response time
• Crawl file types

🔥 Advanced Tips

👉 Improve crawling efficiency:

• Reasonable internal links (important!)
• Provide sitemap.xml
• Avoid dead links
• SSR provides full HTML

2. Indexing

✔ Core cognition

Search engines don’t “read web pages” directly, but “look at index libraries”

✔ "Inclusion ≠ Ranking" (must be able to speak)

Stage	Meaning
Indexed	The page enters the database
Ranking	Whether to display + Ranking

👉 Example:

• Page included ✔
• But on page 100 ❌

✔ Key factors affecting indexing

1. Content quality

• Is it original?
• Is it valuable?

2. Page parsability

❗ CSR issues:

<div id="root"></div>

👉 Crawlers without JavaScript rendering, or failed renders, may see only shell HTML; rendering-capable search engines such as Google still need the corresponding processing.

3. Deduplication mechanism (very critical)

Search engines will automatically determine:

• Is the content duplicated?
• Do you need to combine weights?

✔ What canonical really does

👉 Tell search engines:

Which URL is the "master version"

<link rel="canonical" href="https://example.com/page" />

🔥 Advanced understanding

canonical ≠ mandatory instruction 👉 It’s a “suggestion”, search engines don’t have to listen to it

3. Ranking

✔ Three core elements of ranking

1️⃣ Keyword Relevance (Relevance)

👉 What are search engines doing?

“Is this page answering a user question?”

Matching dimensions:

• title
• h1
• Text
• Semantics (NLP)

2️⃣Authority

👉 Source:

• Backlinks
• Website history
• Brand trust

✔ Link Juice (weight flow)

👉 Concept:

Page authority is passed through links

✔ Example:

A页面（权重高） → B页面

👉B page weight will increase

✔ Internal link optimization (practical focus)

• Home Page → Core Page
• Links between articles
• Avoid island pages

🔥Advanced point: weight dispersion

A → B
A → C

👉 The weight is "divided"

4. The SEO value of semantic HTML (in-depth version)

✔ Why is semantics important?

Semantic structure helps search engines and assistive technology interpret content; it does not mean search engines analyze only tags and never rendered results.

✔ Recommended structure

<main>
  <article>
    <h1>文章标题</h1>
    <section>内容</section>
  </article>
  <aside>推荐内容</aside>
</main>

✔ Tag semantic role

Tags	SEO role
<main>	Main content
<article>	Independent content block
<nav>	Navigation structure
<aside>	Auxiliary content

✔ Value to crawlers

👉 Help it quickly judge:

• Where is the text
• where is navigation
• Where is advertising

❗ Common mistakes

<div class="title">xxx</div>

👉 Semantic headings express structure more clearly; using a div does not make the content completely unintelligible to search engines.

5. Heading system

✔ Standard rules

• Usually give the page one clear primary <h1>; multiple h1 elements do not automatically prevent indexing
• Hierarchical progression:

h1 → h2 → h3

✔ SEO Essence

👉 Heading is the "content structure tree"

✔ Best Practices

<h1>Next.js SEO 完整指南</h1>
<h2>什么是 SEO</h2>
<h3>定义</h3>

❗ Common pitfalls

• Replace semantics with style ❌
• Judge multiple h1 elements by document structure and accessibility; avoid confusing heading hierarchy

6. Sitemap.xml (site map)

✔ Essence

Tell search engines: What pages do I have?

✔ Example

<url>
  <loc>https://example.com/blog</loc>
  <lastmod>2026-03-01</lastmod>
  <priority>0.8</priority>
</url>

✔ Field explanation

Field	Function
loc	page address
lastmod	update time
priority	Relative priority in the sitemap protocol; ignored by Google

❗ Important knowledge

👉 priority ≠ ranking weight Google ignores priority and changefreq; do not use these values to control Google crawling or ranking.

✔ Practical suggestions

• Automatically generated (Next.js/CMS)
• Automatically refresh when updating

7. Canonical (anti-duplication core mechanism)

✔ Why is it needed?

Avoid:

example.com
example.com?ref=ads
example.com?page=1

👉 Treated as multiple pages → weights dispersed ❌

✔ Standard writing

<link rel="canonical" href="https://example.com" />

✔ Summary of functions

• Combined weights
• avoid duplicate content
• Improve ranking stability

🔥 Advanced scenes

✔ Pagination SEO

?page=1
?page=2

👉 canonical points to:

• Distinct paginated pages normally use self-referencing canonical URLs rather than all pointing to page one

✔Multi-language site

Combine:

• canonical
• hreflang

8. How does SEO work?

• Search engines are mainly divided into three stages: crawling, indexing, and ranking.
• The crawling phase is controlled by robots.txt and crawl budget;
• The core of the indexing phase is content quality and deduplication mechanism (canonical);
• The ranking stage is determined by keyword relevance and weight (link juice).
• At the front-end level, I mainly optimize technical SEO through SSR, semantic HTML, heading structure, sitemap and canonical.