Site analysis

Overview

Site analysis (/site-analysis) crawls the URLs from your sitemap and llms.txt configuration, summarizes each page, and produces an architecture-style report. You see topic coverage, thin content clusters, retrieval readiness, content gaps, and intent alignment across your site.

Unlike prompt runs that observe AI answers to specific questions, site analysis evaluates whether your site structure gives models clear, citable material to retrieve.

Who it's for

• Pro and Agency users auditing site-wide readiness for AI retrieval
• Technical SEO and content leads finding thin clusters and structural gaps
• Teams planning a content roadmap informed by how pages are organized, not just individual keywords

Prerequisites

1. A Pro (or Agency) plan—Site analysis is not available on Free.
2. At least one sitemap URL or llms.txt content configured in Sitemaps and llms.txt.
3. (Recommended) Company profile with accurate URL and description.

Step-by-step

Configure crawl sources

1. Add sitemap URLs at /company/sitemaps.
2. Add llms.txt content at /company/llms-txt (optional but recommended for AI discovery files).
3. Ensure URLs resolve correctly—site analysis uses these as the crawl inventory.

Run site analysis

1. Go to Site analysis in the sidebar or navigate to /site-analysis.
2. Review the Run site analysis card—it confirms your sitemap/llms.txt setup.
3. Click Run site analysis.
4. Obsurfable resolves URLs first (shows count), then crawls and analyzes each page.
5. When complete, you're redirected to /site-analysis/[id] with the full report.

Read the report

1. Review high-level findings: topic coverage, thin clusters, and priority areas.
2. Check retrieval readiness signals—whether pages are structured for clear AI extraction.
3. Note content gaps and intent alignment issues across your architecture.
4. Use priorities to decide which clusters to expand or consolidate.

Review past runs

1. Return to /site-analysis to see Past runs with timestamps.
2. Open any previous report to compare before/after a site restructure.
3. Annotate findings for your team's action log.

Connect to other workflows

1. Use findings to inform Query trees—attach better URLs to nodes after fixing gaps.
2. Add or refine company prompts for topics where site coverage is weak.
3. Publish fixes via Content and re-run analysis to validate improvement.

Tips

• Keep sitemaps current—stale sitemaps mean Obsurfable crawls outdated or missing pages.
• llms.txt helps AI crawlers discover priority URLs; include it alongside your sitemap.
• Large sites take longer to crawl; the UI shows URL count before analysis begins.
• Focus on thin clusters first—consolidating weak pages often beats creating more thin content.
• Re-run site analysis after major publishes, migrations, or information architecture changes.

FAQ

Is Site analysis available on the Free plan?

No. Site analysis requires Pro or Agency.

What URLs does Obsurfable crawl?

URLs resolved from your configured sitemap URLs and llms.txt content in Company settings. It does not crawl your entire domain unless those sources list the pages.

I see "No URLs to crawl." What do I do?

Add at least one valid sitemap URL in Sitemaps or content in llms.txt, then return to site analysis.

How is this different from prompt runs?

Prompt runs ask AI how it answers specific buyer questions today. Site analysis evaluates your site's pages and structure for retrieval readiness—regardless of what a single prompt returned.

How often should I run site analysis?

After meaningful site changes: new section launches, major content publishes, migrations, or IA restructuring. Monthly is a reasonable cadence for active sites.

Does site analysis fix my pages?

No—it diagnoses gaps and priorities. Use the report with Queries, Insights, and Content to implement fixes.

Can I annotate site analysis reports?

Yes. Add annotations on report sections—they appear in your central Annotations feed with links back to the report.