GoodSpace grew 20× on content alone

I've spent the last couple of days reading GoodSpace, an AI recruitment platform out of India. It started as idle curiosity about a domain I recognised and turned into the most interesting growth story I've looked at all year.

Here's what pulled me in. Their organic traffic:

Sep 2025      4,638
Oct 2025     30,335
Nov 2025     47,827
Dec 2025    223,911
Aug 2026     74,355

Two flat years, then a wall. Ranking keywords went from 954 to 6,030 in the same window. You can't buy that, and you can't link your way there in sixty days. Somebody built a content engine and it worked.

What they built

Four templated silos, all aimed squarely at Indian job-hunting: leave application formats, offer acceptance letters, CTC explainers, interview questions by technology, resume examples by role. The vocabulary is exact. Casual leave. Experience certificate. Freshers.

The traffic split confirms the aim. 93,000 monthly visits from India in English, 18,700 in Hindi, then Philippines, Pakistan, the UK. The US contributes 770, despite the .ai domain.

What makes it impressive is the authority they did it on. Their backlink profile looks large at 33,000 links, but most of it is inherited noise pointing at a different organisation entirely. Strip that out and roughly 120 genuine referring domains remain. They're pulling 74,000 visits a month on a link profile that belongs to a site a tenth their size. The content is carrying all of it.

The gap

Their two best pages after the homepage are the free tools: an ATS resume checker at /premium/ats, and an AI mock interview. Those are where a reader becomes a user.

I checked what links to them.

homepage        6,163 backlinks   1,587 referring domains
/premium/ats        5 backlinks       5 referring domains
every blog post     0                 0

Which shows up in the rankings:

keywordvolumeposition
ats resume18,10047
ats cv checker12,10029
ats resume checker12,10026
resume score9,90022

They have around four thousand pages about resumes and interviews, and none of them link to the tool that scans your resume. The writing and the product are run as separate concerns, so the traffic arrives and then has nowhere obvious to go.

That's a good problem to find. The hard part is already done. What's left is plumbing, and plumbing is buildable.

linkforge

So I built one. linkforge reads a sitemap, fetches pages politely, pulls out every paragraph, and scores each one against each target page. Where a paragraph is genuinely about ATS scanning and already contains a phrase like "applicant tracking system", it proposes wrapping those words in a link.

One restriction does most of the work: it never invents copy. The anchor is always text the author already wrote. Their /guides/ats opens with "An ATS (Applicant Tracking System) is the software recruiters use to collect, parse, and rank job applications." The tool wraps Applicant Tracking System. The sentence stays exactly as written and three words become a link.

That's what makes the output reviewable. You can approve or reject three hundred of these in a sitting, because nothing is being rewritten.

Four real paragraphs from their site, and the four links linkforge proposed for them:

Constraints, and why they're the interesting part

Automated internal linking done badly is worse than none. Repeated exact-match anchors and link-stuffed pages are a recognisable spam pattern, and a tool that produces them at scale is a liability rather than a feature.

So the rules do the real work. Nothing goes inside headings, existing links, code, nav, or tables. No page links to itself. The output is a JSON patch with byte offsets, so a human approves before anything changes. The rest is a single loop over candidates sorted by relevance, and it is short enough to read in full:

for (const c of candidates) {
  if (onThisPage >= maxLinksPerPage) break;                    // 3 per page
  if ((perTarget.get(c.target.url) || 0) >= maxLinksPerTargetPerPage) continue;
  if (usedParagraphs.has(c.paragraph.index)) continue;         // one per paragraph
 
  // Prefer an anchor phrasing we have not leaned on already.
  const anchor = c.anchors.slice().sort(
    (a, b) =>
      (anchorUsage.get(a.text.toLowerCase()) || 0) -
        (anchorUsage.get(b.text.toLowerCase()) || 0) ||
      b.specificity - a.specificity
  )[0];
 
  const used = anchorUsage.get(anchor.text.toLowerCase()) || 0;
 
  // Hard ceiling on any single exact anchor string across the whole site.
  if (used >= 8) {
    skipped.push({ url: page.url, reason: 'anchor-overused', anchor: anchor.text });
    continue;
  }

}

Those four guards are the whole anti-spam story. The sort is the one that surprised me: because a phrasing already used is pushed to the back, the run produces anchor variation on its own, and nobody has to write variation rules.

Scoring is TF-IDF cosine similarity with IDF computed across the whole crawl, so terms that happen to be common on that particular site get down-weighted for free. I considered embeddings and skipped them. Every score here traces back to shared terms, and when a person reviews each row, an explainable score beats a slightly better one.

The bug that taught me something

First run of linkforge against their blog put an ATS link on an article about workplace kindness. The matched text was "Compare the best free ATS resume checkers in 2026", which really is about ATS, so the score was correct.

It was their related-posts widget, rendered as <p> tags. The same teaser appeared on four unrelated pages.

I reached for a structural fix first: detect the container, match the class name. That works on this site and breaks on the next one. Repetition is the more durable signal. Text appearing verbatim across three or more pages is chrome, whatever markup wraps it. That is twelve lines, and it took most of the false positives with it:

export function dropBoilerplate(pages, threshold = 3) {
  const seen = new Map();
  for (const page of pages) {
    for (const key of new Set(page.paragraphs.map((p) => norm(p.text)))) {
      seen.set(key, (seen.get(key) || 0) + 1);
    }
  }
 
  for (const page of pages) {
    page.paragraphs = page.paragraphs.filter(
      (p) => (seen.get(norm(p.text)) || 0) < threshold
    );
  }
}
 
const norm = (s) => s.toLowerCase().replace(/\s+/g, ' ').trim().slice(0, 200);

It counts distinct pages rather than occurrences, so a phrase repeated three times on one long page survives, and a teaser repeated once each on three pages does not.

What it found

linkforge crawled seventy-four pages, one request every 1.2 seconds, robots.txt honoured.

15 insertions across 14 pages
     9  →  /premium/ats
     3  →  /premium/headshot-generator
     3  →  /premium/ai-mock-interview

Eight distinct anchor phrasings came out of it: "Applicant Tracking System", "Resume Scanner", "ATS score", "check your resume", "Mock Interview". Nobody wrote that spread. It falls out of the reuse ceiling.

You can watch the whole run on the demo page, which replays the crawl and shows each proposed link live in its sentence.

Here's one as it would render:

…Which words to put in your resume is a separate job, covered in the ATS resume keywords guide, and running your file through our scanner to get a score…

The sentence is already about the scanner. The link should exist and doesn't.

What I'm not claiming

Fifteen from seventy-four doesn't extrapolate across a sitemap of 1,388 URLs. Guide pages match far better than blog posts, since guides are already about the tools, and the blog run converted at less than half the rate. Better to say that now than have somebody quote my number and find out later.

I also have no performance data, no view of their code, and no idea what's on their roadmap. This is a site read from the outside with public data, and they may have shipped all of it last week.

Where I'd go next

The linking layer is the first piece. After that: a slug pipeline, since their URLs still say 2025 while the titles say 2026 and somebody is fixing that by hand every January. Schema generation from existing content. Quality gates on the programmatic templates.

All of it is the same shape of work, sitting in the seam between a content operation that works and a product surface that deserves the traffic.

The source

Seven files, Node with no dependencies, about six hundred lines all told. These are the files linkforge actually runs, read straight off disk — open one:

cli.jsthe run itself, and what gets written out144 lines
#!/usr/bin/env node
import { readFile, writeFile, mkdir } from 'node:fs/promises';
import { resolve, join } from 'node:path';
import { get, sleep, loadRobots, allowed } from './fetch.js';
import { readSitemap, filterUrls } from './sitemap.js';
import { paragraphs, title, dropBoilerplate } from './extract.js';
import { buildIdf, makeScorer, tokenize } from './match.js';
import { planInsertions } from './plan.js';
import { htmlReport } from './report.js';

const args = parseArgs(process.argv.slice(2));

if (args.help || !args.config) {
  console.log(`
linkforge — proposes internal links from content pages to conversion pages.

  node src/cli.js --config targets.goodspace.json [options]

  --config <file>   target definition (required)
  --limit <n>       max pages to fetch                     (default 50)
  --delay <ms>      delay between requests                 (default 1200)
  --out <dir>       output directory                       (default ./out)
  --min <n>         minimum relevance, overrides config
  --quiet           suppress progress output

Outputs plan.json (exact offsets, machine-applicable) and report.html.
Nothing is ever written to the target site.
`);
  process.exit(args.help ? 0 : 1);
}

const cfg = JSON.parse(await readFile(resolve(args.config), 'utf8'));
const limit = Number(args.limit ?? 50);
const outDir = resolve(args.out ?? './out');
const rules = { ...cfg.rules, ...(args.min ? { minRelevance: Number(args.min) } : {}) };
const log = args.quiet ? () => {} : (...a) => console.error(...a);

const origin = new URL(cfg.site).origin;
log(`\n  ${cfg.site}`);

// 1. robots.txt
const robots = await loadRobots(origin);
const delay = Math.max(Number(args.delay ?? 1200), robots.crawlDelay ?? 0);
log(`  robots: ${robots.disallow.length} disallow rules · ${delay}ms between requests`);

// 2. sitemap
const all = await readSitemap(cfg.sitemap);
if (!all.length) {
  console.error('  ✗ no URLs found in sitemap — check the sitemap URL');
  process.exit(1);
}
const candidates = filterUrls(all, cfg)
  .filter((u) => allowed(new URL(u).pathname, robots))
  .slice(0, limit);

log(`  sitemap: ${all.length} URLs → ${candidates.length} in scope (limit ${limit})\n`);

// 3. fetch, sequentially and slowly
const pages = [];
for (const [n, url] of candidates.entries()) {
  const res = await get(url);
  if (res.ok) {
    const paras = paragraphs(res.body);
    if (paras.length) {
      pages.push({
        url: res.url,
        path: new URL(res.url).pathname,
        title: title(res.body),
        paragraphs: paras,
      });
    }
    log(`  ${String(n + 1).padStart(3)}/${candidates.length}  ${paras.length ? '·' : '∅'}  ${new URL(url).pathname.slice(0, 72)}`);
  } else {
    log(`  ${String(n + 1).padStart(3)}/${candidates.length}  ✗ ${res.status}  ${new URL(url).pathname.slice(0, 72)}`);
  }
  if (n < candidates.length - 1) await sleep(delay);
}

if (!pages.length) {
  console.error('\n  ✗ no readable pages — the site may render content client-side');
  process.exit(1);
}

// 4. strip nav/related-post chrome that survived the markup filters
const boiler = dropBoilerplate(pages);
if (boiler.removed) {
  log(`\n  boilerplate: dropped ${boiler.removed} repeated paragraphs (${boiler.distinctBoilerplate} distinct)`);
}

// 5. score
const corpus = pages.flatMap((p) => p.paragraphs.map((x) => tokenize(x.text)));
const scorer = makeScorer(buildIdf(corpus));

const { insertions, skipped, anchorUsage } = planInsertions({
  pages,
  targets: cfg.targets,
  scorer,
  rules,
  site: cfg.site,
});

// 5. write
await mkdir(outDir, { recursive: true });

const stats = {
  generated: new Date().toISOString().slice(0, 16).replace('T', ' '),
  pagesFetched: pages.length,
  paragraphs: corpus.length,
  pagesAffected: new Set(insertions.map((i) => i.page)).size,
  uniqueAnchors: anchorUsage.size,
};

await writeFile(
  join(outDir, 'plan.json'),
  JSON.stringify({ site: cfg.site, stats, rules, insertions, skipped }, null, 2)
);
await writeFile(
  join(outDir, 'report.html'),
  htmlReport({ site: cfg.site, stats, insertions, targets: cfg.targets })
);

// 6. summary
const tally = new Map();
for (const i of insertions) tally.set(i.target, (tally.get(i.target) || 0) + 1);

log(`\n  ${insertions.length} insertions across ${stats.pagesAffected} pages\n`);
for (const [t, c] of [...tally].sort((a, b) => b[1] - a[1])) {
  log(`    ${String(c).padStart(4)}  →  ${t}`);
}
if (skipped.length) log(`\n  ${skipped.length} skipped (anchor reuse ceiling)`);
log(`\n  ${join(outDir, 'plan.json')}\n  ${join(outDir, 'report.html')}\n`);

function parseArgs(argv) {
  const o = {};
  for (let i = 0; i < argv.length; i++) {
    const a = argv[i];
    if (!a.startsWith('--')) continue;
    const k = a.slice(2);
    const next = argv[i + 1];
    if (!next || next.startsWith('--')) o[k] = true;
    else { o[k] = next; i++; }
  }
  return o;
}
fetch.jspolite HTTP — one at a time, robots.txt, crawl-delay59 lines
// Polite HTTP: one request at a time per host, fixed delay, robots.txt respected.
// Deliberately slow. We are crawling someone else's site uninvited.

const UA = 'linkforge/0.1 (internal-link research; contact via site)';

export async function get(url, { timeout = 15000 } = {}) {
  const ctrl = new AbortController();
  const t = setTimeout(() => ctrl.abort(), timeout);
  try {
    const res = await fetch(url, {
      headers: { 'user-agent': UA, accept: 'text/html,application/xhtml+xml,application/xml' },
      signal: ctrl.signal,
      redirect: 'follow',
    });
    if (!res.ok) return { ok: false, status: res.status, url };
    return { ok: true, status: res.status, url: res.url, body: await res.text() };
  } catch (err) {
    return { ok: false, status: 0, url, error: err.message };
  } finally {
    clearTimeout(t);
  }
}

export const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

/**
 * Minimal robots.txt check. Reads Disallow rules for `*` and our UA.
 * Not a full spec implementation — errs toward refusing.
 */
export async function loadRobots(origin) {
  const res = await get(new URL('/robots.txt', origin).href);
  if (!res.ok) return { disallow: [], crawlDelay: null }; // no robots.txt => no restrictions

  const disallow = [];
  let crawlDelay = null;
  let applies = false;

  for (const raw of res.body.split('\n')) {
    const line = raw.split('#')[0].trim();
    if (!line) continue;
    const [rawKey, ...rest] = line.split(':');
    const key = rawKey.trim().toLowerCase();
    const value = rest.join(':').trim();

    if (key === 'user-agent') {
      applies = value === '*' || value.toLowerCase().includes('linkforge');
    } else if (applies && key === 'disallow' && value) {
      disallow.push(value);
    } else if (applies && key === 'crawl-delay') {
      const n = Number(value);
      if (!Number.isNaN(n)) crawlDelay = n * 1000;
    }
  }
  return { disallow, crawlDelay };
}

export function allowed(pathname, robots) {
  return !robots.disallow.some((rule) => pathname.startsWith(rule));
}
sitemap.jssitemap parsing and path filters47 lines
import { get } from './fetch.js';

const locs = (xml) =>
  [...xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/gi)].map((m) => decodeXml(m[1]));

const decodeXml = (s) =>
  s
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&quot;/g, '"')
    .replace(/&#39;/g, "'")
    .replace(/&amp;/g, '&');

/**
 * Reads a sitemap, following one level of sitemap-index nesting.
 * Returns a de-duplicated list of page URLs.
 */
export async function readSitemap(url, { depth = 0 } = {}) {
  const res = await get(url);
  if (!res.ok) return [];

  const isIndex = /<sitemapindex[\s>]/i.test(res.body);
  const found = locs(res.body);

  if (!isIndex) return found;
  if (depth > 1) return [];

  const out = [];
  for (const child of found.slice(0, 50)) {
    out.push(...(await readSitemap(child, { depth: depth + 1 })));
  }
  return [...new Set(out)];
}

export function filterUrls(urls, { sourceInclude = [], sourceExclude = [] }) {
  return urls.filter((u) => {
    let path;
    try {
      path = new URL(u).pathname;
    } catch {
      return false;
    }
    if (sourceExclude.some((p) => path.includes(p))) return false;
    if (sourceInclude.length && !sourceInclude.some((p) => path.includes(p))) return false;
    return true;
  });
}
extract.jsparagraphs with byte offsets, and the boilerplate check135 lines
// Extracts linkable paragraphs from raw HTML, with byte offsets so a proposed
// insertion can be expressed as an exact, verifiable replacement.

const BLOCKED_CONTAINERS = [
  'script', 'style', 'noscript', 'svg', 'nav', 'header', 'footer',
  'aside', 'pre', 'code', 'form', 'button', 'table', 'figure',
];

/** Character ranges we must never write into. */
function blockedRanges(html) {
  const ranges = [];
  for (const tag of BLOCKED_CONTAINERS) {
    const re = new RegExp(`<${tag}\\b[^>]*>[\\s\\S]*?<\\/${tag}>`, 'gi');
    for (const m of html.matchAll(re)) ranges.push([m.index, m.index + m[0].length]);
  }
  // Headings: never place a link inside one.
  for (const m of html.matchAll(/<h[1-6]\b[^>]*>[\s\S]*?<\/h[1-6]>/gi)) {
    ranges.push([m.index, m.index + m[0].length]);
  }
  return ranges.sort((a, b) => a[0] - b[0]);
}

const inside = (pos, ranges) => ranges.some(([s, e]) => pos >= s && pos < e);

export function title(html) {
  const m = html.match(/<title[^>]*>([\s\S]*?)<\/title>/i);
  return m ? decode(stripTags(m[1])).trim() : '';
}

export function stripTags(s) {
  return s.replace(/<[^>]+>/g, '');
}

export function decode(s) {
  return s
    .replace(/&nbsp;/g, ' ')
    .replace(/&amp;/g, '&')
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&quot;/g, '"')
    .replace(/&#(\d+);/g, (_, d) => String.fromCharCode(+d));
}

/**
 * @returns {Array<{index:number, start:number, end:number, html:string, text:string,
 *                  words:number, safeSpans:Array<[number,number]>}>}
 *   `start`/`end` bound the paragraph's inner HTML within the document.
 *   `safeSpans` are offsets *within the inner HTML* that contain no markup and
 *   no existing anchor — the only places we are willing to insert.
 */
export function paragraphs(html) {
  const blocked = blockedRanges(html);
  const out = [];

  for (const m of html.matchAll(/<p\b[^>]*>([\s\S]*?)<\/p>/gi)) {
    if (inside(m.index, blocked)) continue;

    const innerStart = m.index + m[0].indexOf(m[1]);
    const inner = m[1];
    const text = decode(stripTags(inner)).replace(/\s+/g, ' ').trim();
    const words = text ? text.split(' ').length : 0;
    if (!words) continue;

    out.push({
      index: out.length,
      start: innerStart,
      end: innerStart + inner.length,
      html: inner,
      text,
      words,
      safeSpans: plainTextSpans(inner),
    });
  }
  return out;
}

/**
 * Drops boilerplate: paragraphs whose exact text recurs across several pages.
 *
 * Found this the hard way. Their blog template renders a "related posts" widget
 * as <p> tags, so teaser copy like "Compare the best free ATS resume checkers"
 * appears on pages about workplace kindness — and scored well, because the teaser
 * really is about ATS. A link placed there would sit inside a recommendations
 * carousel rather than prose.
 *
 * Structural detection (class names, containers) is brittle across sites.
 * Repetition is not: text that appears verbatim on N different pages is chrome,
 * whatever markup wraps it.
 *
 * @param {Array<{paragraphs:Array}>} pages  mutated in place
 * @param {number} threshold                 distinct pages before text is chrome
 */
export function dropBoilerplate(pages, threshold = 3) {
  const seen = new Map();
  for (const page of pages) {
    for (const key of new Set(page.paragraphs.map((p) => norm(p.text)))) {
      seen.set(key, (seen.get(key) || 0) + 1);
    }
  }

  let removed = 0;
  for (const page of pages) {
    const kept = page.paragraphs.filter((p) => (seen.get(norm(p.text)) || 0) < threshold);
    removed += page.paragraphs.length - kept.length;
    page.paragraphs = kept;
  }
  return { removed, distinctBoilerplate: [...seen.values()].filter((n) => n >= threshold).length };
}

const norm = (s) => s.toLowerCase().replace(/\s+/g, ' ').trim().slice(0, 200);

/**
 * Spans of the paragraph's inner HTML that are plain text: outside every tag and
 * outside any existing <a>...</a>. Nested links are invalid HTML, so anchors are
 * excluded wholesale rather than trimmed.
 */
function plainTextSpans(inner) {
  const forbidden = [];
  for (const m of inner.matchAll(/<a\b[^>]*>[\s\S]*?<\/a>/gi)) {
    forbidden.push([m.index, m.index + m[0].length]);
  }
  for (const m of inner.matchAll(/<[^>]+>/g)) {
    forbidden.push([m.index, m.index + m[0].length]);
  }
  forbidden.sort((a, b) => a[0] - b[0]);

  const spans = [];
  let cursor = 0;
  for (const [s, e] of forbidden) {
    if (s > cursor) spans.push([cursor, s]);
    cursor = Math.max(cursor, e);
  }
  if (cursor < inner.length) spans.push([cursor, inner.length]);
  return spans.filter(([s, e]) => e - s > 0);
}
match.jsTF-IDF, cosine, and finding a usable anchor phrase91 lines
// TF-IDF relevance between a paragraph and a target page.
// Deterministic and explainable — every score can be traced to shared terms,
// which matters more than raw accuracy when a human reviews each suggestion.

const STOP = new Set(
  `a an and are as at be but by for from has have how i if in into is it its of on or
   that the their then there these they this to was were what when where which who will
   with you your our we can do does not no yes about more most some such than too very
   just also here his her them out up down over under again once only own same so`
    .split(/\s+/)
);

export const tokenize = (s) =>
  s.toLowerCase().replace(/[^a-z0-9\s'-]/g, ' ').split(/\s+/)
    .filter((t) => t.length > 1 && !STOP.has(t));

function tf(tokens) {
  const m = new Map();
  for (const t of tokens) m.set(t, (m.get(t) || 0) + 1);
  return m;
}

/** IDF computed across every paragraph in the crawl. */
export function buildIdf(docTokenLists) {
  const df = new Map();
  for (const tokens of docTokenLists) {
    for (const t of new Set(tokens)) df.set(t, (df.get(t) || 0) + 1);
  }
  const n = docTokenLists.length || 1;
  const idf = new Map();
  for (const [t, d] of df) idf.set(t, Math.log((n + 1) / (d + 1)) + 1);
  return idf;
}

function vector(tokens, idf) {
  const v = new Map();
  for (const [t, count] of tf(tokens)) {
    v.set(t, (1 + Math.log(count)) * (idf.get(t) ?? 1));
  }
  return v;
}

export function cosine(a, b) {
  let dot = 0;
  const [small, large] = a.size < b.size ? [a, b] : [b, a];
  for (const [t, w] of small) {
    const o = large.get(t);
    if (o) dot += w * o;
  }
  if (!dot) return 0;
  const norm = (m) => Math.sqrt([...m.values()].reduce((s, w) => s + w * w, 0));
  return dot / (norm(a) * norm(b) || 1);
}

export function makeScorer(idf) {
  const cache = new Map();
  const vec = (key, tokens) => {
    if (!cache.has(key)) cache.set(key, vector(tokens, idf));
    return cache.get(key);
  };
  return {
    targetVector: (target) =>
      vec(`t:${target.url}`, tokenize(`${target.name} ${target.keywords.join(' ')}`)),
    paragraphVector: (key, text) => vec(`p:${key}`, tokenize(text)),
    cosine,
  };
}

/**
 * Locates every place a target keyword literally appears in the paragraph's
 * plain text, restricted to spans that carry no markup.
 * We wrap words the author already wrote rather than injecting new copy —
 * safer to review, and it keeps anchor text naturally varied.
 */
export function anchorCandidates(paragraph, target) {
  const found = [];
  for (const kw of target.keywords) {
    const re = new RegExp(`(?<![\\w-])${escapeRe(kw)}(?![\\w-])`, 'gi');
    for (const m of paragraph.html.matchAll(re)) {
      const s = m.index;
      const e = s + m[0].length;
      const safe = paragraph.safeSpans.some(([ss, se]) => s >= ss && e <= se);
      if (!safe) continue;
      found.push({ keyword: kw, start: s, end: e, text: m[0], specificity: kw.split(' ').length });
    }
  }
  // Prefer longer, more specific phrases: "ats resume checker" over "resume".
  return found.sort((a, b) => b.specificity - a.specificity || a.start - b.start);
}

const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
plan.jsthe rules, and the patch it emits120 lines
// Turns scored matches into a reviewable set of insertions.
// Every rule here exists because automated internal linking done carelessly is
// worse than none: over-optimised anchors and link-stuffed pages are the exact
// footprint that gets a site discounted.

import { anchorCandidates } from './match.js';

export function planInsertions({ pages, targets, scorer, rules, site }) {
  const {
    maxLinksPerPage = 3,
    maxLinksPerTargetPerPage = 1,
    minRelevance = 0.18,
    minParagraphWords = 25,
    skipFirstParagraphs = 0,
  } = rules;

  const anchorUsage = new Map(); // exact anchor text -> times used across the run
  const targetTally = new Map(); // target url -> links gained
  const insertions = [];
  const skipped = [];

  for (const page of pages) {
    let onThisPage = 0;
    const perTarget = new Map();

    // Score every (paragraph, target) pair, then take the best first.
    const candidates = [];

    for (const p of page.paragraphs) {
      if (p.index < skipFirstParagraphs) continue;
      if (p.words < minParagraphWords) continue;

      for (const target of targets) {
        // Never link a page to itself.
        if (page.path === target.url) continue;

        const rel = scorer.cosine(
          scorer.paragraphVector(`${page.url}#${p.index}`, p.text),
          scorer.targetVector(target)
        );
        if (rel < minRelevance) continue;

        const anchors = anchorCandidates(p, target);
        if (!anchors.length) continue;

        candidates.push({ paragraph: p, target, relevance: rel, anchors });
      }
    }

    candidates.sort((a, b) => b.relevance - a.relevance);

    const usedParagraphs = new Set();

    for (const c of candidates) {
      if (onThisPage >= maxLinksPerPage) break;
      if ((perTarget.get(c.target.url) || 0) >= maxLinksPerTargetPerPage) continue;
      if (usedParagraphs.has(c.paragraph.index)) continue; // one link per paragraph

      // Prefer an anchor phrasing we have not leaned on already.
      const anchor =
        c.anchors.slice().sort(
          (a, b) =>
            (anchorUsage.get(a.text.toLowerCase()) || 0) -
              (anchorUsage.get(b.text.toLowerCase()) || 0) ||
            b.specificity - a.specificity
        )[0];

      const key = anchor.text.toLowerCase();
      const used = anchorUsage.get(key) || 0;

      // Hard ceiling on any single exact anchor string across the whole site.
      if (used >= 8) {
        skipped.push({ url: page.url, reason: 'anchor-overused', anchor: anchor.text });
        continue;
      }

      const href = new URL(c.target.url, site).href;
      const before = c.paragraph.html;
      const after =
        before.slice(0, anchor.start) +
        `<a href="${href}">${anchor.text}</a>` +
        before.slice(anchor.end);

      insertions.push({
        page: page.url,
        pagePath: page.path,
        pageTitle: page.title,
        target: c.target.url,
        targetName: c.target.name,
        href,
        anchorText: anchor.text,
        matchedKeyword: anchor.keyword,
        relevance: Number(c.relevance.toFixed(4)),
        paragraphIndex: c.paragraph.index,
        // Offsets are into the fetched document, so the patch is verifiable.
        docOffsetStart: c.paragraph.start,
        docOffsetEnd: c.paragraph.end,
        original: before,
        replacement: after,
        excerpt: contextLine(c.paragraph.text, anchor.text),
      });

      anchorUsage.set(key, used + 1);
      targetTally.set(c.target.url, (targetTally.get(c.target.url) || 0) + 1);
      perTarget.set(c.target.url, (perTarget.get(c.target.url) || 0) + 1);
      usedParagraphs.add(c.paragraph.index);
      onThisPage++;
    }
  }

  return { insertions, skipped, targetTally, anchorUsage };
}

function contextLine(text, anchor) {
  const i = text.toLowerCase().indexOf(anchor.toLowerCase());
  if (i === -1) return text.slice(0, 160);
  const s = Math.max(0, i - 70);
  const e = Math.min(text.length, i + anchor.length + 70);
  return (s ? '…' : '') + text.slice(s, e) + (e < text.length ? '…' : '');
}
report.jsthe standalone review page79 lines
const esc = (s = '') =>
  String(s).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;');

function mark(excerpt, anchor) {
  const i = excerpt.toLowerCase().indexOf(anchor.toLowerCase());
  if (i === -1) return esc(excerpt);
  return (
    esc(excerpt.slice(0, i)) +
    '<mark>' + esc(excerpt.slice(i, i + anchor.length)) + '</mark>' +
    esc(excerpt.slice(i + anchor.length))
  );
}

export function htmlReport({ site, stats, insertions, targets }) {
  const byTarget = targets.map((t) => ({
    ...t,
    count: insertions.filter((i) => i.target === t.url).length,
  })).sort((a, b) => b.count - a.count);

  return `<title>Internal link plan — ${esc(site)}</title>
<style>
  :root{--bg:#fbfaf8;--fg:#1a1a1a;--muted:#6b6b6b;--line:#e4e1dc;--card:#fff;--accent:#b4421f;
        --mono:ui-monospace,SFMono-Regular,Menlo,monospace}
  body{background:var(--bg);color:var(--fg);font:15px/1.6 ui-sans-serif,-apple-system,"Segoe UI",sans-serif;margin:0}
  .wrap{max-width:900px;margin:0 auto;padding:48px 24px 80px}
  h1{font-size:1.75rem;letter-spacing:-.02em;margin:0 0 4px}
  .sub{color:var(--muted);margin:0}
  .meta{font:12px var(--mono);color:var(--muted);margin-top:12px}
  h2{font-size:1.1rem;margin:40px 0 4px;padding-bottom:8px;border-bottom:1px solid var(--line)}
  .kpi{display:grid;grid-template-columns:repeat(auto-fit,minmax(120px,1fr));gap:10px;margin:24px 0}
  .kpi div{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:12px}
  .kpi b{display:block;font-size:1.5rem;letter-spacing:-.02em}
  .kpi span{font-size:.7rem;color:var(--muted);text-transform:uppercase;letter-spacing:.05em}
  table{width:100%;border-collapse:collapse;font-size:.88rem;margin:12px 0}
  th,td{text-align:left;padding:8px 10px;border-bottom:1px solid var(--line);vertical-align:top}
  th{font-size:.7rem;text-transform:uppercase;letter-spacing:.06em;color:var(--muted)}
  .card{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:16px 18px;margin:12px 0}
  .row{display:flex;justify-content:space-between;gap:12px;flex-wrap:wrap;font:12px var(--mono);color:var(--muted)}
  .row a{color:var(--muted)}
  mark{background:#ffe9a8;padding:1px 2px;border-radius:3px}
  .ex{margin:10px 0 8px;font-size:.95rem}
  .to{font:12px var(--mono)}
  .to b{color:var(--accent);font-weight:600}
  .scroll{overflow-x:auto}
  code{font:12px var(--mono);background:#f1efeb;padding:1px 5px;border-radius:4px}
  .note{background:#fff8e6;border:1px solid #e8d9a0;border-radius:8px;padding:12px 16px;font-size:.88rem;margin:20px 0}
</style>
<div class="wrap">
<h1>Internal link plan</h1>
<p class="sub">${esc(site)} — proposed insertions, none applied.</p>
<p class="meta">${esc(stats.generated)} · ${stats.pagesFetched} pages fetched · ${stats.paragraphs} paragraphs scanned</p>

<div class="note"><b>Proposals only.</b> Every insertion below wraps text the author already wrote — no copy is invented. Review before applying; the JSON patch carries exact document offsets for each change.</div>

<div class="kpi">
  <div><b>${stats.pagesFetched}</b><span>Pages</span></div>
  <div><b>${insertions.length}</b><span>Insertions</span></div>
  <div><b>${stats.pagesAffected}</b><span>Pages changed</span></div>
  <div><b>${stats.uniqueAnchors}</b><span>Distinct anchors</span></div>
</div>

<h2>Links gained per target</h2>
<div class="scroll"><table>
<tr><th>Target</th><th>Path</th><th>New inbound links</th></tr>
${byTarget.map((t) => `<tr><td>${esc(t.name)}</td><td><code>${esc(t.url)}</code></td><td><b>${t.count}</b></td></tr>`).join('\n')}
</table></div>

<h2>Proposed insertions</h2>
${insertions.map((i) => `<div class="card">
  <div class="row">
    <a href="${esc(i.page)}">${esc(i.pagePath)}</a>
    <span>relevance ${i.relevance.toFixed(3)} · ¶${i.paragraphIndex}</span>
  </div>
  <p class="ex">${mark(i.excerpt, i.anchorText)}</p>
  <div class="to">→ <b>${esc(i.target)}</b> &nbsp;·&nbsp; matched <code>${esc(i.matchedKeyword)}</code></div>
</div>`).join('\n')}

</div>`;
}

Everything interesting is a constraint, which is why extract.js and plan.js are the two worth opening. The other five are plumbing: fetch one URL at a time and obey crawl-delay, parse a sitemap, count terms, write an HTML table.

It proposes; nothing is applied, and nothing is ever written to the site being crawled.