Add server-side search for visitors without JavaScript

README.md

@@ -51,24 +51,31 @@ Check the URL before rendering each external menu entry. The
 ## Documentation search
 
 **Search documentation** searches titles and page text in the selected language
-with [Pagefind](https://pagefind.app/), which runs in the browser and requires
-JavaScript. Results include highlighted excerpts and are limited to 50 pages.
-The sidebar **Filter** still filters titles only.
-
-The phpbbdocs-hugo build writes one Pagefind bundle per language and section
-to `<lang>/<section>/pagefind/`. Rebuild with `phpbbdocs_hugo.sh` after
-changing content. A build without these bundles still serves pages, but search
-reports that it is unavailable.
-
-The search page checks language and section permissions and gives the browser
-only the bundles for sections the user may read. Bundle files are served through
+with [Pagefind](https://pagefind.app/), which runs in the browser. Results
+include highlighted excerpts. The ACP's **Search results limit** sets how many
+are shown (10-200, default 50). The sidebar **Filter** still filters titles only.
+
+Visitors without JavaScript get server-side results instead: every word must
+match, title matches come first, and the same limit applies. The server also
+answers when a build has no Pagefind bundles. With JavaScript, the search form
+tells the server to skip its own search and leave it to Pagefind.
+
+The phpbbdocs-hugo build writes, for each language and top-level section, a
+Pagefind bundle to `<lang>/<section>/pagefind/` and a server-side index to
+`<lang>/<section>/search-index.json`. Rebuild with `phpbbdocs_hugo.sh` after
+changing content. A build with neither still serves pages, but search reports
+that it is unavailable.
+
+The search page checks language and section permissions, searches only the
+server-side indexes of sections the user may read, and gives the browser only
+those sections' bundles. Bundle files are served through
 `/documentation-search-bundle/{lang}/{section}/{path}`, which repeats those
 checks for every file. Index files named by content hash may stay in the
 browser's private cache for the ACP's **Search index cache time** (default 60
 minutes, 0 turns it off), so a user who loses access to a section can still
 search what they already downloaded until it expires. Separate navigation limits search to the current
 documentation side. Keep raw access to the build directory blocked, including
-its Pagefind bundles, as described in [docs/securing-docs-build.md](docs/securing-docs-build.md).
+its Pagefind bundles and search indexes, as described in [docs/securing-docs-build.md](docs/securing-docs-build.md).
 
 `styles/all/template/js/documentation-search.js` is a copy of phpbbdocs-hugo's
 `site/static/js/documentation-search.js`; keep the two in sync.

acp/main_module.php

@@ -81,6 +81,7 @@ public function main($id, $mode)
 			'phpbbmodders_documentation_nav_icon_enabled' => !empty($config['phpbbmodders_documentation_nav_icon_enabled']),
 			'phpbbmodders_documentation_devdocs_nav_icon_enabled' => !empty($config['phpbbmodders_documentation_devdocs_nav_icon_enabled']),
 			'phpbbmodders_documentation_search_cache_minutes' => isset($config['phpbbmodders_documentation_search_cache_minutes']) ? (string) $config['phpbbmodders_documentation_search_cache_minutes'] : '60',
+			'phpbbmodders_documentation_search_max_results' => (string) $doc_helper->get_search_max_results(),
 		);
 
 		if ($cfg_array['phpbbmodders_documentation_fallback_lang'] === '')
@@ -115,6 +116,7 @@ public function main($id, $mode)
 			$cfg_array['phpbbmodders_documentation_nav_icon_enabled'] = $request->variable('phpbbmodders_documentation_nav_icon_enabled', false);
 			$cfg_array['phpbbmodders_documentation_devdocs_nav_icon_enabled'] = $request->variable('phpbbmodders_documentation_devdocs_nav_icon_enabled', false);
 			$cfg_array['phpbbmodders_documentation_search_cache_minutes'] = trim($request->variable('phpbbmodders_documentation_search_cache_minutes', ''));
+			$cfg_array['phpbbmodders_documentation_search_max_results'] = trim($request->variable('phpbbmodders_documentation_search_max_results', ''));
 
 			// Whole minutes from 0 (no caching) to 24 hours.
 			if (!preg_match('/\A\d{1,4}\z/', $cfg_array['phpbbmodders_documentation_search_cache_minutes'])
@@ -123,9 +125,17 @@ public function main($id, $mode)
 				$error[] = $language->lang('ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES_INVALID', $doc_helper::SEARCH_CACHE_MAX_MINUTES);
 			}
 
+			if (!preg_match('/\A\d{1,3}\z/', $cfg_array['phpbbmodders_documentation_search_max_results'])
+				|| (int) $cfg_array['phpbbmodders_documentation_search_max_results'] < $doc_helper::SEARCH_RESULTS_MIN
+				|| (int) $cfg_array['phpbbmodders_documentation_search_max_results'] > $doc_helper::SEARCH_RESULTS_MAX)
+			{
+				$error[] = $language->lang('ACP_DOCUMENTATION_SEARCH_MAX_RESULTS_INVALID', $doc_helper::SEARCH_RESULTS_MIN, $doc_helper::SEARCH_RESULTS_MAX);
+			}
+
 			if (empty($error))
 			{
 				$cfg_array['phpbbmodders_documentation_search_cache_minutes'] = (int) $cfg_array['phpbbmodders_documentation_search_cache_minutes'];
+				$cfg_array['phpbbmodders_documentation_search_max_results'] = (int) $cfg_array['phpbbmodders_documentation_search_max_results'];
 				foreach ($cfg_array as $key => $value)
 				{
 					$config->set($key, $value);
@@ -175,6 +185,9 @@ public function main($id, $mode)
 			'S_DOCUMENTATION_DEVDOCS_NAV_ICON_ENABLED' => $cfg_array['phpbbmodders_documentation_devdocs_nav_icon_enabled'],
 			'DOCUMENTATION_SEARCH_CACHE_MINUTES' => $cfg_array['phpbbmodders_documentation_search_cache_minutes'],
 			'DOCUMENTATION_SEARCH_CACHE_MAX_MINUTES' => $doc_helper::SEARCH_CACHE_MAX_MINUTES,
+			'DOCUMENTATION_SEARCH_MAX_RESULTS' => $cfg_array['phpbbmodders_documentation_search_max_results'],
+			'DOCUMENTATION_SEARCH_RESULTS_MIN' => $doc_helper::SEARCH_RESULTS_MIN,
+			'DOCUMENTATION_SEARCH_RESULTS_MAX' => $doc_helper::SEARCH_RESULTS_MAX,
 
 			'U_ACTION' => $this->u_action,
 		));

adm/style/acp_documentation.html

@@ -53,6 +53,10 @@ <h3>{{ lang('WARNING') }}</h3>
 			<dt><label for="phpbbmodders_documentation_search_cache_minutes">{{ lang('ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES') ~ lang('COLON') }}</label><br><span>{{ lang('ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES_EXPLAIN') }}</span></dt>
 			<dd><input type="number" id="phpbbmodders_documentation_search_cache_minutes" name="phpbbmodders_documentation_search_cache_minutes" value="{{ DOCUMENTATION_SEARCH_CACHE_MINUTES }}" min="0" max="{{ DOCUMENTATION_SEARCH_CACHE_MAX_MINUTES }}" step="1" class="tiny" required data-doc-tooltip="{{ lang('ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES_EXPLAIN')|e }}"> {{ lang('MINUTES') }}</dd>
 		</dl>
+		<dl>
+			<dt><label for="phpbbmodders_documentation_search_max_results">{{ lang('ACP_DOCUMENTATION_SEARCH_MAX_RESULTS') ~ lang('COLON') }}</label><br><span>{{ lang('ACP_DOCUMENTATION_SEARCH_MAX_RESULTS_EXPLAIN', DOCUMENTATION_SEARCH_RESULTS_MIN, DOCUMENTATION_SEARCH_RESULTS_MAX) }}</span></dt>
+			<dd><input type="number" id="phpbbmodders_documentation_search_max_results" name="phpbbmodders_documentation_search_max_results" value="{{ DOCUMENTATION_SEARCH_MAX_RESULTS|e }}" min="{{ DOCUMENTATION_SEARCH_RESULTS_MIN }}" max="{{ DOCUMENTATION_SEARCH_RESULTS_MAX }}" step="1" class="tiny" required data-doc-tooltip="{{ lang('ACP_DOCUMENTATION_SEARCH_MAX_RESULTS_EXPLAIN', DOCUMENTATION_SEARCH_RESULTS_MIN, DOCUMENTATION_SEARCH_RESULTS_MAX)|e }}"></dd>
+		</dl>
 		<dl>
 			<dt><label for="phpbbmodders_documentation_nav_link">{{ lang('ACP_DOCUMENTATION_NAV_LINK') ~ lang('COLON') }}</label><br><span>{{ lang('ACP_DOCUMENTATION_NAV_LINK_EXPLAIN') }}</span></dt>
 			<dd><label><input type="radio" class="radio" id="phpbbmodders_documentation_nav_link" name="phpbbmodders_documentation_nav_link" value="1"{% if S_DOCUMENTATION_NAV_LINK %} checked="checked"{% endif %}> {{ lang('ENABLED') }}</label>

controller/documentation_controller.php

@@ -26,6 +26,10 @@ class documentation_controller
 	/** Stands in for a page path in the page route given to the search script. */
 	const SEARCH_PATH_PLACEHOLDER = 'documentation-search-path';
 
+	/** Search query length limits, matching the search form's minlength/maxlength. */
+	const SEARCH_QUERY_MIN_LENGTH = 2;
+	const SEARCH_QUERY_MAX_LENGTH = 100;
+
 	/** Image types the image route serves. SVG is left out on purpose. */
 	const IMAGE_MIME_TYPES = array('image/png', 'image/gif', 'image/jpeg', 'image/webp', 'image/avif', 'image/bmp', 'image/x-icon');
 
@@ -157,9 +161,13 @@ public function handle($lang, $path)
 	}
 
 	/**
-	 * Search page. Results come from Pagefind in the browser
-	 * (documentation-search.js); this only passes it the search bundles
-	 * of sections the user may read.
+	 * Search page. With JavaScript, results come from Pagefind in the
+	 * browser (documentation-search.js), which gets only the search
+	 * bundles of sections the user may read. Without JavaScript, or when
+	 * no section has a Pagefind bundle, the same sections' server-side
+	 * indexes are searched here instead. The search form sends js=1 when
+	 * its script runs, so the server search is skipped when Pagefind will
+	 * replace it anyway.
 	 *
 	 * @param string $lang
 	 * @return \Symfony\Component\HttpFoundation\Response
@@ -190,11 +198,24 @@ public function search($lang)
 		{
 			$bundles[] = $this->search_bundle_url($lang, $section);
 		}
-		if (empty($bundles))
+		$index_sections = $this->doc_helper->get_search_index_sections($lang, $sections);
+		if (empty($bundles) && empty($index_sections))
 		{
 			throw new http_exception(503, 'DOCUMENTATION_SEARCH_UNAVAILABLE');
 		}
 		$query = trim(htmlspecialchars_decode($this->request->variable('q', '', true), ENT_QUOTES));
+		$max_results = $this->doc_helper->get_search_max_results();
+		$server_search = !empty($index_sections) && ($this->request->variable('js', 0) !== 1 || empty($bundles));
+		$valid = mb_strlen($query) >= self::SEARCH_QUERY_MIN_LENGTH && mb_strlen($query) <= self::SEARCH_QUERY_MAX_LENGTH;
+		$results = ($server_search && $valid) ? $this->doc_helper->search_pages($lang, $query, $index_sections, $max_results) : array();
+		foreach ($results as $result)
+		{
+			$this->template->assign_block_vars('documentation_search_result', array(
+				'TITLE'   => $result['title'],
+				'EXCERPT' => $result['excerpt'],
+				'U_PAGE'  => $result['url'],
+			));
+		}
 		// Pagefind returns Hugo paths (/<lang>/<path>/); the search script
 		// swaps the placeholder in this page route for each <path>.
 		$page_template = $this->controller_helper->route('phpbbmodders_documentation_page',
@@ -203,8 +224,11 @@ public function search($lang)
 			'U_DOCUMENTATION_SEARCH' => $this->controller_helper->route('phpbbmodders_documentation_search', array('lang' => $lang)),
 			'U_DOCUMENTATION_SEARCH_BACK' => $this->route_for($lang, $scope),
 			'DOCUMENTATION_SEARCH_SCOPE' => $scope,
-			'DOCUMENTATION_SEARCH_QUERY' => mb_substr($query, 0, 100),
+			'DOCUMENTATION_SEARCH_QUERY' => mb_substr($query, 0, self::SEARCH_QUERY_MAX_LENGTH),
 			'DOCUMENTATION_SEARCH_BUNDLES' => json_encode($bundles),
+			'DOCUMENTATION_SEARCH_MAX_RESULTS' => $max_results,
+			'S_DOCUMENTATION_SEARCH_SERVER' => $server_search,
+			'S_DOCUMENTATION_SEARCH_VALID' => $valid,
 			'DOCUMENTATION_SEARCH_LINKS' => json_encode(array(
 				'prefix' => '/' . $lang . '/',
 				'template' => $page_template,

controller/documentation_helper.php

@@ -86,6 +86,21 @@ class documentation_helper
 	/** Upper limit of the ACP search cache time, in minutes (24 hours). */
 	const SEARCH_CACHE_MAX_MINUTES = 1440;
 
+	/** Search result limit: default, and the range the ACP accepts. */
+	const SEARCH_RESULTS_DEFAULT = 50;
+	const SEARCH_RESULTS_MIN = 10;
+	const SEARCH_RESULTS_MAX = 200;
+
+	/** Server-side search index file written by Hugo in each top-level section. */
+	const SEARCH_INDEX_FILE = 'search-index.json';
+
+	/** Largest search index file read, so one oversized file can't exhaust PHP's memory. */
+	const SEARCH_INDEX_MAX_BYTES = 16777216;
+
+	/** Characters of page text shown before the first match in an excerpt, and the excerpt length. */
+	const SEARCH_EXCERPT_LEAD = 60;
+	const SEARCH_EXCERPT_LENGTH = 240;
+
 	/** @var config */
 	protected $config;
 
@@ -772,6 +787,170 @@ public function get_search_bundle_cache_control($path)
 			? 'private, max-age=' . ($minutes * 60) : 'private, no-store';
 	}
 
+	/**
+	 * The ACP search result limit, clamped to SEARCH_RESULTS_MIN..MAX.
+	 *
+	 * @return int
+	 */
+	public function get_search_max_results()
+	{
+		if (!isset($this->config['phpbbmodders_documentation_search_max_results']))
+		{
+			return self::SEARCH_RESULTS_DEFAULT;
+		}
+
+		return min(max((int) $this->config['phpbbmodders_documentation_search_max_results'], self::SEARCH_RESULTS_MIN), self::SEARCH_RESULTS_MAX);
+	}
+
+	/**
+	 * Resolves a section's server-side search index, containment checked
+	 * against <docs_root>/<lang>/<section>/. Callers must check the user's
+	 * language and section permissions first.
+	 *
+	 * @param string $lang
+	 * @param string $section
+	 * @return string|false
+	 */
+	public function get_search_index_file($lang, $section)
+	{
+		if (!in_array($section, $this->get_available_sections($lang), true))
+		{
+			return false;
+		}
+
+		$root = $this->get_docs_root();
+		$section_root = realpath($root . '/' . $lang . '/' . $section);
+		$file = realpath($root . '/' . $lang . '/' . $section . '/' . self::SEARCH_INDEX_FILE);
+
+		return $section_root !== false && strpos($section_root, $root . DIRECTORY_SEPARATOR . $lang . DIRECTORY_SEPARATOR) === 0
+			&& $file !== false && is_file($file) && strpos($file, $section_root . DIRECTORY_SEPARATOR) === 0
+			&& filesize($file) <= self::SEARCH_INDEX_MAX_BYTES
+			? $file : false;
+	}
+
+	/**
+	 * Sections of $lang, limited to $allowed_sections, that have a
+	 * server-side search index.
+	 *
+	 * @param string $lang
+	 * @param array $allowed_sections Sections the current user may read.
+	 * @return array Section slugs, in $allowed_sections order.
+	 */
+	public function get_search_index_sections($lang, array $allowed_sections)
+	{
+		$sections = array();
+		foreach ($allowed_sections as $section)
+		{
+			if ($this->get_search_index_file($lang, $section) !== false)
+			{
+				$sections[] = $section;
+			}
+		}
+
+		return $sections;
+	}
+
+	/**
+	 * Server-side search for visitors without JavaScript. Every word of
+	 * $query must appear in a page's title or text; pages with more title
+	 * matches come first, then by title.
+	 *
+	 * @param string $lang
+	 * @param string $query
+	 * @param array $sections Sections to search; the caller has checked the user may read them.
+	 * @param int $limit Most results returned.
+	 * @return array Each ['title', 'excerpt', 'url'], url being a phpBB route.
+	 */
+	public function search_pages($lang, $query, array $sections, $limit)
+	{
+		$terms = preg_split('/\s+/u', mb_strtolower(trim($query)), -1, PREG_SPLIT_NO_EMPTY);
+		if (empty($terms))
+		{
+			return array();
+		}
+
+		$results = array();
+		foreach ($sections as $section)
+		{
+			$file = $this->get_search_index_file($lang, $section);
+			$pages = $file !== false ? json_decode(file_get_contents($file), true) : null;
+			if (!is_array($pages))
+			{
+				continue;
+			}
+
+			$prefix = '/' . $lang . '/' . $section . '/';
+			foreach ($pages as $page)
+			{
+				if (!is_array($page) || !isset($page['url'], $page['title'], $page['text'])
+					|| !is_string($page['url']) || !is_string($page['title']) || !is_string($page['text'])
+					|| strpos($page['url'], $prefix) !== 0)
+				{
+					continue;
+				}
+
+				$result = $this->match_page($page, $terms);
+				$path = trim(substr($page['url'], strlen('/' . $lang . '/')), '/');
+				if ($result === false || $this->get_content_file($lang, $path) === false)
+				{
+					continue;
+				}
+
+				$result['url'] = $this->controller_helper->route('phpbbmodders_documentation_page', array('lang' => $lang, 'path' => $path));
+				$results[] = $result;
+			}
+		}
+
+		usort($results, function ($a, $b) {
+			return $b['score'] <=> $a['score'] ?: strcmp($a['title'], $b['title']);
+		});
+
+		return array_map(function ($result) {
+			unset($result['score']);
+			return $result;
+		}, array_slice($results, 0, $limit));
+	}
+
+	/**
+	 * @param array $page One search index entry.
+	 * @param array $terms Lowercase search words.
+	 * @return array|false ['title', 'excerpt', 'score'], or false if a word is missing.
+	 */
+	protected function match_page(array $page, array $terms)
+	{
+		$text = trim(preg_replace('/\s+/u', ' ', html_entity_decode($page['text'], ENT_QUOTES | ENT_HTML5, 'UTF-8')));
+		$title = html_entity_decode($page['title'], ENT_QUOTES | ENT_HTML5, 'UTF-8');
+		$title_lower = mb_strtolower($title);
+		$text_lower = mb_strtolower($text);
+		$score = 0;
+		$position = null;
+
+		foreach ($terms as $term)
+		{
+			$in_title = mb_strpos($title_lower, $term) !== false;
+			$in_text = mb_strpos($text_lower, $term);
+			if (!$in_title && $in_text === false)
+			{
+				return false;
+			}
+
+			$score += $in_title ? 2 : 1;
+			if ($in_text !== false && ($position === null || $in_text < $position))
+			{
+				$position = $in_text;
+			}
+		}
+
+		$start = max(0, (int) $position - self::SEARCH_EXCERPT_LEAD);
+		$excerpt = ($start > 0 ? '…' : '') . mb_substr($text, $start, self::SEARCH_EXCERPT_LENGTH);
+		if (mb_strlen($text) > $start + self::SEARCH_EXCERPT_LENGTH)
+		{
+			$excerpt .= '…';
+		}
+
+		return array('title' => $title, 'excerpt' => $excerpt, 'score' => $score);
+	}
+
 	/** @return string|false An image contained in its own language build. */
 	public function get_image_file($lang, $path)
 	{

docs/securing-docs-build.md

@@ -22,8 +22,11 @@ Search bundles are served the same way, through
 own Pagefind bundle, and the route serves a file only when the user may read
 that language and section. Content-hashed index files may be cached
 privately by the browser for the ACP's **Search index cache time** (default 60
-minutes; 0 turns it off). Direct access to the build would expose every
-section's bundle, so the deny rule matters for search too.
+minutes; 0 turns it off). The server-side search indexes
+(`<lang>/<section>/search-index.json`) are never served; the search page reads
+the indexes of sections the user may read. Direct access to the build would
+expose every section's bundle and index, so the deny rule matters for search
+too.
 
 ## Trusting the build
 

language/en/info_acp_documentation.php

@@ -41,6 +41,10 @@
 	'ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES_EXPLAIN' => 'How long each visitor’s browser may keep the search index files it downloads, in minutes. Longer makes repeat searches faster, but after you remove someone’s access to a section, their browser can still search that section’s cached files until they expire. 0 turns caching off.',
 	'ACP_DOCUMENTATION_SEARCH_CACHE_MINUTES_INVALID' => 'Search index cache time must be a whole number of minutes from 0 to %d.',
 
+	'ACP_DOCUMENTATION_SEARCH_MAX_RESULTS'         => 'Search results limit',
+	'ACP_DOCUMENTATION_SEARCH_MAX_RESULTS_EXPLAIN' => 'Most results a documentation search shows, from %1$d to %2$d. Each result loads its own excerpt, so a higher limit makes searches with many matches slower.',
+	'ACP_DOCUMENTATION_SEARCH_MAX_RESULTS_INVALID' => 'Search results limit must be a whole number from %1$d to %2$d.',
+
 	'ACP_DOCUMENTATION_NAV_LINK'         => 'Show “Documentation” in navigation bar',
 	'ACP_DOCUMENTATION_NAV_LINK_EXPLAIN' => 'Add a “Documentation” entry to the forum’s top navigation bar. Disabling this only hides the link — the documentation itself stays reachable at its URL for anyone with permission, so you can link it from your own site menu instead.',