Pagefind is a fully static search library that aims to perform well on large sites, while using as little of your users' bandwidth as possible.
Pagefind runs after any static site generator and automatically indexes the built static files. Pagefind then outputs a static search bundle to your website, and exposes a JavaScript search API that can be used anywhere on your site.
I'm building in my package.json like this:
"scripts": {
"all": "npm run build && npm run postbuild",
"build": "npx @11ty/eleventy",
"postbuild": "npx pagefind --source _site"
},
and then npm run all --serve
I'm doing this locally, in dev mode, at the moment, and here's the output:
> [email protected] all
> npm run build && npm run postbuild
> [email protected] build
> npx @11ty/eleventy
Writing _site/... etc etc
Copied 841 files / Wrote 72 files in 25.28 seconds (351.1ms each, v0.12.1)
> [email protected] postbuild
> npx pagefind --source _site
Need to install the following packages:
pagefind
Ok to proceed? (y) y
> [email protected] postbuild
> npx pagefind --source _site
And it stops like this. I do not see any Pagefind output of indexed and created, and no /_pagefind/ directory is in my _site.
I do have the Pagefind search UI in my search page:
<div id="search"></div>
<script>
window.addEventListener('DOMContentLoaded', (event) => {
new PagefindUI({ element: "#search",
showImages: false,
});
});
</script>
and I just omitted <link href="/_pagefind/pagefind-ui.css" rel="stylesheet">;
while <script src="/_pagefind/pagefind-ui.js"></script> is in my 11ty _includes layout with other JS that work fine.
I've also tried to run separately in my Windows Power Shell npx -y pagefind --source _site --serve but no success:
What am I missing? Thanks!
bug Pagefind CLII don’t know whether this is a host issue (I use GitHub Actions to publish to Cloudflare Pages) or something that’s changed in Pagefind 0.10.6, but I note it here just in case it’s the latter...
Since 0.10.6 came out, I’ve seen a marked increase in exit code 1 failures in the build process. I use two different repos, one for Hugo and one for Eleventy, to build my site as I choose at the time, and have seen this in both. In each case, the GitHub Action has a backup that restarts a failed build, and it seems the backup typically does better with the Pagefind part.
Examples (in each case, the failure followed by the success):
Please advise if I can provide more information. And please don’t waste any holidays-related time on this — it’ll wait. 🎄 👪 🥳
How do you index compressed html files?
I have a large static site with around ~100k html files. My static site generator outputs gzipped compressed html files, which get uploaded to S3. The compression saves a ton of money a month on hosting costs, and every modern browser than auto-detect and depress gzipped html files, so it's a necessary publication format.
However, I'm walking through you getting started docs and npx pagefind is failing when it tries to walk my build directory:
Running Pagefind v0.9.1 (Extended)
Running from: "/var/project"
Source: "build"
Bundle Directory: "_pagefind"
[Walking source directory]
Found 90544 files matching **/*.{html}
[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.
90544 pages found without an <html> element.
Pages without an outer <html> element will not be processed by default.
If adding this element is not possible, use the root selector config to target a different root element.
[Reading languages]
Discovered 0 languages:
[Building search indexes]
Total:
Indexed 0 languages
Indexed 0 pages
Indexed 0 words
Indexed 0 filters
Indexed 0 sorts
Error: Pagefind wasn't able to build an index.
Most likely, the directory passed to Pagefind was empty or did not contain any html files.
So it looks like it's seeing my files but not finding any tags, even though each page definitely contains an tag. Why is this?
My build directory isn't flat and is structured like:
build
category1
subcategory1
index.html
subcategory2
index.html
subcategory3
index.html
...
...
Is it failing to find any files because of my structure, or because of the gzip compression?
Either way, how would I resolve this?
For practicality reasons, I'm unable to change the folder structure nor remove gzip compression.
improvement Pagefind CLIHi, very exciting project!
Trying to install on a hugo website and nothing is happening.
I made a build a my website
hugo
I installed it:
npx pagefind --source "public"
I created a pagefind.yml in the root directory
# pagefind.yml
source: public
bundle_dir: _pagefind
Then when I type
npx pagefind
After a couple seconds...nothing happens.
I tried installing it via npm and then run
npm run
Nothing happens there either.
bug Pagefind CLIConsider the following output, running pagefind --verbose --source _site:
Running Pagefind v0.8.1
Running in verbose mode
Running from: "/home/itamarst/devel/pythonspeed.com"
Source: "_site"
Bundle Directory: "_pagefind"
[Walking source directory]
Found 180 files matching **/*.{html}
[Parsing files]
Did not find a data-pagefind-body element on the site.
↳ Indexing all <body> elements on the site.
5 pages found without an <html> element.
Pages without an outer <html> element will not be processed by default.
...
[Reading languages]
Discovered 2 languages: en-us, unknown
* en-us: 169 pages
* unknown: 1 page
[Building search indexes]
Language en-us:
Indexed 1 page
Indexed 824 words
Indexed 0 filters
Total:
Indexed 1 language
Indexed 1 page
Indexed 824 words
Indexed 0 filters
Finished in 0.250 seconds
It found 170-180 HTML pages it knows how to index, but it only indexes one page? And when I try to search it does indeed seem to only have one page indexed.
bug Pagefind CLIIn Hugo, the goldmark parser renders a footnote reference (i.e., the <sup>’d footnote number itself within text, as opposed to the actual resulting footnote below the text) as, e.g., <sup id="fnref:1">...</sup>. Since neither Hugo nor goldmark makes it possible to edit the HTML of footnote references* — and, thus, there’s no way to specify data-pagefind-ignore for them — it would be nice if one could make Pagefind ignore certain HTML elements and/or item IDs (in the latter case, perhaps any ID that begins with fnref).
[If one wished to exclude the actual footnotes, that also wouldn’t be available unless Pagefind also had the ability to exclude certain CSS classes (<div class="footnotes" role="doc-endnotes">), so I’m also mentioning that FYI.]
* I searched the Hugo Discourse extensively concerning this item, and it appears it’s a long-standing request (at least 2016) deemed as undoable due to issues relating to both goldmark and Commonmark itself.
improvement Pagefind CLII have a prototype search feature for my website using pagefind, https://dotat.at/search.html. It's nice and whizzy, and it fits in well with my Rust static site generator. Thanks for making pagefind!
The only significant problem is that HTML entities in page titles are escaped, so my results page displays them like
2022-04-20 – really divisionless random numbers
Entities in page bodies are not escaped, so if you search (for example) for nbsp, you get a lot of highlighted spaces in the results. This is probably a bug but it isn't a showstopper for me.
I'm trying to enable pagefind with Content-Security-Policy enabled and run into the following error:
pagefind.js:1
Uncaught (in promise) ReferenceError: url is not defined
at Pagefind.loadWasm (pagefind.js:1:12922)
at async Promise.all (/blog/index 0)
at async Pagefind.init (pagefind.js:1:12207)
loadWasm @ pagefind.js:1
await in loadWasm (async)
Pagefind @ pagefind.js:1
(anonymous) @ pagefind.js:1
Hugo config:
server:
headers:
- for: /**
values:
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: document-domain=()
Content-Security-Policy: default-src 'none'; img-src 'self' data:; form-action 'self'; base-uri 'self';
block-all-mixed-content;
style-src 'unsafe-inline' 'self' https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css https://fonts.googleapis.com/css;
font-src https://fonts.gstatic.com/s/roboto/;
script-src 'self' https://cdn.jsdelivr.net/npm/[email protected]/dist/js/bootstrap.bundle.min.js https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js;
frame-src https://www.youtube-nocookie.com/ https://player.vimeo.com/;
media-src https://i.ytimg.com https://www.rovid.nl/def/dco/2016/def-dco-20160823-idoa9bivg-web-hd.mp4;
connect-src ws://localhost:1313/livereload 'self';
The connect-src option of the Content-Security-Policy is set to self which permits the script to connect (without this option you would get a policy error).
** copied the /public/_pagefind directory in my static Hugo folder for testing
This error is also live on: https://pkic.org/blog/
When running without Content-Security-Policy using the build in --serve option, the search runs fine:
hugo; ../pagefind --source ./public/ --serve
Let me just say great job on Pagefind's performance.
Although it was difficult for me to index all my pages (took about an hour), the resulting index is very fast to search and looks great.
However, all my search results are showing up with an "undefined" title. How do I correct this?
My html has a valid