Compare commits

...
Author SHA1 Message Date
Jared Dillard e37f5483e8 Fix double extension in llms.txt URLs when source suffix matches sourcelink suffix 2026-03-22 15:55:35 -07:00
Jared Dillard 10794dce65 Add note to docs about sphinx-llm 2026-03-18 23:56:37 -07:00
Jacob TomlinsonandGitHub f6596dd1ec Update sphinx-llm link to NVIDIA repository (#64) 2026-02-05 20:50:40 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
29c3932488 Bump actions/checkout from 5 to 6 in the all-github-actions group (#62)
Bumps the all-github-actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 5 to 6
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all-github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-08 11:34:16 -08:00
Kayce BasquesandGitHub 1dd8119cfd Don't process includes within code blocks (#58) 2025-12-16 14:30:38 -08:00
Jared Dillard dacabd62b4 update modules to 0.3.0 2025-12-11 12:49:49 -08:00
Jared Dillard 2f8e64cc6d Make more generic 2025-12-11 11:58:35 -08:00
Jared Dillard 71dc28e7d8 update to 0.2.0 2025-12-11 01:57:02 -08:00
Jared DillardandGitHub cbbf4b572b [docs] Use FetchContent to grab CMake modules (#61) 2025-12-11 00:55:00 -08:00
Jared DillardandGitHub 8a1a73512d It's parallel!
Updated footnote reference for sphinx-llm to clarify usage.
2025-12-10 21:58:49 -08:00
Jared Dillard 097f2c1084 add note about sphinx-llm 2025-12-08 14:13:24 -08:00
Jared DillardandGitHub 61f9b38d4d [docs] Demo all format options (#60) 2025-12-08 00:40:59 -08:00
Jared DillardandGitHub 8254002622 [docs] Improve docs wording (#59) 2025-12-07 23:26:27 -08:00
Jared Dillard 5b1b72bafb Change markdown suffix 2025-12-07 22:45:31 -08:00
Jared Dillard 3a4ccca5bb Update highlights 2025-12-06 19:38:54 -08:00
Jared DillardandGitHub 0a3da6b1f9 [docs] Add docs on the CMake workflow (#56) 2025-12-05 18:37:24 -08:00
Jared DillardandGitHub f1a3963a0d [docs] Discuss output formats in the docs (#55) 2025-12-03 13:51:09 -08:00
Jared Dillard c23c6d4468 Add tabs 2025-12-02 22:09:14 -08:00
Jared Dillard db32dc60a7 Improve CMake bits 2025-12-02 16:48:29 -08:00
Jared DillardandGitHub 543efabebb [docs] Add restbuilder builder for single page .rst builds (#53) 2025-12-01 13:34:45 -08:00
Jared Dillard 141e0e29f6 clean up the cmake 2025-10-28 12:26:45 -07:00
Jared DillardandGitHub 1b08b2f362 Add docs on markdown usage (#50) 2025-10-27 20:33:28 -07:00
Jared DillardandGitHub 5bfbb0168f Support customizable URI templates in llms.txt (#48) 2025-10-27 17:41:04 -07:00
Jared DillardandGitHub e2a80faf04 Add CMake support for also building Markdown docs in parallel (#49) 2025-10-26 20:11:09 -07:00
Jared DillardandGitHub b63801bcff Improve _sources directory handling (#47) 2025-10-13 23:53:46 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
c45ebb0369 Bump the all-github-actions group with 2 updates (#45)
Bumps the all-github-actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [actions/setup-python](https://github.com/actions/setup-python).


Updates `actions/checkout` from 4 to 5
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

Updates `actions/setup-python` from 5 to 6
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all-github-actions
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: all-github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-04 18:59:29 -07:00
Jared DillardandGitHub f5dcd15889 Fix optional sphinx dependency (#44) 2025-09-17 10:51:07 -07:00
Jared DillardandGitHub e64e20133a Remove support for singlehtml (#40) 2025-08-29 15:22:32 -07:00
Jared Dillard 52949a952a Update changelog 2025-08-20 16:01:44 -07:00
Jared DillardandGitHub 3d7edbf7d9 Only allow builders that have a _sources directory (#38) 2025-08-20 15:57:47 -07:00
Jared Dillard 7e390546ba Improve docstring 2025-08-18 18:25:58 -07:00
Jared DillardandGitHub 75380589e1 fix syntax 2025-08-09 01:40:01 -07:00
Jared DillardandGitHub 19c224c199 Add contributing to README.md 2025-08-09 01:39:06 -07:00
Jared DillardandGitHub ebd0e13594 Add support for page and block level ignores (#33) 2025-08-09 01:32:13 -07:00
Jared DillardandGitHub b4cab5ab52 Allow configuration for behavior when llms-full.txt is too large (#35) 2025-08-08 22:00:30 -07:00
Jared DillardandGitHub c24f92031c Fix include paths and spacing (#31) 2025-08-05 14:14:51 -07:00
Jared DillardandGitHub 0ee28290db Add source code files to llms-full.txt (#24) 2025-08-04 18:22:57 -07:00
Jared DillardandGitHub 368c349d58 Fix image paths to deployed images (#30) 2025-08-03 20:47:38 -07:00
Jared DillardandGitHub c816fb1ea8 Fix issue when source_suffix equals source_link_suffix (#29) 2025-07-31 16:17:02 -07:00
Jared DillardandGitHub 92f810592e Add conda-forge version to index.rst 2025-07-21 22:29:24 -07:00
Jared DillardandGitHub ab0eb1dd29 Add conda-forge version to README.md 2025-07-21 14:57:47 -07:00
Jared DillardandGitHub 57f716b2f9 fix docs path 2025-07-12 21:32:01 -07:00
Jared DillardandGitHub 236822885e configure docs theme 2025-07-12 21:19:19 -07:00
Jared DillardandGitHub 46c2dec254 Use first paragraph as summary by default (#22) 2025-06-22 21:00:40 -07:00
Jared DillardandGitHub b56d93d265 Support source file suffix detection (#21) 2025-06-22 19:28:54 -07:00
Jared Dillard 5db5395889 add downloads badge to docs 2025-05-20 01:38:14 -07:00
Jared Dillard 04b0657dc5 add more badges 2025-05-20 01:35:40 -07:00
Jared Dillard 935a964c7e bump version to 0.2.3 2025-05-20 01:20:36 -07:00
Jared Dillard 51f6c71de3 update changelog 2025-05-20 01:18:44 -07:00
Jared DillardandGitHub 70defd3996 Remove get_and_resolve_toctree method (#19) 2025-05-20 01:08:40 -07:00
Jared DillardandGitHub 9ae05c6c13 Simplify _sources lookup (#18) 2025-05-20 00:50:06 -07:00
Jared Dillard 5581979cac make a href 2025-05-18 21:24:14 -07:00
Jared Dillard f70f1a26ec add soft transfer 2025-05-18 21:21:34 -07:00
Jared DillardandGitHub ed50138ae4 Update README.md 2025-05-18 21:12:16 -07:00
Jared DillardandGitHub 8f4d2c07c6 Update docs and README (#17)
* Move readme content to index.rst

* Clean up project name

* Add advanced configuration
2025-05-18 21:10:49 -07:00
Jared Dillard da7ee68076 support multi-line summaries 2025-05-18 18:17:30 -07:00
Jared Dillard 8ed31f13c4 strip whitespace from summary 2025-05-18 18:11:20 -07:00
Jared Dillard cc38abc8f2 add example links in docs 2025-05-18 17:57:09 -07:00
Jared Dillard bf368670db install pypi version 2025-05-18 17:51:58 -07:00
Jared Dillard a5cbdf15fa rename rtd config file 2025-05-18 17:36:15 -07:00
Jared DillardandGitHub e661ba3da7 Add sphinx docs (#16) 2025-05-18 17:31:00 -07:00
31 changed files with 4152 additions and 302 deletions
+5 -5
View File
@@ -10,9 +10,9 @@ jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set up Python 3.10
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: "3.10"
- uses: pre-commit/action@v3.0.1
@@ -23,17 +23,17 @@ jobs:
python-version: ['3.9', '3.10', '3.11', '3.12']
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"
pip install -e . --group dev
# - name: Run mypy
# run: |
+18
View File
@@ -0,0 +1,18 @@
version: 2
build:
os: ubuntu-24.04
tools:
python: "3.13"
commands:
- pip install cmake
- pip install -r docs/requirements.txt
- pip install -e .
- cmake --workflow --preset documentation-workflow
# Generate llms.txt variants for demo purposes
- python docs/generate_llms_variants.py build/html
# Copy built documentation to Read the Docs output directory
- mkdir -p $READTHEDOCS_OUTPUT/html
- cp -r build/html/* $READTHEDOCS_OUTPUT/html/
- cp -r build/markdown/* $READTHEDOCS_OUTPUT/html/
- cp -r build/rst/* $READTHEDOCS_OUTPUT/html/
+95
View File
@@ -1,6 +1,101 @@
Changelog
=========
0.7.2
-----
- Fix double extension in llms.txt URLs when source suffix matches sourcelink suffix
`#63 <https://github.com/jdillard/sphinx-llms-txt/issues/63>`_
0.7.1
-----
- Don't process includes within code blocks
0.7.0
-----
- Add :confval:`llms_txt_uri_template` configuration option to control the link behavior in :confval:`llms_txt_filename`.
`#48 <https://github.com/jdillard/sphinx-llms-txt/pull/48>`_
0.6.0
-----
- Improve _sources directory handling
`#47 <https://github.com/jdillard/sphinx-llms-txt/pull/47>`_
0.5.3
-----
- Make sphinx a required dependency since there are imports from Sphinx
`#44 <https://github.com/jdillard/sphinx-llms-txt/pull/44>`_
0.5.2
-----
- Remove support for singlehtml
`#40 <https://github.com/jdillard/sphinx-llms-txt/pull/40>`_
0.5.1
-----
- Only allow builders that have _sources directory
`#38 <https://github.com/jdillard/sphinx-llms-txt/pull/38>`_
0.5.0
-----
- Add :ref:`block_level_ignore` and :ref:`page_level_ignore`
`#33 <https://github.com/jdillard/sphinx-llms-txt/pull/33>`_
- Add :confval:`llms_txt_full_size_policy` configuration option to control behavior when :confval:`llms_txt_full_max_size` is exceeded.
`#35 <https://github.com/jdillard/sphinx-llms-txt/pull/35>`_
0.4.1
-----
- Fix include paths and spacing
`#31 <https://github.com/jdillard/sphinx-llms-txt/pull/31>`_
0.4.0
-----
- Add support for including source code files with :confval:`llms_txt_code_files` and :confval:`llms_txt_code_base_path` configuration options
`#24 <https://github.com/jdillard/sphinx-llms-txt/pull/24>`_
0.3.2
-----
- Fix image paths to deployed images
`#30 <https://github.com/jdillard/sphinx-llms-txt/pull/30>`_
0.3.1
-----
- Fix issue when ``source_suffix`` equals ``source_link_suffix``
`#29 <https://github.com/jdillard/sphinx-llms-txt/pull/29>`_
0.3.0
-----
- Use first paragraph as default for ``llms_txt_summary``
`#22 <https://github.com/jdillard/sphinx-llms-txt/pull/22>`_
0.2.4
-----
- Support source file suffix detection
`#21 <https://github.com/jdillard/sphinx-llms-txt/pull/21>`_
0.2.3
-----
- Remove ``get_and_resolve_toctree`` method
`#19 <https://github.com/jdillard/sphinx-llms-txt/pull/19>`_
- Simplify ``_sources`` lookup
`#18 <https://github.com/jdillard/sphinx-llms-txt/pull/18>`_
- Add sphinx docs
`#16 <https://github.com/jdillard/sphinx-llms-txt/pull/16>`_
0.2.2
-----
+15
View File
@@ -0,0 +1,15 @@
cmake_minimum_required(VERSION 3.15)
project(SphinxDocs VERSION 1.0.0 LANGUAGES NONE)
# Fetch Sphinx CMake modules
include(FetchContent)
FetchContent_Declare(
sphinx_cmake_modules
GIT_REPOSITORY https://github.com/jdillard/sphinx-cmake-modules.git
GIT_TAG main
)
FetchContent_MakeAvailable(sphinx_cmake_modules)
list(APPEND CMAKE_MODULE_PATH "${sphinx_cmake_modules_SOURCE_DIR}/cmake/modules")
# Add documentation
add_subdirectory(docs)
+53
View File
@@ -0,0 +1,53 @@
{
"version": 6,
"configurePresets": [
{
"name": "documentation",
"displayName": "Documentation Build",
"description": "Configure project with documentation environment setup",
"binaryDir": "${sourceDir}/build"
}
],
"buildPresets": [
{
"name": "html",
"displayName": "Build HTML Documentation",
"configurePreset": "documentation",
"targets": ["html"]
},
{
"name": "markdown",
"displayName": "Build Markdown Documentation",
"configurePreset": "documentation",
"targets": ["markdown"]
},
{
"name": "rst",
"displayName": "Build reStructuredText Documentation",
"configurePreset": "documentation",
"targets": ["rst"]
},
{
"name": "docs-parallel",
"displayName": "Build all output formats in parallel",
"configurePreset": "documentation",
"targets": ["html", "markdown", "rst"]
}
],
"workflowPresets": [
{
"name": "documentation-workflow",
"displayName": "Documentation Build Workflow",
"steps": [
{
"type": "configure",
"name": "documentation"
},
{
"type": "build",
"name": "docs-parallel"
}
]
}
]
}
+7 -79
View File
@@ -1,91 +1,19 @@
# Sphinx llms.txt generator
A Sphinx extension that generates a summary `llms.txt` file, written in Markdown, and a single combined documentation `llms-full.txt` file, written in reStructuredText.
A Sphinx extension that generates a summary `llms.txt` file and a single combined documentation `llms-full.txt` file.
[![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg)](https://anaconda.org/conda-forge/sphinx-llms-txt)
[![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt)
[![Parallel Safe](https://img.shields.io/badge/parallel%20safe-true-brightgreen)](#)
## Installation
## Documentation
```bash
pip install sphinx-llms-txt
```
See [sphinx-llms-txt documentation](https://sphinx-llms-txt.readthedocs.io/en/latest/index.html) for installation and configuration instructions.
## Usage
## Contributing
1. Add the extension to your Sphinx configuration (`conf.py`):
```python
extensions = [
'sphinx_llms_txt',
]
```
## Configuration Options
### `llms_txt_full_file`
- **Type**: boolean
- **Default**: `'True'`
- **Description**: Whether to write the single output file
### `llms_txt_full_filename`
- **Type**: string
- **Default**: `'llms-full.txt'`
- **Description**: Name of the single output file
### `llms_txt_full_max_size`
- **Type**: integer or `None`
- **Default**: `None` (no limit)
- **Description**: Sets a maximum line count for `llms_txt_full_filename`.
If exceeded, the file is skipped and a warning is shown, but the build still completes.
### `llms_txt_file`
- **Type**: boolean
- **Default**: `True`
- **Description**: Whether to write the summary information file
### `llms_txt_filename`
- **Type**: string
- **Default**: `llms.txt`
- **Description**: Name of the summary information file
### `llms_txt_directives`
- **Type**: list of strings
- **Default**: `[]`
- **Description**: List of custom directive names to process for path resolution.
### `llms_txt_title`
- **Type**: string or `None`
- **Default**: `None`
- **Description**: Overrides the Sphinx project name as the heading in `llms.txt`.
### `llms_txt_summary`
- **Type**: string or `None`
- **Default**: `None`
- **Description**: Optional, but recommended, summary description for `llms.txt`.
### `llms_txt_exclude`
- **Type**: list of strings
- **Default**: `[]`
- **Description**: A list of pages to ignore (e.g., `["page1", "page_with_*"]`).
## Features
- Creates `llms.txt` and `llms-full.txt`
- Automatically add content from `include` directives
- Resolves relative paths in directives like `image` and `figure` to use full paths
- Ability to add list of custom directives with `llms_txt_directives`
- Optionally, prepend a base URL using Sphinx's `html_baseurl`
- Ability to exclude pages
Pull Requests welcome! See [Contributing](https://sphinx-llms-txt.readthedocs.io/en/latest/contributing.html) for instructions on how best to contribute.
## License
+7
View File
@@ -0,0 +1,7 @@
include(SphinxUtils)
setup_sphinx_environment()
add_sphinx_builder(html)
add_sphinx_builder(markdown)
add_sphinx_builder(rst)
+20
View File
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = SphinxLLMsTxt
SOURCEDIR = source
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""
Generate variant llms.txt files for demo purposes.
Takes the generated llms.txt (with _sources links) and creates:
- llms.txt - default with _sources links (unchanged)
- llms.md.txt - .html.md links
- llms.rst.txt - .rst links
"""
import re
import sys
from pathlib import Path
def get_base_url() -> str:
"""Import base_url from conf.py."""
sys.path.insert(0, str(Path(__file__).parent / "source"))
from conf import html_baseurl # noqa: E402
return html_baseurl
def generate_variants(build_dir: Path) -> None:
"""Generate llms.txt variants from the original file."""
original = build_dir / "llms.txt"
if not original.exists():
print(f"Error: {original} not found")
sys.exit(1)
content = original.read_text()
base_url = get_base_url()
# Pattern to match links like: https://.../_sources/{docname}.rst.txt
link_pattern = re.compile(
rf"({re.escape(base_url)})_sources/([a-zA-Z0-9_/\-]+)\.rst\.txt"
)
# Generate .html.md variant
md_content = link_pattern.sub(r"\1\2.html.md", content)
(build_dir / "llms.md.txt").write_text(md_content)
print(f"Generated: {build_dir / 'llms.md.txt'} (.html.md links)")
# Generate .rst variant
rst_content = link_pattern.sub(r"\1\2.rst", content)
(build_dir / "llms.rst.txt").write_text(rst_content)
print(f"Generated: {build_dir / 'llms.rst.txt'} (.rst links)")
print(f"Kept: {build_dir / 'llms.txt'} (_sources links)")
if __name__ == "__main__":
if len(sys.argv) > 1:
build_dir = Path(sys.argv[1])
else:
build_dir = Path("build/html")
generate_variants(build_dir)
+10
View File
@@ -0,0 +1,10 @@
furo
esbonio
sphinx-contributors
sphinx
sphinx-design
sphinx-llms-txt
sphinx-inline-tabs
sphinxext-opengraph
sphinx-markdown-builder
sphinxcontrib-restbuilder
+475
View File
@@ -0,0 +1,475 @@
Advanced Configuration
======================
This page covers advanced configuration options for the sphinx-llms-txt extension.
.. _customizing_llms_files:
Customizing the LLMs Files
^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, the extension generates two files:
1. ``llms.txt`` - A summary file in Markdown format
2. ``llms-full.txt`` - A complete documentation file in reStructuredText format
You can customize these files in several ways:
.. _changing_filenames:
Changing Filenames
~~~~~~~~~~~~~~~~~~
You can change the default filenames by setting these values in your ``conf.py``:
.. code-block:: python
llms_txt_filename = "custom-summary.txt"
llms_txt_full_filename = "custom-docs.txt"
.. _disabling_file_generation:
Disabling File Generation
~~~~~~~~~~~~~~~~~~~~~~~~~
If you only want one of the files, you can disable generation of the other:
.. code-block:: python
# Disable summary file
llms_txt_file = False
# Disable full documentation file
llms_txt_full_file = False
.. _custom_summary:
Adding a Custom Summary
~~~~~~~~~~~~~~~~~~~~~~~
The summary file can include a custom description of your project:
.. code-block:: python
llms_txt_summary = """
This documentation explains how to use MyProject to build amazing
applications. The project provides a comprehensive API for handling
data processing and visualization.
"""
.. note:: The summary can span multiple lines and will be properly formatted in the output file.
.. _custom_title:
Custom Title
~~~~~~~~~~~~
By default, the project name from Sphinx is used as the title in ``llms.txt``. You can override this:
.. code-block:: python
llms_txt_title = "My Custom Project Documentation"
.. _handling_large_documentation:
Handling Large Documentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For very large documentation sets, generating the full documentation file might exceed reasonable size limits.
You can set a maximum line count and control what happens when that limit is exceeded:
.. code-block:: python
llms_txt_full_max_size = 10000 # Maximum 10,000 lines
llms_txt_full_size_policy = "warn_skip" # Default behavior
The ``llms_txt_full_size_policy`` setting controls both the log level and action taken when the size limit is exceeded.
It uses the format ``"<loglevel>_<action>"``:
**Log levels:**
- ``warn``: Log as a warning (default)
- ``info``: Log as informational message
**Actions:**
- ``skip``: Don't create the file (default)
- ``keep``: Create the file anyway, ignoring the size limit
- ``note``: Create a placeholder file explaining why the full file wasn't generated
.. tip:: Use :ref:`excluding_content` to remove less relevant pages and reduce the file size.
.. _custom_directive_handling:
Custom Directive Handling
^^^^^^^^^^^^^^^^^^^^^^^^^
.. _path_resolution:
Path Resolution
~~~~~~~~~~~~~~~
The extension resolves paths in the common directives ``[ 'image', 'figure']`` by default.
You can add custom directives to this list:
.. code-block:: python
llms_txt_directives = [
"my-custom-image-directive",
"another-directive-with-paths",
]
This ensures that paths in your custom directives are properly resolved in the generated files.
.. _excluding_content:
Excluding Content
^^^^^^^^^^^^^^^^^
There are several ways to exclude content from the generated ``llms-full.txt`` file:
.. _global_exclusion:
Global Page Exclusion
~~~~~~~~~~~~~~~~~~~~~~
You can exclude specific pages from being included in the generated files:
.. code-block:: python
llms_txt_exclude = [
"search", # Exclude the search page
"genindex", # Exclude the index page
"private_*", # Exclude all pages starting with 'private_'
]
This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption.
It can also be used to reduce the size of llms-full.txt.
.. _page_level_ignore:
Page-Level Ignore Metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can exclude individual pages by adding metadata at the top of any reStructuredText file:
.. code-block:: restructuredtext
:llms-txt-ignore: true
Page Title
==========
This entire page will be excluded from llms-full.txt
When this metadata is present, the entire page is skipped during processing.
.. _block_level_ignore:
Block-Level Ignore Directives
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can exclude specific sections within a page using ignore directives:
.. code-block:: restructuredtext
Page Title
==========
This content will be included in llms-full.txt.
.. llms-txt-ignore-start
This content will be excluded from llms-full.txt.
Section To Ignore
-----------------
This entire section and any nested content will be ignored.
.. code-block:: python
# This code block will also be ignored
def ignored_function():
pass
.. llms-txt-ignore-end
This content will be included again.
Block-level ignores can be useful for:
- Removing internal notes or TODOs
- Hiding implementation details while keeping user-facing documentation
.. note::
- Multiple ignore blocks can be used within the same file
- Ignore directives work with any indentation level
.. _including_code_files:
Including Source Code Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can include source code files from your project at the end of :confval:`llms_txt_full_filename`.
Use include/exclude syntax to precisely control which files are included:
.. code-block:: python
llms_txt_code_files = [
"+:src/**/*.py", # Include all Python files in src
"-:src/**/__pycache__/**", # Exclude Python cache files
]
Pattern syntax:
- **+:pattern**: Include files matching the pattern. Processed first to collect matching files.
- **-:pattern**: Exclude files matching the pattern. Applied to filter out unwanted files.
Code files are processed as follows:
- **Glob patterns**: Use standard glob patterns (``*``, ``**``, ``?``) to match files
- **Relative paths**: Patterns are resolved relative to your Sphinx source directory
- **Formatting**: Each file is presented with a title and syntax-highlighted code block
.. _customizing_code_paths:
Customizing Code File Paths
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By default, the extension automatically detects the relative path from your Sphinx source directory to the git root and strips that prefix from displayed file paths. You can customize this behavior:
.. code-block:: python
# Manually specify base path to strip
llms_txt_code_base_path = "../../"
# Disable path stripping entirely
llms_txt_code_base_path = ""
This helps create cleaner, more readable file paths in the generated documentation.
.. _using_html_baseurl:
Using HTML Base URL
^^^^^^^^^^^^^^^^^^^
If you want to include absolute URLs for resources in your documentation, you can use Sphinx's built-in ``html_baseurl`` configuration:
.. code-block:: python
html_baseurl = "https://example.com/docs/"
When this option is set, all resolved paths in directives will be prefixed with this URL, creating absolute paths in the generated files.
.. _customizing_uri_links:
Customizing URI Links in llms.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, the ``llms.txt`` file links to source files in the ``_sources`` directory when available, falling back to HTML pages when sources aren't available.
You can customize this behavior using URI templates with :confval:`llms_txt_uri_template`:
.. code-block:: python
# Default: Link to source files, if _sources exists
llms_txt_uri_template = "{base_url}_sources/{docname}{suffix}{sourcelink_suffix}"
# Default: Link to HTML pages instead, if _sources doesn't exist
llms_txt_uri_template = "{base_url}{docname}.html"
# Manual: Link to a custom markdown build
llms_txt_uri_template = "{base_url}{docname}.md"
.. _available_template_variables:
Available Template Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your URI template can use the following variables:
- ``{base_url}`` - The base URL from ``html_baseurl`` configuration (includes trailing slash)
- ``{docname}`` - The document name (e.g., ``index``, ``guide/intro``)
- ``{suffix}`` - The source file suffix (e.g., ``.rst``, ``.md``) - may be empty if no source file exists
- ``{sourcelink_suffix}`` - The suffix from ``html_sourcelink_suffix`` configuration (e.g., ``.txt``)
.. tip::
Instead of using the default of linking to ``_sources``, you can generate Markdown and/or reStructuredText files from your documentation and link to those in ``llms.txt``.
See :ref:`cmake_workflow` for an example of building both HTML and Markdown and/or reStructuredText in parallel.
Note that ``_sources`` is still needed for ``llms-full.txt`` at this time.
.. _cmake_workflow:
CMake Workflow
^^^^^^^^^^^^^^
This project uses CMake to orchestrate documentation builds across multiple output formats, serving as a simple demo of the functionality.
This approach enables parallel builds and integrates well with CI/CD platforms like Read the Docs.
Building multiple formats allows you to compare what works best for your docs, as well as allows users to choose which format to feed to their LLM.
Use :confval:`llms_txt_uri_template` to configure links to point to your preferred format.
Key Files
~~~~~~~~~
These configuration files serve as a simple example of a Sphinx site hosted on Read The Docs, some modification may be needed.
.. code-block:: text
.
├── .readthedocs.yml
├── CMakeLists.txt
├── CMakePresets.json
└── docs/
└── CMakeLists.txt
Each section below contains a summary of the file's purpose, the full contents of the file, and a table describing key lines that may need modification.
.. dropdown:: .readthedocs.yml
:chevron: down-up
A Read The Docs config file that installs dependencies, then runs the full documentation workflow which builds all output formats in parallel, and copies them into a single deploy location.
.. literalinclude:: ../../.readthedocs.yml
:language: yaml
:lines: 1-9,11,14-
:linenos:
:emphasize-lines: 9, 14-15
.. list-table::
:header-rows: 1
:width: 100%
:widths: 15 85
* - Line
- Description
* - **9**
- Update the path if your requirements file is in a different location
* - **13-14**
- Modify the copy commands for the output formats you deploy
.. dropdown:: CMakeLists.txt
:chevron: down-up
A CMake config file that sets up the project, fetches the shared `sphinx-cmake-modules <https://github.com/jdillard/sphinx-cmake-modules>`_, and includes the docs subdirectory.
.. literalinclude:: ../../CMakeLists.txt
:language: cmake
:linenos:
:emphasize-lines: 9, 15
.. list-table::
:header-rows: 1
:width: 100%
:widths: 15 85
* - Line
- Description
* - **9**
- Update the ``GIT_TAG`` to use a different version or commit hash
* - **15**
- Change if your docs subdirectory has a different location
.. dropdown:: docs/CMakeLists.txt
:chevron: down-up
A CMake config file that includes the `SphinxUtils <https://github.com/jdillard/sphinx-cmake-modules/blob/v0.1.0/SphinxUtils.cmake>`_ module from FetchContent and defines the documentation-specific build targets.
.. literalinclude:: ../CMakeLists.txt
:language: cmake
:linenos:
:emphasize-lines: 5-7
.. list-table::
:header-rows: 1
:width: 100%
:widths: 15 85
* - Line
- Description
* - **5-7**
- Add or remove calls based on which output formats you need
.. dropdown:: CMakePresets.json
:chevron: down-up
Defines presets for configuring and building documentation:
- **Configure Presets:** Sets up the build directory.
- **Build Presets:** Defines build formats individually and all in parallel.
- **Workflow Presets:** Runs the configure preset followed by the parallel build preset.
.. literalinclude:: ../../CMakePresets.json
:language: json
:linenos:
:emphasize-lines: 18-23, 24-29, 34
.. list-table::
:header-rows: 1
:width: 100%
:widths: 15 85
* - Line
- Description
* - **18-23**
- Remove this preset to disable Markdown documentation builds
* - **24-29**
- Remove this preset to disable reStructuredText documentation builds
* - **34**
- Modify the targets list to build only the output formats you need in parallel
Usage
~~~~~
To build documentation locally using CMake:
.. code-block:: console
# Run the full workflow (configure + build all formats)
cmake --workflow --preset documentation-workflow
# Or configure and build separately
cmake --preset documentation
cmake --build --preset html # Build HTML only
cmake --build --preset docs-parallel # Build all formats
.. _integration_examples:
Integration Examples
^^^^^^^^^^^^^^^^^^^^
Complete Configuration Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's a complete example showing multiple :doc:`configuration-values`:
.. code-block:: python
# File names and generation options
llms_txt_filename = "ai-summary.txt"
llms_txt_full_filename = "ai-full-docs.txt"
llms_txt_full_max_size = 50000
llms_txt_full_size_policy = "warn_note"
# Content customization
llms_txt_title = "Project Documentation for AI Assistants"
llms_txt_summary = """
This is a comprehensive documentation set for our project.
It includes API references, usage examples, and tutorials.
"""
llms_txt_uri_template = "{base_url}{docname}.md"
# Path handling
html_baseurl = "https://docs.example.com/"
llms_txt_directives = ["custom-image", "custom-include"]
# Content filtering
llms_txt_exclude = ["search", "genindex", "404", "private_*"]
# Source code inclusion with include/exclude patterns
llms_txt_code_files = [
"+:../../src/**/*.py", # Include Python files
"+:../../config/*.yaml", # Include config files
"-:../../src/**/__pycache__/**", # Exclude cache files
]
llms_txt_code_base_path = "../../"
+1
View File
@@ -0,0 +1 @@
.. include:: ../../CHANGELOG.rst
+115
View File
@@ -0,0 +1,115 @@
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
import re
import subprocess
# -- Project information -----------------------------------------------------
project = "sphinx-llms-txt"
copyright = "Jared Dillard"
author = "Jared Dillard"
llms_txt_code_files = ["+:../../sphinx_llms_txt/*.py"]
llms_txt_summary = """
A Sphinx extension that generates a summary llms.txt file,written in Markdown,
and a single combined documentation llms-full.txt file, written in reStructuredText.
"""
# This doesn't seem to be supported
# rst_file_suffix = ".html.rst"
markdown_file_suffix = ".html.md"
# check if the current commit is tagged as a release (vX.Y.Z)
try:
GIT_TAG_OUTPUT = subprocess.check_output(["git", "tag", "--points-at", "HEAD"])
current_tag = GIT_TAG_OUTPUT.decode().strip()
if re.match(r"^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$", current_tag):
version = current_tag
else:
version = "latest"
except (subprocess.CalledProcessError, FileNotFoundError):
version = "latest"
# The full version, including alpha/beta/rc tags
release = ""
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.intersphinx",
"sphinx_contributors",
"sphinx_llms_txt",
"sphinxcontrib.restbuilder",
"sphinx_inline_tabs",
"sphinx.ext.extlinks",
"sphinx_design",
]
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = "en"
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
intersphinx_mapping = {
"sphinx": ("https://www.sphinx-doc.org/en/master/", None),
}
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "furo"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
"source_repository": "https://github.com/jdillard/sphinx-llms-txt/",
"source_branch": "main",
"source_directory": "docs/source/",
}
html_baseurl = "https://sphinx-llms-txt.readthedocs.org/en/latest/"
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = "SphinxLLMsTxtdoc"
def setup(app):
app.add_object_type(
"confval",
"confval",
objname="configuration value",
indextemplate="pair: %s; configuration value",
)
+123
View File
@@ -0,0 +1,123 @@
Project Configuration Values
============================
.. confval:: llms_txt_full_file
- **Type**: boolean
- **Default**: ``True``
- **Description**: Whether to write the single output file.
See :ref:`disabling_file_generation`.
.. versionadded:: 0.1.0
.. confval:: llms_txt_full_filename
- **Type**: string
- **Default**: ``'llms-full.txt'``
- **Description**: Name of the single output file.
See :ref:`changing_filenames`.
.. versionadded:: 0.1.0
.. confval:: llms_txt_full_max_size
- **Type**: integer or ``None``
- **Default**: ``None`` (no limit)
- **Description**: Sets a maximum line count for ``llms_txt_full_filename``.
Behavior when exceeded is controlled by :confval:`llms_txt_full_size_policy`.
See :ref:`handling_large_documentation`.
.. versionadded:: 0.2.0
.. confval:: llms_txt_full_size_policy
- **Type**: string
- **Default**: ``'warn_skip'``
- **Description**: Controls what happens when :confval:`llms_txt_full_max_size` is exceeded.
Format is ``<loglevel>_<action>``. Log levels: ``warn``, ``info``.
Actions: ``skip``, ``keep``, ``note``.
See :ref:`handling_large_documentation`.
.. versionadded:: 0.5.0
.. confval:: llms_txt_file
- **Type**: boolean
- **Default**: ``True``
- **Description**: Whether to write the summary information file.
See :ref:`disabling_file_generation`.
.. versionadded:: 0.2.0
.. confval:: llms_txt_filename
- **Type**: string
- **Default**: ``llms.txt``
- **Description**: Name of the summary information file.
See :ref:`changing_filenames`.
.. versionadded:: 0.2.0
.. confval:: llms_txt_uri_template
- **Type**: string or ``None``
- **Default**: ``None``
- **Description**: Template string for generating URIs in ``llms.txt``.
See :ref:`customizing_uri_links`.
.. versionadded:: 0.7.0
.. confval:: llms_txt_directives
- **Type**: list of strings
- **Default**: ``[]`` (empty list)
- **Description**: List of custom directive names to process for path resolution.
See :ref:`path_resolution`.
.. versionadded:: 0.1.0
.. confval:: llms_txt_title
- **Type**: string or ``None``
- **Default**: ``None``
- **Description**: Overrides the Sphinx project name as the heading in ``llms.txt``.
See :ref:`custom_title`.
.. versionadded:: 0.2.0
.. confval:: llms_txt_summary
- **Type**: string
- **Default**: The first paragraph in the root document, else an empty string
- **Description**: Optional, but recommended, summary description for ``llms.txt``.
See :ref:`custom_summary`.
.. versionadded:: 0.2.0
.. confval:: llms_txt_exclude
- **Type**: list of strings
- **Default**: ``[]``
- **Description**: A list of pages to ignore using glob patterns.
See :ref:`excluding_content`.
.. versionadded:: 0.2.1
.. confval:: llms_txt_code_files
- **Type**: list of strings
- **Default**: ``[]``
- **Description**: A list of glob patterns that appends source code files to :confval:`llms_txt_full_filename`.
See :ref:`including_code_files`.
.. versionadded:: 0.4.0
.. confval:: llms_txt_code_base_path
- **Type**: string or ``None``
- **Default**: ``None`` (auto-detect from git root)
- **Description**: Base path to strip from code file paths when displaying titles.
When ``None``, automatically detects the relative path from the Sphinx source
directory to the git root and strips that prefix from file paths.
.. versionadded:: 0.4.0
+48
View File
@@ -0,0 +1,48 @@
Contributing
============
You will need to set up a development environment to make and test your changes before submitting them.
Local development
-----------------
#. Clone the `sphinx-llms-txt repository`_.
#. Create and activate a virtual environment:
.. code-block:: console
python3 -m venv .venv
source .venv/bin/activate
#. Install development dependencies:
.. code-block:: console
pip install -e . --group dev
#. Install pre-commit Git hook scripts:
.. code-block:: console
pre-commit install
Testing changes
---------------
Run ``pytest`` before committing changes.
Current contributors
--------------------
Thanks to all who have contributed!
The people that have improved the code:
.. contributors:: jdillard/sphinx-llms-txt
:avatars:
:limit: 100
:exclude: pre-commit-ci[bot],dependabot[bot]
:order: ASC
.. _sphinx-llms-txt repository: https://github.com/jdillard/sphinx-llms-txt
+92
View File
@@ -0,0 +1,92 @@
Getting Started
===============
Installation
------------
Directly install by using:
.. tab:: via pip
.. code-block:: bash
pip install sphinx-llms-txt
.. tab:: via conda:
.. code-block:: bash
conda install -c conda-forge sphinx-llms-txt
Usage
-----
Add the extension to your Sphinx configuration (``conf.py``):
.. code-block:: python
extensions = [
'sphinx_llms_txt',
]
After the HTML finishes building, **sphinx-llms-txt** will output the location of the output files::
sphinx-llms-txt: Created /path/to/_build/html/llms-full.txt with 45 sources and 6879 lines
sphinx-llms-txt: created /path/to/_build/html/llms.txt
.. _choosing-output-format:
Choosing an Output Format
-------------------------
By default, **sphinx-llms-txt** requires no additional configuration and links to raw reStructuredText source files created by the HTML builder.
For optimal LLM support, see the alternative builders below and the :ref:`CMake workflow <cmake_workflow>` for setup.
.. list-table:: Output Format Comparison
:header-rows: 1
:widths: 18 27 27 27
* -
- Default
- Markdown
- reStructuredText
* - **Setup**
- No config
- CMake [#sphinxllm]_
- CMake
* - **Builder**
- Native [#native]_
- `sphinx-markdown-builder`_
- `sphinxcontrib-restbuilder`_
* - **Format**
- Raw reStructuredText source
- Rendered Markdown [#rendered]_
- Rendered reStructuredText [#rendered]_
* - **LLM Readability**
- Good - preserves structure for simple syntax
- Excellent - native LLM format
- Good - Can provide more structured content
* - **Key Advantage**
- Zero setup required
- More compact (less input tokens)
- Can preserve Sphinx semantics
* - **Key Disadvantage**
- Raw directives won't be parsed [#autodoc]_
- Loses structure from complex directives
- Can lose structure from complex directives
* - **llms-full.txt support**
- Supported with above caveats
- Pending `support <https://github.com/liran-funaro/sphinx-markdown-builder/pull/37>`__ [#pending]_
- Pending `support <https://github.com/sphinx-contrib/restbuilder/pull/35>`__ [#pending]_
.. _sphinx-markdown-builder: https://pypi.org/project/sphinx-markdown-builder/
.. _sphinxcontrib-restbuilder: https://pypi.org/project/sphinxcontrib-restbuilder/
.. rubric:: Footnotes
.. [#sphinxllm] See `sphinx-llm <https://github.com/NVIDIA/sphinx-llm>`_ as an alternative for CMake-free Markdown builds.
.. [#native] Uses raw :confval:`_sources/ <sphinx:html_copy_source>` files created by Sphinx's HTML builder with some minor enhancements.
.. [#autodoc] Directives like ``autodoc`` will appear as raw directive syntax rather than the extracted docstrings.
.. [#pending] PRs that add ``llms-full.txt`` concatenation support have yet to be released.
.. [#rendered] Directives are expanded and processed before output, so content like autodoc docstrings will be included.
+64
View File
@@ -0,0 +1,64 @@
Sphinx llms.txt Generator
=========================
A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Markdown, and a single combined documentation ``llms-full.txt`` file, written in reStructuredText.
|PyPI version| |Conda Version| |Downloads| |Parallel Safe| |GitHub Stars|
Demo
----
This Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as an example of the default output format.
Alternative :ref:`output formats <choosing-output-format>` are also available. For example: `Markdown`_ and `reStructuredText`_.
Highlights
----------
**Zero Configuration**
Add the extension to your ``conf.py`` and you're done.
The extension automatically collects your documentation and generates both ``llms.txt`` and ``llms-full.txt`` during your normal Sphinx build.
**Intelligent Content Processing**
Automatically resolves ``include`` directives, transforms relative paths, and handles your documentation structure without manual intervention.
**Customizable When Needed**
Filter content, include source code files, or integrate with alternative output formats like Markdown for even better LLM compatibility.
See :doc:`getting-started` for output format options and :doc:`configuration-values` for all settings.
.. seealso::
For better default output without configuration, see `sphinx-llm <https://github.com/NVIDIA/sphinx-llm>`_ from NVIDIA.
sphinx-llms-txt is best when customized with alternative output formats, content filtering, or source code inclusion.
.. toctree::
:maxdepth: 2
getting-started
advanced-configuration
configuration-values
contributing
changelog
.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt
.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt
.. _Markdown: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.md.txt
.. _reStructuredText: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.rst.txt
.. _Sphinx: http://sphinx-doc.org/
.. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg
:target: https://pypi.python.org/pypi/sphinx-llms-txt
:alt: Latest PyPi Version
.. |Conda Version| image:: https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg
:target: https://anaconda.org/conda-forge/sphinx-llms-txt
:alt: Latest Conda Version
.. |Downloads| image:: https://static.pepy.tech/badge/sphinx-llms-txt/month
:target: https://pepy.tech/project/sphinx-llms-txt
:alt: PyPi Downloads per month
.. |Parallel Safe| image:: https://img.shields.io/badge/parallel%20safe-true-brightgreen
:target: #
:alt: Parallel read/write safe
.. |GitHub Stars| image:: https://img.shields.io/github/stars/jdillard/sphinx-llms-txt?style=social
:target: https://github.com/jdillard/sphinx-llms-txt
:alt: GitHub Repository stars
+6 -2
View File
@@ -26,13 +26,16 @@ classifiers = [
license = {text = "MIT"}
readme = "README.md"
dynamic = ["version"]
dependencies = [
"sphinx",
]
[project.urls]
download = "https://pypi.org/project/sphinx-llms-txt/"
source = "https://github.com/jdillard/sphinx-llms-txt"
changelog = "https://github.com/jdillard/sphinx-llms-txt/blob/master/CHANGELOG.rst"
[project.optional-dependencies]
[dependency-groups]
dev = [
"pytest>=7.0.0",
"black",
@@ -40,12 +43,13 @@ dev = [
"mypy",
"isort",
"pre-commit",
"sphinx",
]
test = [
"pytest>=7.0.0",
]
[tool.setuptools]
packages = ["sphinx_llms_txt"]
[tool.setuptools.dynamic]
version = {attr = "sphinx_llms_txt.__version__"}
+56 -8
View File
@@ -1,5 +1,14 @@
"""
Sphinx extension to create a combined sources file (llms-full.txt)
Sphinx extension that generates llms.txt and llms-full.txt files for LLM consumption.
This extension collects documentation content from Sphinx projects and generates
two output files:
- llms.txt: A concise Markdown summary with project overview and page links
- llms-full.txt: A comprehensive reStructuredText file containing all documentation
content with resolved includes and path references
The extension processes content during the build phase, handles page-level and
block-level ignore directives, and can optionally include source code files.
"""
from typing import Any, Dict
@@ -12,7 +21,7 @@ from .manager import LLMSFullManager
from .processor import DocumentProcessor
from .writer import FileWriter
__version__ = "0.2.2"
__version__ = "0.7.2"
# Export classes needed by tests
__all__ = [
@@ -25,9 +34,21 @@ __all__ = [
# Global manager instance
_manager = LLMSFullManager()
# Store root document first paragraph
_root_first_paragraph = ""
def doctree_resolved(app: Sphinx, doctree, docname: str):
"""Called when a docname has been resolved to a document."""
global _root_first_paragraph
# Check for llms-txt-ignore metadata at the page level
if hasattr(app.env, "metadata") and docname in app.env.metadata:
metadata = app.env.metadata[docname]
if metadata.get("llms-txt-ignore", "").lower() in ("true", "1", "yes"):
_manager.mark_page_ignored(docname)
return
# Extract title from the document
title = None
# findall() returns a generator, convert to list to check if it has elements
@@ -38,6 +59,14 @@ def doctree_resolved(app: Sphinx, doctree, docname: str):
if title:
_manager.update_page_title(docname, title)
# Extract first paragraph from root document
if docname == app.config.master_doc:
for node in doctree.traverse(nodes.paragraph):
first_para = node.astext()
if first_para:
_root_first_paragraph = first_para
break
def build_finished(app: Sphinx, exception):
"""Called when the build is finished."""
@@ -47,17 +76,26 @@ def build_finished(app: Sphinx, exception):
_manager.set_master_doc(app.config.master_doc)
_manager.set_app(app)
# Get the summary - use configured value or extracted first paragraph
summary = app.config.llms_txt_summary
if summary is None:
summary = _root_first_paragraph
# Set up configuration
config = {
"llms_txt_file": app.config.llms_txt_file,
"llms_txt_filename": app.config.llms_txt_filename,
"llms_txt_uri_template": app.config.llms_txt_uri_template,
"llms_txt_title": app.config.llms_txt_title,
"llms_txt_summary": app.config.llms_txt_summary,
"llms_txt_summary": summary,
"llms_txt_full_file": app.config.llms_txt_full_file,
"llms_txt_full_filename": app.config.llms_txt_full_filename,
"llms_txt_full_max_size": app.config.llms_txt_full_max_size,
"llms_txt_full_size_policy": app.config.llms_txt_full_size_policy,
"llms_txt_directives": app.config.llms_txt_directives,
"llms_txt_exclude": app.config.llms_txt_exclude,
"llms_txt_code_files": app.config.llms_txt_code_files,
"llms_txt_code_base_path": app.config.llms_txt_code_base_path,
"html_baseurl": getattr(app.config, "html_baseurl", ""),
}
_manager.set_config(config)
@@ -76,24 +114,34 @@ def build_finished(app: Sphinx, exception):
def setup(app: Sphinx) -> Dict[str, Any]:
"""Set up the Sphinx extension."""
# Add configuration options
app.add_config_value("llms_txt_file", True, "env")
app.add_config_value("llms_txt_filename", "llms.txt", "env")
app.add_config_value("llms_txt_uri_template", None, "env")
app.add_config_value("llms_txt_full_file", True, "env")
app.add_config_value("llms_txt_full_filename", "llms-full.txt", "env")
app.add_config_value("llms_txt_full_max_size", None, "env")
app.add_config_value("llms_txt_full_size_policy", "warn_skip", "env")
app.add_config_value("llms_txt_directives", [], "env")
app.add_config_value("llms_txt_title", None, "env")
app.add_config_value("llms_txt_summary", None, "env")
app.add_config_value("llms_txt_exclude", [], "env")
app.add_config_value("llms_txt_code_files", [], "env")
app.add_config_value("llms_txt_code_base_path", None, "env")
def builder_inited(app):
"""Used to limit what builders are allowed to run the extension."""
allowed_builders = ["html", "dirhtml"]
if hasattr(app, "builder") and app.builder.name in allowed_builders:
# Reset manager and root paragraph for each build
global _manager, _root_first_paragraph
_manager = LLMSFullManager()
_root_first_paragraph = ""
# Connect to Sphinx events
app.connect("doctree-resolved", doctree_resolved)
app.connect("build-finished", build_finished)
# Reset manager for each build
global _manager
_manager = LLMSFullManager()
app.connect("builder-inited", builder_inited)
return {
"version": __version__,
+124 -25
View File
@@ -3,7 +3,7 @@ Document collector module for sphinx-llms-txt.
"""
import fnmatch
from typing import Any, Dict, List
from typing import Any, Dict, List, Tuple
from sphinx.environment import BuildEnvironment
from sphinx.util import logging
@@ -19,6 +19,7 @@ class DocumentCollector:
self.master_doc: str = None
self.env: BuildEnvironment = None
self.config: Dict[str, Any] = {}
self.app = None
def set_master_doc(self, master_doc: str):
"""Set the master document name."""
@@ -37,8 +38,79 @@ class DocumentCollector:
"""Set configuration options."""
self.config = config
def get_page_order(self) -> List[str]:
"""Get the correct page order from the toctree structure."""
def set_app(self, app):
"""Set the Sphinx application reference."""
self.app = app
def _get_source_suffixes(self):
"""Get all valid source file suffixes from Sphinx configuration.
Returns:
list: List of source file suffixes (e.g., ['.rst', '.md', '.txt'])
"""
if not self.app:
return [".rst"] # Default fallback
source_suffix = self.app.config.source_suffix
if isinstance(source_suffix, dict):
return list(source_suffix.keys())
elif isinstance(source_suffix, list):
return source_suffix
else:
return [source_suffix] # String format
def _get_docname_suffix(self, docname: str, sources_dir) -> str:
"""
Determine the source suffix for a given docname by checking which
file exists.
Args:
docname: The document name to check
sources_dir: Path to the _sources directory
Returns:
The source suffix if found, or None if no matching file exists
"""
if not sources_dir or not sources_dir.exists():
return None
# Get the source link suffix from Sphinx config
source_link_suffix = ""
if self.app and hasattr(self.app.config, "html_sourcelink_suffix"):
source_link_suffix = self.app.config.html_sourcelink_suffix
# Handle empty string case specially
if source_link_suffix == "":
source_link_suffix = "" # Keep it empty
elif not source_link_suffix.startswith("."):
source_link_suffix = "." + source_link_suffix
# Get the source file suffixes from Sphinx config
source_suffixes = self._get_source_suffixes()
# Try to find the source file with any of the valid source suffixes
for src_suffix in source_suffixes:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
candidate_file = sources_dir / f"{docname}{src_suffix}"
else:
candidate_file = (
sources_dir / f"{docname}{src_suffix}{source_link_suffix}"
)
if candidate_file.exists():
return src_suffix
return None
def get_page_order(self, sources_dir=None) -> List[Tuple[str, str]]:
"""Get the correct page order from the toctree structure.
Args:
sources_dir: Optional path to _sources directory for suffix detection
Returns:
List of tuples (docname, source_suffix) in toctree order
"""
if not self.env or not self.master_doc:
return []
@@ -52,9 +124,12 @@ class DocumentCollector:
visited.add(docname)
# Add the current document
if docname not in page_order:
page_order.append(docname)
# Add the current document with its suffix
if docname not in [doc for doc, _ in page_order]:
suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
# Check for toctree entries in this document
try:
@@ -65,20 +140,33 @@ class DocumentCollector:
):
for child_docname in self.env.toctree_includes[docname]:
collect_from_toctree(child_docname)
else:
# Fallback: try to resolve and parse the toctree
toctree = self.env.get_and_resolve_toctree(docname, None)
if toctree:
from docutils import nodes
for node in list(toctree.findall(nodes.reference)):
if "refuri" in node.attributes:
refuri = node.attributes["refuri"]
if refuri and refuri.endswith(".html"):
child_docname = refuri[:-5] # Remove .html
# Try to use dependencies to find related documents
elif (
hasattr(self.env, "dependencies")
and docname in self.env.dependencies
):
# Extract the dependent documents from the dependencies dict
for child_docname in self.env.dependencies[docname]:
# Only add documents actually in the document set
if (
child_docname != docname
): # Avoid circular references
hasattr(self.env, "all_docs")
and child_docname in self.env.all_docs
):
collect_from_toctree(child_docname)
# Fallback to titles or other available references
elif hasattr(self.env, "titles") and hasattr(self.env, "all_docs"):
# Get all document names
all_docnames = list(self.env.all_docs.keys())
# Look for documents that might be related (have similar paths)
current_prefix = "/".join(docname.split("/")[:-1])
if current_prefix:
for child_docname in all_docnames:
# Documents in the same directory might be related
if (
child_docname.startswith(current_prefix)
and child_docname != docname
):
collect_from_toctree(child_docname)
except Exception as e:
logger.debug(f"Could not get toctree for {docname}: {e}")
@@ -88,22 +176,33 @@ class DocumentCollector:
# Add any remaining documents not in the toctree (sorted)
if hasattr(self.env, "all_docs"):
processed_docnames = {doc for doc, _ in page_order}
remaining = sorted(
[doc for doc in self.env.all_docs.keys() if doc not in page_order]
[
doc
for doc in self.env.all_docs.keys()
if doc not in processed_docnames
]
)
page_order.extend(remaining)
for docname in remaining:
suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
return page_order
def filter_excluded_pages(self, page_order: List[str]) -> List[str]:
def filter_excluded_pages(
self, page_order: List[Tuple[str, str]]
) -> List[Tuple[str, str]]:
"""Filter out excluded pages from the page order."""
exclude_patterns = self.config.get("llms_txt_exclude")
if exclude_patterns:
return [
page
for page in page_order
(docname, suffix)
for docname, suffix in page_order
if not any(
self._match_exclude_pattern(page, pattern)
self._match_exclude_pattern(docname, pattern)
for pattern in exclude_patterns
)
]
+786 -112
View File
File diff suppressed because it is too large Load Diff
+192 -10
View File
@@ -44,7 +44,10 @@ class DocumentProcessor:
Returns:
Processed content with directives properly resolved
"""
# First process include directives
# First process llms-txt-ignore blocks
content = self._process_ignore_blocks(content)
# Then process include directives
content = self._process_includes(content, source_path)
# Then process path directives (image, figure, etc.)
@@ -94,8 +97,14 @@ class DocumentProcessor:
if not base_url:
return path
# Ensure base URL ends with slash
if not base_url.endswith("/"):
base_url += "/"
# Remove leading slash from path to avoid double slashes
if path.startswith("/"):
path = path[1:]
return f"{base_url}{path}"
def _is_absolute_or_url(self, path: str) -> bool:
@@ -119,8 +128,11 @@ class DocumentProcessor:
Returns:
Processed content with directive paths properly resolved
"""
# Get code block ranges to skip directives inside them
code_block_ranges = self._get_code_block_ranges(content)
# Get the configured path directives to process
default_path_directives = ["image", "figure"]
default_path_directives = ["image", "figure", "literalinclude"]
custom_path_directives = self.config.get("llms_txt_directives")
path_directives = set(default_path_directives + custom_path_directives)
@@ -134,11 +146,49 @@ class DocumentProcessor:
is_test = "pytest" in str(source_path) and "subdir" in str(source_path)
def replace_directive_path(match, base_url=base_url, is_test=is_test):
# Check if this directive is within a code block
if self._is_in_code_block(match.start(), code_block_ranges):
# This directive is inside a code block, don't process it
return match.group(0)
prefix = match.group(1) # The entire directive prefix including whitespace
path = match.group(3).strip() # The path argument
# Only process relative paths, not absolute paths or URLs
if not self._is_absolute_or_url(path):
# Handle URLs and data URIs - leave unchanged
if path.startswith(("http://", "https://", "data:")):
return match.group(0)
# For ALL paths, check if image exists in _images first
# Extract filename from the path
filename = os.path.basename(path)
# Check if image exists in _images directory
# First determine the build directory from source_path
build_dir = None
if "_sources" in str(source_path):
# Extract build directory (parent of _sources)
path_parts = str(source_path).split("_sources/")
if len(path_parts) > 1:
build_dir = path_parts[0].rstrip("/")
# If we can determine the build directory, check if image exists in _images
if build_dir:
images_path = os.path.join(build_dir, "_images", filename)
if os.path.exists(images_path):
# Image exists in _images, use _images path
full_path = f"/_images/{filename}"
# Add base URL if configured
full_path = self._add_base_url(full_path, base_url)
return f"{prefix}{full_path}"
# Image doesn't exist in _images, handle based on path type
# Handle absolute paths (starting with /) - add base URL if configured
if path.startswith("/"):
# Add base URL to absolute paths if configured
full_path = self._add_base_url(path, base_url)
return f"{prefix}{full_path}"
# Handle relative paths with original logic for backward compatibility
# Special case for test files
if is_test:
# Add subdir/ prefix to match test expectations
@@ -168,9 +218,7 @@ class DocumentProcessor:
elif rel_doc_dir:
# Join with the original path to form full path relative
# to srcdir
full_path = os.path.normpath(
os.path.join(rel_doc_dir, path)
)
full_path = os.path.normpath(os.path.join(rel_doc_dir, path))
else:
full_path = path
@@ -180,7 +228,12 @@ class DocumentProcessor:
# Return the updated directive with the full path
return f"{prefix}{full_path}"
# If we couldn't resolve the path or it's already absolute, return unchanged
# Fallback for relative paths - add base URL if configured
else:
full_path = self._add_base_url(path, base_url)
return f"{prefix}{full_path}"
# If we couldn't resolve the path, return unchanged
return match.group(0)
# Replace directive paths in the content
@@ -201,9 +254,12 @@ class DocumentProcessor:
"""
possible_paths = []
# If it's an absolute path, use it directly
# If it's an absolute path, treat it as relative to srcdir
if os.path.isabs(include_path):
possible_paths.append(Path(include_path))
# Remove the leading slash and treat as relative to srcdir
relative_path = include_path.lstrip("/")
if self.srcdir:
possible_paths.append((Path(self.srcdir) / relative_path).resolve())
else:
# Relative to the source file (in _sources directory)
possible_paths.append((source_path.parent / include_path).resolve())
@@ -228,6 +284,71 @@ class DocumentProcessor:
return possible_paths
def _get_code_block_ranges(self, content: str) -> List[Tuple[int, int]]:
"""Find all code block ranges in the content.
Args:
content: The source content to analyze
Returns:
List of (start, end) tuples representing code block character
ranges
"""
code_block_ranges = []
# Match code block as well as `code` and `sourcecode` aliases
code_block_pattern = re.compile(
r"^(\s*)\.\.\s+(code-block|code|sourcecode)::\s*\S*\s*$", re.MULTILINE
)
for match in code_block_pattern.finditer(content):
start_pos = match.start()
indent = match.group(1)
indent_len = len(indent)
# Find the end of the code block by looking for the next line
# that is not indented more than the directive
block_start = match.end()
pos = block_start
# Skip any blank lines immediately after the directive
while pos < len(content) and content[pos] in "\n":
pos += 1
# Find where the code block ends
lines = content[pos:].split("\n")
block_end = pos
for line in lines:
if line.strip(): # Non-empty line
# Check indentation level
line_indent = len(line) - len(line.lstrip())
if line_indent <= indent_len:
# The block ends when we find a line that is indented
# less than the directive itself
break
block_end += len(line) + 1 # +1 for the newline
code_block_ranges.append((start_pos, block_end))
return code_block_ranges
def _is_in_code_block(
self, match_start: int, code_block_ranges: List[Tuple[int, int]]
) -> bool:
"""Check if a match position is within a code block.
Args:
match_start: The starting position of the match
code_block_ranges: List of (start, end) tuples for code blocks
Returns:
True if the match is within a code block, False otherwise
"""
for block_start, block_end in code_block_ranges:
if block_start <= match_start < block_end:
return True
return False
def _process_includes(self, content: str, source_path: Path) -> str:
"""Process include directives in content.
@@ -238,12 +359,22 @@ class DocumentProcessor:
Returns:
Processed content with include directives replaced with included content
"""
code_block_ranges = self._get_code_block_ranges(content)
# Find all include directives using regex
include_pattern = build_directive_pattern(["include"])
# Function to replace each include with content
def replace_include(match):
# Check if this include is within a code block
if self._is_in_code_block(match.start(), code_block_ranges):
# This include is inside a code block, don't process it
return match.group(0)
include_path = match.group(3)
directive_part = match.group(
1
) # The ".. include:: " part with leading whitespace
# Get all possible paths to try
possible_paths = self._resolve_include_paths(include_path, source_path)
@@ -254,7 +385,18 @@ class DocumentProcessor:
if path_to_try.exists():
with open(path_to_try, "r", encoding="utf-8") as f:
included_content = f.read()
# Find where the actual directive starts, after any whitespace
directive_start = directive_part.find("..")
if directive_start > 0:
# There's leading whitespace/newlines before the directive
leading_part = directive_part[:directive_start]
# Replace directive with content, preserving the structure
return leading_part + included_content
else:
# No leading whitespace, just return the content
return included_content
except Exception as e:
logger.error(
f"sphinx-llms-txt: Error reading include file {path_to_try}:"
@@ -266,8 +408,48 @@ class DocumentProcessor:
paths_tried = ", ".join(str(p) for p in possible_paths)
logger.warning(f"sphinx-llms-txt: Include file not found: {include_path}")
logger.debug(f"sphinx-llms-txt: Tried paths: {paths_tried}")
# Preserve spacing structure for error message too
directive_start = match.group(1).find("..")
if directive_start > 0:
leading_part = match.group(1)[:directive_start]
return leading_part + f"[Include file not found: {include_path}]"
else:
return f"[Include file not found: {include_path}]"
# Replace all includes with their content
processed_content = include_pattern.sub(replace_include, content)
return processed_content
def _process_ignore_blocks(self, content: str) -> str:
"""Process llms-txt-ignore-start/end blocks by removing their content.
Args:
content: The source content to process
Returns:
Processed content with ignore blocks removed
"""
# Process ignore blocks iteratively to handle nested cases correctly
while True:
# Pattern to match ignore blocks - handles whitespace and indentation
ignore_pattern = re.compile(
r"^\s*\.\.\s+llms-txt-ignore-start\s*\n" # Start directive line
r"(.*?)" # Content to ignore (non-greedy)
r"^\s*\.\.\s+llms-txt-ignore-end\s*$", # End directive line
re.MULTILINE | re.DOTALL,
)
# Find and remove one ignore block at a time
match = ignore_pattern.search(content)
if not match:
break
# Remove the matched block
content = content[: match.start()] + content[match.end() :]
# Clean up any extra blank lines that might be left
# Replace multiple consecutive newlines with at most 2 newlines
processed_content = re.sub(r"\n\n\n+", "\n\n", content)
return processed_content
+86 -7
View File
@@ -3,7 +3,7 @@ File writer module for sphinx-llms-txt.
"""
from pathlib import Path
from typing import Any, Dict, List
from typing import Any, Dict, List, Tuple, Union
from sphinx.application import Sphinx
from sphinx.util import logging
@@ -19,6 +19,42 @@ class FileWriter:
self.outdir = outdir
self.app = app
def _resolve_uri_template(self, sources_dir: Path = None) -> str:
"""Resolve which URI template to use based on configuration and sources_dir.
Args:
sources_dir: Path to _sources directory (None if not found)
Returns:
The template string to use for generating URIs
"""
# If custom template exists
custom_template = self.config.get("llms_txt_uri_template")
if custom_template:
# Validate user's template by checking for valid variable names
try:
# Try formatting with test valid values to validate syntax
test_values = {
"base_url": "http://example.com/",
"docname": "test",
"suffix": ".rst",
"sourcelink_suffix": ".txt",
}
custom_template.format(**test_values)
return custom_template
except (KeyError, ValueError) as e:
logger.warning(
f"sphinx-llms-txt: Invalid llms_txt_uri_template: {e}. "
f"Falling back to default."
)
# Else, use one of the default templates
if sources_dir:
return "{base_url}_sources/{docname}{suffix}{sourcelink_suffix}"
else:
return "{base_url}{docname}.html"
def write_combined_file(
self, content_parts: List[str], output_path: Path, total_line_count: int
) -> bool:
@@ -37,26 +73,28 @@ class FileWriter:
f.write("\n".join(content_parts))
logger.info(
f"sphinx-llms-txt: created {output_path} with {len(content_parts)}"
f"sphinx-llms-txt: Created {output_path} with {len(content_parts)}"
f" sources and {total_line_count} lines"
)
return True
except Exception as e:
logger.error(f"sphinx-llm-txt: Error writing combined sources file: {e}")
logger.error(f"sphinx-llms-txt: Error writing combined sources file: {e}")
return False
def write_verbose_info_to_file(
self,
page_order: List[str],
page_order: Union[List[str], List[Tuple[str, str]]],
page_titles: Dict[str, str],
total_line_count: int = 0,
sources_dir: Path = None,
) -> bool:
"""Write summary information to the llms.txt file.
Args:
page_order: Ordered list of document names
page_order: Ordered list of document names or (docname, suffix) tuples
page_titles: Dictionary mapping docnames to titles
total_line_count: Total number of lines in the combined content
sources_dir: Path to _sources directory (None if not found)
Returns:
True if successful, False otherwise
@@ -86,6 +124,13 @@ class FileWriter:
# Add description if available
description = self.config.get("llms_txt_summary", "")
if description:
# Trim leading and trailing whitespace
description = description.strip()
if description:
# Only add blockquote if description is not empty
# Replace newlines with newline + blockquote marker to maintain
# blockquote formatting
description = description.replace("\n", "\n> ")
f.write(f"> {description}\n\n")
f.write("## Docs\n\n")
@@ -95,9 +140,43 @@ class FileWriter:
if not base_url.endswith("/"):
base_url += "/"
for docname in page_order:
# Get sourcelink suffix from Sphinx config
sourcelink_suffix = ""
if self.app and hasattr(self.app.config, "html_sourcelink_suffix"):
sourcelink_suffix = self.app.config.html_sourcelink_suffix
# Handle empty string case specially
if sourcelink_suffix == "":
sourcelink_suffix = "" # Keep it empty
elif not sourcelink_suffix.startswith("."):
sourcelink_suffix = "." + sourcelink_suffix
# Resolve which template to use
uri_template = self._resolve_uri_template(sources_dir)
for item in page_order:
# Handle both old format (str) and new format (tuple)
if isinstance(item, tuple):
docname, suffix = item
else:
docname = item
suffix = None
title = page_titles.get(docname, docname)
f.write(f"- [{title}]({base_url}{docname}.html)\n")
# Avoid duplicate extensions when suffix matches
# sourcelink_suffix (e.g., both are ".txt")
effective_sourcelink_suffix = sourcelink_suffix
if suffix and suffix == sourcelink_suffix:
effective_sourcelink_suffix = ""
uri = uri_template.format(
base_url=base_url,
docname=docname,
suffix=suffix or "",
sourcelink_suffix=effective_sourcelink_suffix,
)
f.write(f"- [{title}]({uri})\n")
logger.info(f"sphinx-llms-txt: created {output_path}")
return True
+2
View File
@@ -8,6 +8,8 @@ Welcome to Test Project's documentation!
page1
page2
page_with_include
page_ignored_metadata
page_with_ignore_blocks
Indices and tables
==================
@@ -0,0 +1,16 @@
:llms-txt-ignore: true
Page Ignored by Metadata
========================
This page should not appear in llms-full.txt because of the metadata directive.
Section 1
---------
This content should be completely ignored.
Section 2
---------
This content should also be ignored.
@@ -0,0 +1,39 @@
Page With Ignore Blocks
=======================
This content should appear in llms-full.txt.
.. llms-txt-ignore-start
This content should be ignored and not appear in llms-full.txt.
Section Ignored
---------------
This section should also be ignored.
.. llms-txt-ignore-end
This content after the ignore block should appear in llms-full.txt.
Another Section
---------------
This content should definitely appear.
.. llms-txt-ignore-start
Another ignored block with multiple lines.
- Item 1 (ignored)
- Item 2 (ignored)
.. code-block:: python
# This code should be ignored
def ignored_function():
pass
.. llms-txt-ignore-end
Final content that should appear.
+220
View File
@@ -0,0 +1,220 @@
"""Tests for llms-txt ignore features."""
from pathlib import Path
from sphinx_llms_txt import DocumentProcessor
def test_process_ignore_blocks():
"""Test that ignore blocks are properly removed from content."""
processor = DocumentProcessor({}, None)
content = """This content should remain.
.. llms-txt-ignore-start
This content should be removed.
Section Ignored
---------------
This section should also be removed.
.. llms-txt-ignore-end
This content should remain after the ignore block.
.. llms-txt-ignore-start
Another ignored block.
Multiple lines here.
.. llms-txt-ignore-end
Final content that should remain."""
processed = processor._process_ignore_blocks(content)
# Check that ignored content is removed
assert "This content should be removed." not in processed
assert "Section Ignored" not in processed
assert "Another ignored block." not in processed
assert "Multiple lines here." not in processed
# Check that non-ignored content remains
assert "This content should remain." in processed
assert "This content should remain after the ignore block." in processed
assert "Final content that should remain." in processed
def test_process_ignore_blocks_with_indentation():
"""Test that ignore blocks work with different indentation levels."""
processor = DocumentProcessor({}, None)
content = """Section Title
=============
Normal content.
.. llms-txt-ignore-start
Indented ignored content.
More indented content.
.. llms-txt-ignore-end
Back to normal content."""
processed = processor._process_ignore_blocks(content)
# Check that ignored content is removed
assert "Indented ignored content." not in processed
assert "More indented content." not in processed
# Check that non-ignored content remains
assert "Section Title" in processed
assert "Normal content." in processed
assert "Back to normal content." in processed
def test_process_ignore_blocks_multiple():
"""Test that multiple ignore blocks are handled correctly."""
processor = DocumentProcessor({}, None)
content = """Start content.
.. llms-txt-ignore-start
First ignore block.
.. llms-txt-ignore-end
Middle content that should remain.
.. llms-txt-ignore-start
Second ignore block.
.. llms-txt-ignore-end
End content."""
processed = processor._process_ignore_blocks(content)
# Check that ignored content is removed
assert "First ignore block." not in processed
assert "Second ignore block." not in processed
# Check that non-ignored content remains
assert "Start content." in processed
assert "Middle content that should remain." in processed
assert "End content." in processed
def test_build_with_ignore_features(basic_sphinx_app):
"""Test building HTML documentation with ignore features."""
app = basic_sphinx_app
app.build()
# Check if the output file was created
output_file = Path(app.outdir) / "test-llms-full.txt"
assert output_file.exists(), f"Output file {output_file} does not exist"
# Read the content of the output file
content = output_file.read_text()
# Check that page with metadata ignore is completely excluded
assert "Page Ignored by Metadata" not in content
assert "This page should not appear in llms-full.txt" not in content
# Check that page with ignore blocks has the right content
assert "Page With Ignore Blocks" in content
assert "This content should appear in llms-full.txt." in content
assert "This content after the ignore block should appear" in content
assert "Another Section" in content
assert "Final content that should appear." in content
# Check that ignored block content is not present
assert "This content should be ignored and not appear" not in content
assert "Section Ignored" not in content
assert "Another ignored block with multiple lines." not in content
assert "Item 1 (ignored)" not in content
assert "def ignored_function():" not in content
def test_manager_mark_page_ignored():
"""Test that manager can mark pages as ignored."""
from sphinx_llms_txt import LLMSFullManager
manager = LLMSFullManager()
# Initially no pages are ignored
assert len(manager.ignored_pages) == 0
# Mark a page as ignored
manager.mark_page_ignored("test_page")
# Check that page is in ignored set
assert "test_page" in manager.ignored_pages
assert len(manager.ignored_pages) == 1
# Mark another page as ignored
manager.mark_page_ignored("another_page")
# Check both pages are ignored
assert "test_page" in manager.ignored_pages
assert "another_page" in manager.ignored_pages
assert len(manager.ignored_pages) == 2
def test_process_ignore_blocks_empty_blocks():
"""Test that empty ignore blocks are handled correctly."""
processor = DocumentProcessor({}, None)
content = """Content before.
.. llms-txt-ignore-start
.. llms-txt-ignore-end
Content after."""
processed = processor._process_ignore_blocks(content)
# Check that content remains
assert "Content before." in processed
assert "Content after." in processed
# Check that we don't have excessive newlines
lines = processed.strip().split("\n")
non_empty_lines = [line for line in lines if line.strip()]
assert len(non_empty_lines) == 2
def test_ignore_metadata_affects_both_files(basic_sphinx_app):
"""Test that :llms-txt-ignore: true affects both files."""
app = basic_sphinx_app
# Enable both llms.txt and llms-full.txt file generation
app.config.llms_txt_file = True
app.config.llms_txt_filename = "test-llms.txt"
app.build()
# Check if both output files were created
llms_full_file = Path(app.outdir) / "test-llms-full.txt"
llms_summary_file = Path(app.outdir) / "test-llms.txt"
assert llms_full_file.exists(), f"Output file {llms_full_file} does not exist"
assert llms_summary_file.exists(), f"Output file {llms_summary_file} does not exist"
# Read the content of both files
llms_full_content = llms_full_file.read_text()
llms_summary_content = llms_summary_file.read_text()
# Check that page with metadata ignore is excluded from llms-full.txt
assert "Page Ignored by Metadata" not in llms_full_content
assert "This page should not appear in llms-full.txt" not in llms_full_content
# Check that page with metadata ignore is also excluded from llms.txt
# This should NOT contain a link to the ignored page
assert "Page Ignored by Metadata" not in llms_summary_content
assert "page_ignored_metadata.html" not in llms_summary_content
+146
View File
@@ -117,6 +117,152 @@ def test_max_lines_limit(temp_dir, rootdir):
app.docutils_conf_path.unlink()
def test_on_exceed_skip(temp_dir, rootdir):
"""Test that skip action works when size limit is exceeded."""
from sphinx.testing.util import SphinxTestApp
src_dir = rootdir / "basic"
app = SphinxTestApp(
srcdir=src_dir,
builddir=temp_dir,
buildername="html",
freshenv=True,
confoverrides={
"llms_txt_full_filename": "skip-test.txt",
"llms_txt_full_max_size": 20,
"llms_txt_full_size_policy": "warn_skip",
},
)
app.build()
# Check that the output file was NOT created
output_file = Path(app.outdir) / "skip-test.txt"
assert (
not output_file.exists()
), f"Output file {output_file} should not exist with skip action"
# Cleanup
sys.path[:] = app._saved_path
_clean_up_global_state()
if hasattr(app, "docutils_conf_path") and app.docutils_conf_path.exists():
app.docutils_conf_path.unlink()
def test_on_exceed_keep(temp_dir, rootdir):
"""Test that keep action works when size limit is exceeded."""
from sphinx.testing.util import SphinxTestApp
src_dir = rootdir / "basic"
app = SphinxTestApp(
srcdir=src_dir,
builddir=temp_dir,
buildername="html",
freshenv=True,
confoverrides={
"llms_txt_full_filename": "keep-test.txt",
"llms_txt_full_max_size": 20,
"llms_txt_full_size_policy": "info_keep",
},
)
app.build()
# Check that the output file WAS created despite exceeding limit
output_file = Path(app.outdir) / "keep-test.txt"
assert (
output_file.exists()
), f"Output file {output_file} should exist with keep action"
# Verify it has content
content = output_file.read_text()
assert len(content) > 0, "Output file should have content with keep action"
# Cleanup
sys.path[:] = app._saved_path
_clean_up_global_state()
if hasattr(app, "docutils_conf_path") and app.docutils_conf_path.exists():
app.docutils_conf_path.unlink()
def test_on_exceed_note(temp_dir, rootdir):
"""Test that note action works when size limit is exceeded."""
from sphinx.testing.util import SphinxTestApp
src_dir = rootdir / "basic"
app = SphinxTestApp(
srcdir=src_dir,
builddir=temp_dir,
buildername="html",
freshenv=True,
confoverrides={
"llms_txt_full_filename": "note-test.txt",
"llms_txt_full_max_size": 20,
"llms_txt_full_size_policy": "warn_note",
},
)
app.build()
# Check that the output file WAS created with placeholder content
output_file = Path(app.outdir) / "note-test.txt"
assert (
output_file.exists()
), f"Output file {output_file} should exist with note action"
# Verify it has the placeholder content
content = output_file.read_text()
assert (
"This file was not generated because it exceeded the configured size limit."
in content
)
assert "llms_txt_full_max_size" in content
assert "llms_txt_full_size_policy" in content
assert "Configured max size: 20 lines" in content
# Cleanup
sys.path[:] = app._saved_path
_clean_up_global_state()
if hasattr(app, "docutils_conf_path") and app.docutils_conf_path.exists():
app.docutils_conf_path.unlink()
def test_on_exceed_invalid_config(temp_dir, rootdir):
"""Test behavior with invalid configuration values."""
from sphinx.testing.util import SphinxTestApp
src_dir = rootdir / "basic"
app = SphinxTestApp(
srcdir=src_dir,
builddir=temp_dir,
buildername="html",
freshenv=True,
confoverrides={
"llms_txt_full_filename": "invalid-test.txt",
"llms_txt_full_max_size": 20,
"llms_txt_full_size_policy": "invalid_format", # Invalid config
},
)
app.build()
# Should fall back to default behavior (warn_skip)
output_file = Path(app.outdir) / "invalid-test.txt"
assert (
not output_file.exists()
), f"Output file {output_file} should not exist with invalid config fallback"
# Cleanup
sys.path[:] = app._saved_path
_clean_up_global_state()
if hasattr(app, "docutils_conf_path") and app.docutils_conf_path.exists():
app.docutils_conf_path.unlink()
def test_title_override(temp_dir, rootdir):
"""Test that the title override works correctly."""
from sphinx.testing.util import SphinxTestApp
+848
View File
@@ -41,6 +41,42 @@ def test_setup_returns_valid_dict():
assert "parallel_write_safe" in result
def test_builder_inited_with_disallowed_builder():
"""Test that disallowed builders do not trigger extension setup."""
import sphinx_llms_txt
# Reset global state
sphinx_llms_txt._manager = sphinx_llms_txt.LLMSFullManager()
sphinx_llms_txt._root_first_paragraph = ""
# Mock a Sphinx app with a disallowed builder
class MockBuilder:
name = "text" # Not in allowed list
class MockApp:
def __init__(self):
self.config_values = {}
self.connections = {}
self.builder = MockBuilder()
def add_config_value(self, name, default, rebuild):
self.config_values[name] = (default, rebuild)
def connect(self, event, handler):
self.connections[event] = handler
app = MockApp()
setup(app)
# Trigger builder-inited
builder_inited_handler = app.connections["builder-inited"]
builder_inited_handler(app)
# With disallowed builder, other events should NOT be connected
assert "doctree-resolved" not in app.connections
assert "build-finished" not in app.connections
def test_document_collector_initialization():
"""Test initialization of DocumentCollector."""
collector = DocumentCollector()
@@ -162,6 +198,31 @@ def test_process_includes(tmp_path):
assert processed_content == expected_content
def test_process_includes_in_code_block(tmp_path):
"""Test that an `include` within a `code-block` is not processed."""
# Create a processor
config = {"llms_txt_directives": []}
processor = DocumentProcessor(config)
# Create a source file that uses include syntax within a `code-block`
source_content = (
"Normal paragraph.\n\n"
".. code-block:: rst\n\n"
" .. include:: foo.txt\n\n"
"Another normal paragraph."
)
source_file = tmp_path / "source.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Run the include directive processor
processed_content = processor._process_includes(source_content, source_file)
# Check that the include directive was not processed
expected_content = source_content
assert processed_content == expected_content
def test_process_includes_with_relative_paths(tmp_path):
"""Test that include directives with relative paths are processed correctly."""
# Create a processor
@@ -334,3 +395,790 @@ def test_write_verbose_info_with_baseurl(tmp_path):
assert "- [Home Page](https://example.org/index.html)" in content
assert "- [About Us](https://example.org/about.html)" in content
def test_get_source_suffixes_with_dict():
"""Test _get_source_suffixes method with dict source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with dict source_suffix
class MockApp:
class Config:
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert set(suffixes) == {".rst", ".md", ".txt"}
def test_get_source_suffixes_with_list():
"""Test _get_source_suffixes method with list source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with list source_suffix
class MockApp:
class Config:
source_suffix = [".rst", ".md"]
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst", ".md"]
def test_get_source_suffixes_with_string():
"""Test _get_source_suffixes method with string source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with string source_suffix
class MockApp:
class Config:
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"]
def test_get_source_suffixes_no_app():
"""Test _get_source_suffixes method with no app set."""
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"] # Default fallback
def test_html_sourcelink_suffix_default():
"""Test html_sourcelink_suffix defaults to .txt when no app is set."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with default .txt suffix
test_file = f"{sources_dir}/index.rst.txt"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .txt as the default suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed (check if output file exists)
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_custom():
"""Test html_sourcelink_suffix uses custom value from Sphinx config."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with custom html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = "source"
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with custom .source suffix
test_file = f"{sources_dir}/index.rst.source"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .source as the custom suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_with_dot():
"""Test html_sourcelink_suffix adds dot if missing."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with html_sourcelink_suffix without leading dot
class MockApp:
class Config:
html_sourcelink_suffix = "src" # No leading dot
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with .src suffix (dot should be added automatically)
test_file = f"{sources_dir}/index.rst.src"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it adds the dot and finds the file
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_mixed_source_file_formats():
"""Test handling of mixed source file formats (.rst, .md, .txt)."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with multiple source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create test source files with different formats
files_to_create = [
f"{sources_dir}/page1.rst.txt",
f"{sources_dir}/page2.md.txt",
f"{sources_dir}/page3.txt.txt",
]
for test_file in files_to_create:
with open(test_file, "w") as f:
f.write(f"Content for {os.path.basename(test_file)}")
# Mock env with all documents
class MockEnv:
all_docs = {"page1": None, "page2": None, "page3": None}
titles = {
"page1": type("TitleNode", (), {"astext": lambda: "Page 1"})(),
"page2": type("TitleNode", (), {"astext": lambda: "Page 2"})(),
"page3": type("TitleNode", (), {"astext": lambda: "Page 3"})(),
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("page1")
# Test that all file formats are found and processed
manager.combine_sources(outdir, srcdir)
# Verify the output file was created and contains content from all formats
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# Should contain content from all three files
assert "Content for page1.rst.txt" in content
assert "Content for page2.md.txt" in content
assert "Content for page3.txt.txt" in content
def test_source_suffix_detection_priority():
"""Test source suffix detection tries formats in correct order for docnames."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with ordered source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = [".rst", ".md"] # rst has priority over md
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create both .rst and .md versions of the same document
# Only create files for the specific docname "index"
rst_file = f"{sources_dir}/index.rst.txt"
md_file = f"{sources_dir}/index.md.txt"
with open(rst_file, "w") as f:
f.write("RST content for index")
with open(md_file, "w") as f:
f.write("Markdown content for index")
# Mock env with only the index document
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Index Page"})()
}
toctree_includes = {"index": []}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test the priority behavior
manager.combine_sources(outdir, srcdir)
# Check that output file was created
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# The system should prefer RST over MD for the "index" docname
# But since both files exist and the second phase adds remaining files,
# both will be included. The test verifies that RST appears first
# (indicating it was found first in the priority order)
assert "RST content for index" in content
# Find positions to verify order
rst_pos = content.find("RST content for index")
md_pos = content.find("Markdown content for index")
# RST should come before MD (due to priority in toctree processing)
assert rst_pos < md_pos, "RST content should appear before MD content"
def test_summary_default_uses_first_paragraph():
"""
Test that summary defaults to first paragraph of root document when not configured.
"""
from docutils import nodes
from docutils.frontend import OptionParser
from docutils.parsers.rst import Parser
from docutils.utils import new_document
from sphinx_llms_txt import build_finished, doctree_resolved
# Create a proper document with settings
settings = OptionParser(components=(Parser,)).get_default_values()
doctree = new_document("<rst-doc>", settings)
title = nodes.title(text="Test Title")
paragraph = nodes.paragraph(
text="This is the first paragraph that should be used as summary."
)
doctree.append(title)
doctree.append(paragraph)
# Mock Sphinx app
class MockApp:
class Config:
master_doc = "index"
llms_txt_summary = None # Not configured
llms_txt_file = True
llms_txt_filename = "llms.txt"
llms_txt_uri_template = None
llms_txt_title = None
llms_txt_full_file = True
llms_txt_full_filename = "llms-full.txt"
llms_txt_full_max_size = None
llms_txt_full_size_policy = "warn_skip"
llms_txt_directives = []
llms_txt_exclude = []
llms_txt_code_files = []
llms_txt_code_base_path = None
html_baseurl = ""
config = Config()
outdir = "/tmp/build"
srcdir = "/tmp/source"
class Env:
titles = {
"index": type("TitleNode", (), {"astext": lambda self: "Test Title"})()
}
env = Env()
app = MockApp()
# Reset the global state
import sphinx_llms_txt
sphinx_llms_txt._root_first_paragraph = ""
# Call doctree_resolved to extract the first paragraph
doctree_resolved(app, doctree, "index")
# Verify the first paragraph was extracted
assert (
sphinx_llms_txt._root_first_paragraph
== "This is the first paragraph that should be used as summary."
)
# Mock the manager methods to avoid actual file operations
original_combine_sources = sphinx_llms_txt._manager.combine_sources
sphinx_llms_txt._manager.combine_sources = lambda outdir, srcdir: None
# Call build_finished and verify the summary is set correctly
build_finished(app, None)
# Check that the summary was properly configured
assert (
sphinx_llms_txt._manager.config["llms_txt_summary"]
== "This is the first paragraph that should be used as summary."
)
# Restore original method
sphinx_llms_txt._manager.combine_sources = original_combine_sources
def test_code_files_include_exclude_patterns(tmp_path):
"""Test the +/- pattern syntax for llms_txt_code_files configuration."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
cache_dir = docs_dir / "__pycache__"
cache_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
(docs_dir / "guide.rst").write_text("Guide RST content")
(docs_dir / "backup.bak").write_text("Backup file content")
(cache_dir / "compiled.pyc").write_text("Compiled Python")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with include/exclude patterns
config = {
"llms_txt_code_files": [
"+:docs/**/*.rst", # Include all RST files in docs
"-:docs/**/__pycache__/**", # Exclude pycache files
"-:docs/**/*.bak", # Exclude backup files
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Verify we have the expected number of files
assert len(code_parts) == 2, f"Expected 2 files, got {len(code_parts)}"
# Extract file titles from code blocks
titles = []
for part in code_parts:
lines = part.strip().split("\n")
if lines:
titles.append(lines[0])
# Verify expected files are included
assert "docs/example.rst" in titles
assert "docs/guide.rst" in titles
# Verify excluded files are not present
content = "\n".join(code_parts)
assert "backup.bak" not in content
assert "__pycache__" not in content
assert "compiled.pyc" not in content
def test_code_files_exclude_only_patterns(tmp_path):
"""Test that exclude-only patterns result in no files being included."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with only exclude patterns
config = {
"llms_txt_code_files": [
"-:docs/**/*.rst", # Only exclude pattern, no includes
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should have no files with exclude-only patterns
assert len(code_parts) == 0, "Should have no files with exclude-only patterns"
def test_code_files_no_prefix_patterns(tmp_path):
"""Test that patterns without prefix are ignored."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
(docs_dir / "backup.bak").write_text("Backup file content")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with no prefix (should be ignored)
config = {
"llms_txt_code_files": [
"docs/**/*.rst", # No prefix = ignored
"+:docs/**/*.rst", # Include RST files
"-:docs/**/*.bak", # Exclude backup files
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should include RST files (from +: pattern) and exclude BAK files (from -: pattern)
assert len(code_parts) == 1, "Should include RST files and exclude BAK files"
content = "\n".join(code_parts)
assert "Example RST content" in content
assert "backup.bak" not in content
def test_code_files_ignored_patterns(tmp_path, caplog):
"""Test that patterns without +: or -: prefix log a warning and are ignored."""
from unittest.mock import patch
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
# Use a mock to capture the warning message directly
captured_warnings = []
def capture_warning(message, *args, **kwargs):
captured_warnings.append(message)
# Patch the logger to capture warnings
with patch("sphinx_llms_txt.manager.logger.warning", side_effect=capture_warning):
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with only no-prefix patterns (should result in no files)
config = {
"llms_txt_code_files": [
"docs/**/*.rst", # No prefix = ignored with warning
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should have no files since the pattern without prefix is ignored
assert (
len(code_parts) == 0
), "Should have no files when only using patterns without prefix"
# Check that a warning was logged
assert (
len(captured_warnings) == 1
), f"Expected 1 warning, got {len(captured_warnings)}"
assert (
"Code file pattern 'docs/**/*.rst' ignored." in captured_warnings[0]
), f"Warning message should contain expected text. Got: {captured_warnings[0]}"
def test_llms_txt_generated_without_sources_dir(tmp_path):
"""Test that llms.txt is generated even when _sources directory doesn't exist."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create manager
manager = LLMSFullManager()
# Set config to enable llms.txt
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"llms_txt_full_file": True,
"llms_txt_full_filename": "llms-full.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
manager.set_config(config)
# Create directories (but no _sources)
outdir = tmp_path / "build"
srcdir = tmp_path / "source"
outdir.mkdir()
srcdir.mkdir()
# Mock env with documents
class MockEnv:
all_docs = {"index": None, "about": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda self: "Home"})(),
"about": type("TitleNode", (), {"astext": lambda self: "About"})(),
}
toctree_includes = {"index": ["about"]}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Update page titles directly in the collector
manager.update_page_title("index", "Home")
manager.update_page_title("about", "About")
# Call combine_sources - should generate llms.txt even without _sources
manager.combine_sources(str(outdir), str(srcdir))
# Verify llms.txt was created
llms_txt = outdir / "llms.txt"
assert llms_txt.exists(), "llms.txt should be generated even without _sources"
# Verify llms-full.txt was NOT created (since no _sources)
llms_full_txt = outdir / "llms-full.txt"
assert (
not llms_full_txt.exists()
), "llms-full.txt should not be generated without _sources"
# Read llms.txt and verify it has content
with open(llms_txt, "r", encoding="utf-8") as f:
content = f.read()
# Should contain page titles and links
assert "Home" in content
assert "About" in content
assert "index.html" in content
assert "about.html" in content
def test_llms_txt_no_warning_when_full_file_disabled(tmp_path, caplog):
"""
Test that no warning is logged when llms_txt_full_file=False and
_sources doesn't exist.
"""
from unittest.mock import patch
from sphinx_llms_txt.manager import LLMSFullManager
# Create manager
manager = LLMSFullManager()
# Set config with llms_txt_full_file=False
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"llms_txt_full_file": False, # User doesn't want llms-full.txt
"llms_txt_full_filename": "llms-full.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
manager.set_config(config)
# Create directories (but no _sources)
outdir = tmp_path / "build"
srcdir = tmp_path / "source"
outdir.mkdir()
srcdir.mkdir()
# Mock env with documents
class MockEnv:
all_docs = {"index": None}
titles = {"index": type("TitleNode", (), {"astext": lambda self: "Home"})()}
toctree_includes = {"index": []}
manager.set_env(MockEnv())
manager.set_master_doc("index")
manager.update_page_title("index", "Home")
# Capture warnings
captured_warnings = []
def capture_warning(message, *args, **kwargs):
if "_sources" in str(message):
captured_warnings.append(message)
with patch("sphinx_llms_txt.manager.logger.warning", side_effect=capture_warning):
# Call combine_sources
manager.combine_sources(str(outdir), str(srcdir))
# Verify NO warning was logged since llms_txt_full_file=False
assert (
len(captured_warnings) == 0
), "No warning should be logged when llms_txt_full_file=False"
# Verify llms.txt was still created
llms_txt = outdir / "llms.txt"
assert llms_txt.exists()
+156 -3
View File
@@ -102,7 +102,7 @@ def test_process_path_directives_with_html_baseurl(tmp_path):
def test_process_path_directives_absolute_urls(tmp_path):
"""Test that absolute URLs are not modified."""
"""Test that absolute URLs are not modified but absolute paths get base URL."""
# Create a processor
config = {
"llms_txt_directives": [],
@@ -127,10 +127,17 @@ def test_process_path_directives_absolute_urls(tmp_path):
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives (should remain unchanged)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
assert processed_content == source_content
# Expected: URLs and data URIs unchanged, absolute paths get base URL
expected_content = (
".. image:: https://othersite.com/images/test.png\n"
".. image:: https://example.com/docs/absolute/path/image.png\n"
".. image:: data:image/png;base64,iVBORw0KG...\n"
)
assert processed_content == expected_content
def test_process_path_directives_custom_directives(tmp_path):
@@ -251,3 +258,149 @@ def test_process_content_end_to_end(tmp_path):
)
assert processed_content == expected_content
def test_process_path_directives_images_directory(tmp_path):
"""Test that _images directory paths are handled correctly."""
# Create a processor with base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "https://example.com/docs",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create _sources directory to mimic Sphinx output
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Create a source file with various _images directory paths
source_content = (
"Some content.\n"
".. image:: _images/test.png\n" # Relative _images should become /_images
".. image:: /_images/absolute.png\n" # Absolute _images should get base URL
".. figure:: _images/figure.png\n" # Test with figure directive too
" :alt: A test figure\n"
".. image:: images/normal.png\n" # Normal relative path should be unchanged
)
# Create source file in sources directory to simulate Sphinx build output
source_file = sources_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: _images paths should be converted and get base URL
expected_content = (
"Some content.\n"
".. image:: https://example.com/docs/_images/test.png\n"
".. image:: https://example.com/docs/_images/absolute.png\n"
".. figure:: https://example.com/docs/_images/figure.png\n"
" :alt: A test figure\n"
".. image:: https://example.com/docs/images/normal.png\n"
)
assert processed_content == expected_content
def test_process_path_directives_images_directory_no_baseurl(tmp_path):
"""
Test that _images directory paths work correctly without base URL.
Only converts when image exists.
"""
# Create a processor without base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create _sources directory to mimic Sphinx output
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Create _images directory and one test image
images_dir = build_dir / "_images"
images_dir.mkdir()
(images_dir / "test.png").write_text("fake image content")
# Note: absolute.png is not created, so it won't be converted
# Create a source file with _images directory paths
source_content = (
".. image:: _images/test.png\n" # Should become /_images (image exists)
".. image:: /_images/absolute.png\n" # Should stay unchanged (absolute path)
)
# Create source file in sources directory to simulate Sphinx build output
source_file = sources_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: only test.png gets converted because it exists in _images
expected_content = (
".. image:: /_images/test.png\n" # Converted because image exists
".. image:: /_images/absolute.png\n" # Absolute path unchanged
)
assert processed_content == expected_content
def test_process_path_directives_all_absolute_paths_get_baseurl(tmp_path):
"""Test that all absolute paths (starting with /) get base URL prepended."""
# Create a processor with base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "https://mysite.com/docs/",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create a source file with various absolute paths
source_content = (
".. image:: /static/images/logo.png\n"
".. figure:: /assets/diagrams/flow.svg\n"
".. image:: /media/photos/team.jpg\n"
" :alt: Team photo\n"
".. image:: relative/path.png\n" # This should still get normal processing
)
# Create source file
source_file = src_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: All absolute paths get base URL prepended
expected_content = (
".. image:: https://mysite.com/docs/static/images/logo.png\n"
".. figure:: https://mysite.com/docs/assets/diagrams/flow.svg\n"
".. image:: https://mysite.com/docs/media/photos/team.jpg\n"
" :alt: Team photo\n"
".. image:: https://mysite.com/docs/relative/path.png\n"
)
assert processed_content == expected_content
+217
View File
@@ -0,0 +1,217 @@
"""Test URI template functionality for llms.txt links."""
from sphinx_llms_txt import FileWriter
def test_uri_template_with_sources_dir(tmp_path):
"""Test that default template uses _sources links when sources_dir exists."""
build_dir = tmp_path / "build"
build_dir.mkdir()
# Create _sources directory to simulate its existence
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Mock app with html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
config = Config()
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"llms_txt_uri_template": (
"{base_url}_sources/{docname}{suffix}{sourcelink_suffix}"
),
"html_baseurl": "https://example.com",
}
writer = FileWriter(config, str(build_dir), MockApp())
page_titles = {
"index": "Home Page",
"about": "About Us",
}
# Page order with suffixes (simulating _sources files exist)
page_order = [("index", ".rst"), ("about", ".md")]
writer.write_verbose_info_to_file(page_order, page_titles, 0, sources_dir)
# Check that the file was created
verbose_file = build_dir / "llms.txt"
assert verbose_file.exists()
# Read the file content
with open(verbose_file, "r", encoding="utf-8") as f:
content = f.read()
# Should link to _sources files
assert "- [Home Page](https://example.com/_sources/index.rst.txt)" in content
assert "- [About Us](https://example.com/_sources/about.md.txt)" in content
def test_uri_template_without_sources_dir(tmp_path):
"""
Test that HTML template is used when sources_dir doesn't exist and no custom
template.
"""
build_dir = tmp_path / "build"
build_dir.mkdir()
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
# No custom template set
"html_baseurl": "https://example.com",
}
writer = FileWriter(config, str(build_dir))
page_titles = {
"index": "Home Page",
"about": "About Us",
}
# Page order without suffixes (simulating no _sources)
page_order = [("index", None), ("about", None)]
# Pass None for sources_dir to simulate it doesn't exist
writer.write_verbose_info_to_file(page_order, page_titles, 0, None)
# Check that the file was created
verbose_file = build_dir / "llms.txt"
assert verbose_file.exists()
# Read the file content
with open(verbose_file, "r", encoding="utf-8") as f:
content = f.read()
# Should fallback to HTML links
assert "- [Home Page](https://example.com/index.html)" in content
assert "- [About Us](https://example.com/about.html)" in content
def test_uri_template_custom(tmp_path):
"""Test that custom URI template works correctly."""
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Mock app with html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
config = Config()
# Custom template that uses different path
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"llms_txt_uri_template": "{base_url}raw/{docname}{suffix}",
"html_baseurl": "https://example.com/",
}
writer = FileWriter(config, str(build_dir), MockApp())
page_titles = {
"index": "Home Page",
}
page_order = [("index", ".rst")]
writer.write_verbose_info_to_file(page_order, page_titles, 0, sources_dir)
verbose_file = build_dir / "llms.txt"
with open(verbose_file, "r", encoding="utf-8") as f:
content = f.read()
# Should use custom template
assert "- [Home Page](https://example.com/raw/index.rst)" in content
def test_uri_template_no_double_extension_when_suffix_matches_sourcelink(tmp_path):
"""Test that .txt suffix + .txt sourcelink_suffix doesn't produce .txt.txt URLs."""
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
config = Config()
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"html_baseurl": "https://example.com",
}
writer = FileWriter(config, str(build_dir), MockApp())
page_titles = {
"index": "Home Page",
"contents": "Table of Contents",
}
# .txt source files — suffix matches sourcelink_suffix
page_order = [("index", ".txt"), ("contents", ".txt")]
writer.write_verbose_info_to_file(page_order, page_titles, 0, sources_dir)
verbose_file = build_dir / "llms.txt"
with open(verbose_file, "r", encoding="utf-8") as f:
content = f.read()
# Should NOT have double .txt.txt extension
assert ".txt.txt" not in content
# Should have single .txt extension
assert "- [Home Page](https://example.com/_sources/index.txt)" in content
assert "- [Table of Contents](https://example.com/_sources/contents.txt)" in content
def test_uri_template_invalid_fallback(tmp_path):
"""
Test that invalid template falls back to default sources template when
sources_dir exists.
"""
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Mock app with html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
config = Config()
# Invalid template with typo in variable name
config = {
"llms_txt_file": True,
"llms_txt_filename": "llms.txt",
"llms_txt_uri_template": "{base_urll}/{docname}",
"html_baseurl": "https://example.com",
}
writer = FileWriter(config, str(build_dir), MockApp())
page_titles = {
"index": "Home Page",
}
page_order = [("index", ".rst")]
writer.write_verbose_info_to_file(page_order, page_titles, 0, sources_dir)
verbose_file = build_dir / "llms.txt"
with open(verbose_file, "r", encoding="utf-8") as f:
content = f.read()
# Should fallback to default sources template
assert "- [Home Page](https://example.com/_sources/index.rst.txt)" in content