Compare commits

..
Author SHA1 Message Date
Jared DillardandGitHub 0ee28290db Add source code files to llms-full.txt (#24) 2025-08-04 18:22:57 -07:00
Jared DillardandGitHub 368c349d58 Fix image paths to deployed images (#30) 2025-08-03 20:47:38 -07:00
Jared DillardandGitHub c816fb1ea8 Fix issue when source_suffix equals source_link_suffix (#29) 2025-07-31 16:17:02 -07:00
Jared DillardandGitHub 92f810592e Add conda-forge version to index.rst 2025-07-21 22:29:24 -07:00
Jared DillardandGitHub ab0eb1dd29 Add conda-forge version to README.md 2025-07-21 14:57:47 -07:00
Jared DillardandGitHub 57f716b2f9 fix docs path 2025-07-12 21:32:01 -07:00
Jared DillardandGitHub 236822885e configure docs theme 2025-07-12 21:19:19 -07:00
Jared DillardandGitHub 46c2dec254 Use first paragraph as summary by default (#22) 2025-06-22 21:00:40 -07:00
Jared DillardandGitHub b56d93d265 Support source file suffix detection (#21) 2025-06-22 19:28:54 -07:00
Jared Dillard 5db5395889 add downloads badge to docs 2025-05-20 01:38:14 -07:00
Jared Dillard 04b0657dc5 add more badges 2025-05-20 01:35:40 -07:00
Jared Dillard 935a964c7e bump version to 0.2.3 2025-05-20 01:20:36 -07:00
Jared Dillard 51f6c71de3 update changelog 2025-05-20 01:18:44 -07:00
Jared DillardandGitHub 70defd3996 Remove get_and_resolve_toctree method (#19) 2025-05-20 01:08:40 -07:00
Jared DillardandGitHub 9ae05c6c13 Simplify _sources lookup (#18) 2025-05-20 00:50:06 -07:00
Jared Dillard 5581979cac make a href 2025-05-18 21:24:14 -07:00
Jared Dillard f70f1a26ec add soft transfer 2025-05-18 21:21:34 -07:00
Jared DillardandGitHub ed50138ae4 Update README.md 2025-05-18 21:12:16 -07:00
Jared DillardandGitHub 8f4d2c07c6 Update docs and README (#17)
* Move readme content to index.rst

* Clean up project name

* Add advanced configuration
2025-05-18 21:10:49 -07:00
14 changed files with 1781 additions and 181 deletions
+40
View File
@@ -1,6 +1,46 @@
Changelog Changelog
========= =========
0.4.0
-----
- Add support for including source code files with :confval:`llms_txt_code_files` and :confval:`llms_txt_code_base_path` configuration options
`#24 <https://github.com/jdillard/sphinx-llms-txt/pull/24>`_
0.3.2
-----
- Fix image paths to deployed images
`#30 <https://github.com/jdillard/sphinx-llms-txt/pull/30>`_
0.3.1
-----
- Fix issue when ``source_suffix`` equals ``source_link_suffix``
`#29 <https://github.com/jdillard/sphinx-llms-txt/pull/29>`_
0.3.0
-----
- Use first paragraph as default for ``llms_txt_summary``
`#22 <https://github.com/jdillard/sphinx-llms-txt/pull/22>`_
0.2.4
-----
- Support source file suffix detection
`#21 <https://github.com/jdillard/sphinx-llms-txt/pull/21>`_
0.2.3
-----
- Remove ``get_and_resolve_toctree`` method
`#19 <https://github.com/jdillard/sphinx-llms-txt/pull/19>`_
- Simplify ``_sources`` lookup
`#18 <https://github.com/jdillard/sphinx-llms-txt/pull/18>`_
- Add sphinx docs
`#16 <https://github.com/jdillard/sphinx-llms-txt/pull/16>`_
0.2.2 0.2.2
----- -----
+3 -1
View File
@@ -1,9 +1,11 @@
# Sphinx llms.txt generator # Sphinx llms.txt generator
A Sphinx extension that generates a summary `llms.txt` file, written in Markdown, and a single combined documentation `llms-full.txt` file, written in reStructuredText. A Sphinx extension that generates a summary `llms.txt` file and a single combined documentation `llms-full.txt` file.
[![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt) [![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg)](https://anaconda.org/conda-forge/sphinx-llms-txt)
[![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt) [![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt)
[![Parallel Safe](https://img.shields.io/badge/parallel%20safe-true-brightgreen)](#)
## Documentation ## Documentation
+53 -1
View File
@@ -125,6 +125,50 @@ You can exclude specific pages from being included in the generated files:
This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption. This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption.
.. _including_code_files:
Including Source Code Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can include source code files from your project at the end of :confval:`llms_txt_full_filename`.
Use include/exclude syntax to precisely control which files are included:
.. code-block:: python
llms_txt_code_files = [
"+:src/**/*.py", # Include all Python files in src
"-:src/**/__pycache__/**", # Exclude Python cache files
]
Pattern syntax:
- **+:pattern**: Include files matching the pattern. Processed first to collect matching files.
- **-:pattern**: Exclude files matching the pattern. Applied to filter out unwanted files.
Code files are processed as follows:
- **Glob patterns**: Use standard glob patterns (``*``, ``**``, ``?``) to match files
- **Relative paths**: Patterns are resolved relative to your Sphinx source directory
- **Formatting**: Each file is presented with a title and syntax-highlighted code block
.. _customizing_code_paths:
Customizing Code File Paths
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By default, the extension automatically detects the relative path from your Sphinx source directory to the git root and strips that prefix from displayed file paths. You can customize this behavior:
.. code-block:: python
# Manually specify base path to strip
llms_txt_code_base_path = "../../"
# Disable path stripping entirely
llms_txt_code_base_path = ""
This helps create cleaner, more readable file paths in the generated documentation.
.. _using_html_baseurl: .. _using_html_baseurl:
Using HTML Base URL Using HTML Base URL
@@ -146,7 +190,7 @@ Integration Examples
Complete Configuration Example Complete Configuration Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's a complete example showing multiple configuration options: Here's a complete example showing multiple :doc:`configuration-values`:
.. code-block:: python .. code-block:: python
@@ -168,3 +212,11 @@ Here's a complete example showing multiple configuration options:
# Content filtering # Content filtering
llms_txt_exclude = ["search", "genindex", "404", "private_*"] llms_txt_exclude = ["search", "genindex", "404", "private_*"]
# Source code inclusion with include/exclude patterns
llms_txt_code_files = [
"+:../../src/**/*.py", # Include Python files
"+:../../config/*.yaml", # Include config files
"-:../../src/**/__pycache__/**", # Exclude cache files
]
llms_txt_code_base_path = "../../"
+6 -1
View File
@@ -15,6 +15,7 @@ import subprocess
project = "sphinx-llms-txt" project = "sphinx-llms-txt"
copyright = "Jared Dillard" copyright = "Jared Dillard"
author = "Jared Dillard" author = "Jared Dillard"
llms_txt_code_files = ["+:../../sphinx_llms_txt/*.py"]
llms_txt_summary = """ llms_txt_summary = """
A Sphinx extension that generates a summary llms.txt file,written in Markdown, A Sphinx extension that generates a summary llms.txt file,written in Markdown,
and a single combined documentation llms-full.txt file, written in reStructuredText. and a single combined documentation llms-full.txt file, written in reStructuredText.
@@ -81,7 +82,11 @@ html_theme = "furo"
# further. For a list of options available for each theme, see the # further. For a list of options available for each theme, see the
# documentation. # documentation.
# #
html_theme_options = {} html_theme_options = {
"source_repository": "https://github.com/jdillard/sphinx-llms-txt/",
"source_branch": "main",
"source_directory": "docs/source/",
}
html_baseurl = "https://sphinx-llms-txt.readthedocs.org/" html_baseurl = "https://sphinx-llms-txt.readthedocs.org/"
+21 -2
View File
@@ -67,8 +67,8 @@ Project Configuration Values
.. confval:: llms_txt_summary .. confval:: llms_txt_summary
- **Type**: string or ``None`` - **Type**: string
- **Default**: ``None`` - **Default**: The first paragraph in the root document, else an empty string
- **Description**: Optional, but recommended, summary description for ``llms.txt``. - **Description**: Optional, but recommended, summary description for ``llms.txt``.
See :ref:`custom_summary`. See :ref:`custom_summary`.
@@ -82,3 +82,22 @@ Project Configuration Values
See :ref:`excluding_content`. See :ref:`excluding_content`.
.. versionadded:: 0.2.1 .. versionadded:: 0.2.1
.. confval:: llms_txt_code_files
- **Type**: list of strings
- **Default**: ``[]``
- **Description**: A list of glob patterns that appends source code files to :confval:`llms_txt_full_filename`.
See :ref:`including_code_files`.
.. versionadded:: 0.4.0
.. confval:: llms_txt_code_base_path
- **Type**: string or ``None``
- **Default**: ``None`` (auto-detect from git root)
- **Description**: Base path to strip from code file paths when displaying titles.
When ``None``, automatically detects the relative path from the Sphinx source
directory to the git root and strips that prefix from file paths.
.. versionadded:: 0.4.0
+1 -1
View File
@@ -31,7 +31,7 @@ Once added, the extension will automatically generate the LLMs.txt files during
See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**. See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**.
How It Works How It Works
----------- ------------
During the Sphinx build process: During the Sphinx build process:
+13 -1
View File
@@ -3,7 +3,7 @@ Sphinx llms.txt Generator
A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Markdown, and a single combined documentation ``llms-full.txt`` file, written in reStructuredText. A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Markdown, and a single combined documentation ``llms-full.txt`` file, written in reStructuredText.
|PyPI version| |PyPI version| |Conda Version| |Downloads| |Parallel Safe| |GitHub Stars|
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
@@ -20,3 +20,15 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar
.. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg .. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg
:target: https://pypi.python.org/pypi/sphinx-llms-txt :target: https://pypi.python.org/pypi/sphinx-llms-txt
:alt: Latest PyPi Version :alt: Latest PyPi Version
.. |Conda Version| image:: https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg
:target: https://anaconda.org/conda-forge/sphinx-llms-txt
:alt: Latest Conda Version
.. |Downloads| image:: https://static.pepy.tech/badge/sphinx-llms-txt/month
:target: https://pepy.tech/project/sphinx-llms-txt
:alt: PyPi Downloads per month
.. |Parallel Safe| image:: https://img.shields.io/badge/parallel%20safe-true-brightgreen
:target: #
:alt: Parallel read/write safe
.. |GitHub Stars| image:: https://img.shields.io/github/stars/jdillard/sphinx-llms-txt?style=social
:target: https://github.com/jdillard/sphinx-llms-txt
:alt: GitHub Repository stars
+27 -4
View File
@@ -12,7 +12,7 @@ from .manager import LLMSFullManager
from .processor import DocumentProcessor from .processor import DocumentProcessor
from .writer import FileWriter from .writer import FileWriter
__version__ = "0.2.2" __version__ = "0.4.0"
# Export classes needed by tests # Export classes needed by tests
__all__ = [ __all__ = [
@@ -25,9 +25,14 @@ __all__ = [
# Global manager instance # Global manager instance
_manager = LLMSFullManager() _manager = LLMSFullManager()
# Store root document first paragraph
_root_first_paragraph = ""
def doctree_resolved(app: Sphinx, doctree, docname: str): def doctree_resolved(app: Sphinx, doctree, docname: str):
"""Called when a docname has been resolved to a document.""" """Called when a docname has been resolved to a document."""
global _root_first_paragraph
# Extract title from the document # Extract title from the document
title = None title = None
# findall() returns a generator, convert to list to check if it has elements # findall() returns a generator, convert to list to check if it has elements
@@ -38,6 +43,14 @@ def doctree_resolved(app: Sphinx, doctree, docname: str):
if title: if title:
_manager.update_page_title(docname, title) _manager.update_page_title(docname, title)
# Extract first paragraph from root document
if docname == app.config.master_doc:
for node in doctree.traverse(nodes.paragraph):
first_para = node.astext()
if first_para:
_root_first_paragraph = first_para
break
def build_finished(app: Sphinx, exception): def build_finished(app: Sphinx, exception):
"""Called when the build is finished.""" """Called when the build is finished."""
@@ -47,17 +60,24 @@ def build_finished(app: Sphinx, exception):
_manager.set_master_doc(app.config.master_doc) _manager.set_master_doc(app.config.master_doc)
_manager.set_app(app) _manager.set_app(app)
# Get the summary - use configured value or extracted first paragraph
summary = app.config.llms_txt_summary
if summary is None:
summary = _root_first_paragraph
# Set up configuration # Set up configuration
config = { config = {
"llms_txt_file": app.config.llms_txt_file, "llms_txt_file": app.config.llms_txt_file,
"llms_txt_filename": app.config.llms_txt_filename, "llms_txt_filename": app.config.llms_txt_filename,
"llms_txt_title": app.config.llms_txt_title, "llms_txt_title": app.config.llms_txt_title,
"llms_txt_summary": app.config.llms_txt_summary, "llms_txt_summary": summary,
"llms_txt_full_file": app.config.llms_txt_full_file, "llms_txt_full_file": app.config.llms_txt_full_file,
"llms_txt_full_filename": app.config.llms_txt_full_filename, "llms_txt_full_filename": app.config.llms_txt_full_filename,
"llms_txt_full_max_size": app.config.llms_txt_full_max_size, "llms_txt_full_max_size": app.config.llms_txt_full_max_size,
"llms_txt_directives": app.config.llms_txt_directives, "llms_txt_directives": app.config.llms_txt_directives,
"llms_txt_exclude": app.config.llms_txt_exclude, "llms_txt_exclude": app.config.llms_txt_exclude,
"llms_txt_code_files": app.config.llms_txt_code_files,
"llms_txt_code_base_path": app.config.llms_txt_code_base_path,
"html_baseurl": getattr(app.config, "html_baseurl", ""), "html_baseurl": getattr(app.config, "html_baseurl", ""),
} }
_manager.set_config(config) _manager.set_config(config)
@@ -86,14 +106,17 @@ def setup(app: Sphinx) -> Dict[str, Any]:
app.add_config_value("llms_txt_title", None, "env") app.add_config_value("llms_txt_title", None, "env")
app.add_config_value("llms_txt_summary", None, "env") app.add_config_value("llms_txt_summary", None, "env")
app.add_config_value("llms_txt_exclude", [], "env") app.add_config_value("llms_txt_exclude", [], "env")
app.add_config_value("llms_txt_code_files", [], "env")
app.add_config_value("llms_txt_code_base_path", None, "env")
# Connect to Sphinx events # Connect to Sphinx events
app.connect("doctree-resolved", doctree_resolved) app.connect("doctree-resolved", doctree_resolved)
app.connect("build-finished", build_finished) app.connect("build-finished", build_finished)
# Reset manager for each build # Reset manager and root paragraph for each build
global _manager global _manager, _root_first_paragraph
_manager = LLMSFullManager() _manager = LLMSFullManager()
_root_first_paragraph = ""
return { return {
"version": __version__, "version": __version__,
+124 -25
View File
@@ -3,7 +3,7 @@ Document collector module for sphinx-llms-txt.
""" """
import fnmatch import fnmatch
from typing import Any, Dict, List from typing import Any, Dict, List, Tuple
from sphinx.environment import BuildEnvironment from sphinx.environment import BuildEnvironment
from sphinx.util import logging from sphinx.util import logging
@@ -19,6 +19,7 @@ class DocumentCollector:
self.master_doc: str = None self.master_doc: str = None
self.env: BuildEnvironment = None self.env: BuildEnvironment = None
self.config: Dict[str, Any] = {} self.config: Dict[str, Any] = {}
self.app = None
def set_master_doc(self, master_doc: str): def set_master_doc(self, master_doc: str):
"""Set the master document name.""" """Set the master document name."""
@@ -37,8 +38,79 @@ class DocumentCollector:
"""Set configuration options.""" """Set configuration options."""
self.config = config self.config = config
def get_page_order(self) -> List[str]: def set_app(self, app):
"""Get the correct page order from the toctree structure.""" """Set the Sphinx application reference."""
self.app = app
def _get_source_suffixes(self):
"""Get all valid source file suffixes from Sphinx configuration.
Returns:
list: List of source file suffixes (e.g., ['.rst', '.md', '.txt'])
"""
if not self.app:
return [".rst"] # Default fallback
source_suffix = self.app.config.source_suffix
if isinstance(source_suffix, dict):
return list(source_suffix.keys())
elif isinstance(source_suffix, list):
return source_suffix
else:
return [source_suffix] # String format
def _get_docname_suffix(self, docname: str, sources_dir) -> str:
"""
Determine the source suffix for a given docname by checking which
file exists.
Args:
docname: The document name to check
sources_dir: Path to the _sources directory
Returns:
The source suffix if found, or None if no matching file exists
"""
if not sources_dir or not sources_dir.exists():
return None
# Get the source link suffix from Sphinx config
source_link_suffix = ""
if self.app and hasattr(self.app.config, "html_sourcelink_suffix"):
source_link_suffix = self.app.config.html_sourcelink_suffix
# Handle empty string case specially
if source_link_suffix == "":
source_link_suffix = "" # Keep it empty
elif not source_link_suffix.startswith("."):
source_link_suffix = "." + source_link_suffix
# Get the source file suffixes from Sphinx config
source_suffixes = self._get_source_suffixes()
# Try to find the source file with any of the valid source suffixes
for src_suffix in source_suffixes:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
candidate_file = sources_dir / f"{docname}{src_suffix}"
else:
candidate_file = (
sources_dir / f"{docname}{src_suffix}{source_link_suffix}"
)
if candidate_file.exists():
return src_suffix
return None
def get_page_order(self, sources_dir=None) -> List[Tuple[str, str]]:
"""Get the correct page order from the toctree structure.
Args:
sources_dir: Optional path to _sources directory for suffix detection
Returns:
List of tuples (docname, source_suffix) in toctree order
"""
if not self.env or not self.master_doc: if not self.env or not self.master_doc:
return [] return []
@@ -52,9 +124,12 @@ class DocumentCollector:
visited.add(docname) visited.add(docname)
# Add the current document # Add the current document with its suffix
if docname not in page_order: if docname not in [doc for doc, _ in page_order]:
page_order.append(docname) suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
# Check for toctree entries in this document # Check for toctree entries in this document
try: try:
@@ -65,20 +140,33 @@ class DocumentCollector:
): ):
for child_docname in self.env.toctree_includes[docname]: for child_docname in self.env.toctree_includes[docname]:
collect_from_toctree(child_docname) collect_from_toctree(child_docname)
else: # Try to use dependencies to find related documents
# Fallback: try to resolve and parse the toctree elif (
toctree = self.env.get_and_resolve_toctree(docname, None) hasattr(self.env, "dependencies")
if toctree: and docname in self.env.dependencies
from docutils import nodes ):
# Extract the dependent documents from the dependencies dict
for node in list(toctree.findall(nodes.reference)): for child_docname in self.env.dependencies[docname]:
if "refuri" in node.attributes: # Only add documents actually in the document set
refuri = node.attributes["refuri"]
if refuri and refuri.endswith(".html"):
child_docname = refuri[:-5] # Remove .html
if ( if (
child_docname != docname hasattr(self.env, "all_docs")
): # Avoid circular references and child_docname in self.env.all_docs
):
collect_from_toctree(child_docname)
# Fallback to titles or other available references
elif hasattr(self.env, "titles") and hasattr(self.env, "all_docs"):
# Get all document names
all_docnames = list(self.env.all_docs.keys())
# Look for documents that might be related (have similar paths)
current_prefix = "/".join(docname.split("/")[:-1])
if current_prefix:
for child_docname in all_docnames:
# Documents in the same directory might be related
if (
child_docname.startswith(current_prefix)
and child_docname != docname
):
collect_from_toctree(child_docname) collect_from_toctree(child_docname)
except Exception as e: except Exception as e:
logger.debug(f"Could not get toctree for {docname}: {e}") logger.debug(f"Could not get toctree for {docname}: {e}")
@@ -88,22 +176,33 @@ class DocumentCollector:
# Add any remaining documents not in the toctree (sorted) # Add any remaining documents not in the toctree (sorted)
if hasattr(self.env, "all_docs"): if hasattr(self.env, "all_docs"):
processed_docnames = {doc for doc, _ in page_order}
remaining = sorted( remaining = sorted(
[doc for doc in self.env.all_docs.keys() if doc not in page_order] [
doc
for doc in self.env.all_docs.keys()
if doc not in processed_docnames
]
) )
page_order.extend(remaining) for docname in remaining:
suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
return page_order return page_order
def filter_excluded_pages(self, page_order: List[str]) -> List[str]: def filter_excluded_pages(
self, page_order: List[Tuple[str, str]]
) -> List[Tuple[str, str]]:
"""Filter out excluded pages from the page order.""" """Filter out excluded pages from the page order."""
exclude_patterns = self.config.get("llms_txt_exclude") exclude_patterns = self.config.get("llms_txt_exclude")
if exclude_patterns: if exclude_patterns:
return [ return [
page (docname, suffix)
for page in page_order for docname, suffix in page_order
if not any( if not any(
self._match_exclude_pattern(page, pattern) self._match_exclude_pattern(docname, pattern)
for pattern in exclude_patterns for pattern in exclude_patterns
) )
] ]
+580 -94
View File
@@ -2,8 +2,10 @@
Main manager module for sphinx-llms-txt. Main manager module for sphinx-llms-txt.
""" """
import glob
import subprocess
from pathlib import Path from pathlib import Path
from typing import Any, Dict, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
from sphinx.application import Sphinx from sphinx.application import Sphinx
from sphinx.environment import BuildEnvironment from sphinx.environment import BuildEnvironment
@@ -16,6 +18,104 @@ from .writer import FileWriter
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def _get_git_root(path: Path) -> Optional[Path]:
"""Get the git root directory for a given path."""
try:
result = subprocess.run(
["git", "rev-parse", "--show-toplevel"],
cwd=path,
capture_output=True,
text=True,
check=True,
)
return Path(result.stdout.strip())
except (subprocess.CalledProcessError, FileNotFoundError):
return None
def _get_language_from_extension(file_path: Path) -> str:
"""Map file extension to language identifier for code blocks."""
extension_map = {
".py": "python",
".js": "javascript",
".jsx": "jsx",
".ts": "typescript",
".tsx": "tsx",
".java": "java",
".c": "c",
".cpp": "cpp",
".cc": "cpp",
".cxx": "cpp",
".h": "c",
".hpp": "cpp",
".cs": "csharp",
".php": "php",
".rb": "ruby",
".go": "go",
".rs": "rust",
".swift": "swift",
".kt": "kotlin",
".scala": "scala",
".sh": "bash",
".bash": "bash",
".zsh": "zsh",
".fish": "fish",
".ps1": "powershell",
".html": "html",
".htm": "html",
".xml": "xml",
".css": "css",
".scss": "scss",
".sass": "sass",
".less": "less",
".json": "json",
".yaml": "yaml",
".yml": "yaml",
".toml": "toml",
".ini": "ini",
".cfg": "ini",
".conf": "ini",
".sql": "sql",
".md": "markdown",
".rst": "rst",
".txt": "text",
".dockerfile": "dockerfile",
".dockerignore": "text",
".gitignore": "text",
".gitattributes": "text",
".editorconfig": "ini",
".makefile": "makefile",
".r": "r",
".R": "r",
".m": "matlab",
".pl": "perl",
".lua": "lua",
".vim": "vim",
".vimrc": "vim",
".proto": "protobuf",
".thrift": "thrift",
".graphql": "graphql",
".gql": "graphql",
}
# Get the extension from the file path
ext = file_path.suffix.lower()
# Handle special cases like Makefile, Dockerfile without extension
if not ext:
name = file_path.name.lower()
if name in ["makefile", "gnumakefile"]:
return "makefile"
elif name in ["dockerfile", "dockerfile.dev", "dockerfile.prod"]:
return "dockerfile"
elif name.startswith("dockerfile."):
return "dockerfile"
else:
return "text"
return extension_map.get(ext, "text")
class LLMSFullManager: class LLMSFullManager:
"""Manages the collection and ordering of documentation sources.""" """Manages the collection and ordering of documentation sources."""
@@ -56,6 +156,7 @@ class LLMSFullManager:
def set_app(self, app: Sphinx): def set_app(self, app: Sphinx):
"""Set the Sphinx application reference.""" """Set the Sphinx application reference."""
self.app = app self.app = app
self.collector.set_app(app)
if self.writer: if self.writer:
self.writer.app = app self.writer.app = app
@@ -69,23 +170,7 @@ class LLMSFullManager:
self.processor = DocumentProcessor(self.config, srcdir) self.processor = DocumentProcessor(self.config, srcdir)
self.writer = FileWriter(self.config, outdir, self.app) self.writer = FileWriter(self.config, outdir, self.app)
# Get the correct page order # Find sources directory first so we can pass it to get_page_order
page_order = self.collector.get_page_order()
if not page_order:
logger.warning(
"Could not determine page order, skipping llms-full creation"
)
return
# Apply exclusion filter if configured
page_order = self.collector.filter_excluded_pages(page_order)
# Determine output file name and location
output_filename = self.config.get("llms_txt_full_filename")
output_path = Path(outdir) / output_filename
# Find sources directory
sources_dir = None sources_dir = None
possible_sources = [ possible_sources = [
Path(outdir) / "_sources", Path(outdir) / "_sources",
@@ -104,14 +189,23 @@ class LLMSFullManager:
) )
return return
# Collect all available source files # Get the correct page order with source suffixes
txt_files = {} page_order = self.collector.get_page_order(sources_dir)
for f in sources_dir.glob("**/*.txt"):
logger.debug(f"sphinx-llms-txt: Found source file: {f.stem} at {f}") if not page_order:
txt_files[f.stem] = f logger.warning(
"Could not determine page order, skipping llms-full creation"
)
return
# Apply exclusion filter if configured
page_order = self.collector.filter_excluded_pages(page_order)
# Determine output file name and location
output_filename = self.config.get("llms_txt_full_filename")
output_path = Path(outdir) / output_filename
# Log discovered files and page order # Log discovered files and page order
logger.debug(f"sphinx-llms-txt: Found {len(txt_files)} source files")
logger.debug(f"sphinx-llms-txt: Page order (after exclusion): {page_order}") logger.debug(f"sphinx-llms-txt: Page order (after exclusion): {page_order}")
# Log exclusion patterns # Log exclusion patterns
@@ -119,44 +213,69 @@ class LLMSFullManager:
if exclude_patterns: if exclude_patterns:
logger.debug(f"sphinx-llms-txt: Exclusion patterns: {exclude_patterns}") logger.debug(f"sphinx-llms-txt: Exclusion patterns: {exclude_patterns}")
# Create a mapping from docnames to actual file names # Create a mapping from docnames to source files
docname_to_file = {} docname_to_file = {}
# Try exact matches first # Get the source link suffix from Sphinx config
for docname in page_order: source_link_suffix = (
self.app.config.html_sourcelink_suffix if self.app else ".txt"
)
# Handle empty string case specially
if source_link_suffix == "":
source_link_suffix = "" # Keep it empty
elif not source_link_suffix.startswith("."):
source_link_suffix = "." + source_link_suffix
# Process each (docname, suffix) in the page order
for docname, src_suffix in page_order:
# Skip excluded pages # Skip excluded pages
if any( if exclude_patterns and any(
self.collector._match_exclude_pattern(docname, pattern) self.collector._match_exclude_pattern(docname, pattern)
for pattern in exclude_patterns for pattern in exclude_patterns
): ):
continue continue
if docname in txt_files: # Build the source file path directly using the known suffix
docname_to_file[docname] = txt_files[docname] if src_suffix:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
source_file = sources_dir / f"{docname}{src_suffix}"
expected_suffix = src_suffix
else: else:
# Try with .rst extension source_file = (
if f"{docname}.rst" in txt_files: sources_dir / f"{docname}{src_suffix}{source_link_suffix}"
docname_to_file[docname] = txt_files[f"{docname}.rst"] )
# Try with .txt extension expected_suffix = f"{src_suffix}{source_link_suffix}"
elif f"{docname}.txt" in txt_files:
docname_to_file[docname] = txt_files[f"{docname}.txt"] if source_file.exists():
# Try with underscores instead of hyphens docname_to_file[docname] = source_file
elif docname.replace("-", "_") in txt_files: else:
docname_to_file[docname] = txt_files[docname.replace("-", "_")] logger.warning(
# Try with hyphens instead of underscores f"sphinx-llms-txt: Source file not found for: {docname}."
elif docname.replace("_", "-") in txt_files: f"Expected: {docname}{expected_suffix}"
docname_to_file[docname] = txt_files[docname.replace("_", "-")] )
else:
logger.warning(
f"sphinx-llms-txt: No source suffix determined for: {docname}"
)
# Generate content # Generate content
content_parts = [] content_parts = []
# Track code files for later processing
code_file_parts = []
# Count lines in code files (initially 0)
code_files_line_count = 0
# Add pages in order # Add pages in order
added_files = set() added_files = set()
total_line_count = 0 total_line_count = code_files_line_count
max_lines = self.config.get("llms_txt_full_max_size") max_lines = self.config.get("llms_txt_full_max_size")
abort_due_to_max_lines = False abort_due_to_max_lines = False
for docname in page_order: for docname, _ in page_order:
if docname in docname_to_file: if docname in docname_to_file:
file_path = docname_to_file[docname] file_path = docname_to_file[docname]
content, line_count = self._read_source_file(file_path, docname) content, line_count = self._read_source_file(file_path, docname)
@@ -190,75 +309,118 @@ class LLMSFullManager:
added_files.add(file_path.stem) added_files.add(file_path.stem)
total_line_count += line_count total_line_count += line_count
else: else:
logger.warning(f"sphinx-llm-txt: Source file not found for: {docname}") logger.warning(
f"sphinx-llms-txt: Source file not found for: {docname}. Check that"
f" file exists at _sources/{docname}[suffix]{source_link_suffix}"
)
# Add any remaining files (in alphabetical order) if not aborted # Add any remaining files (in alphabetical order) that aren't in the page order
if not abort_due_to_max_lines: if not abort_due_to_max_lines:
# Apply the same exclusion filter to remaining files # Get all source files in the _sources directory using configured suffixes
exclude_patterns = self.config.get("llms_txt_exclude") source_suffixes = self._get_source_suffixes()
all_source_files = []
for src_suffix in source_suffixes:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
glob_pattern = f"**/*{src_suffix}"
else:
glob_pattern = f"**/*{src_suffix}{source_link_suffix}"
all_source_files.extend(sources_dir.glob(glob_pattern))
# Create a set of files to exclude based on their basename processed_paths = set(file.resolve() for file in docname_to_file.values())
excluded_files = set()
for pattern in exclude_patterns:
if "*" not in pattern and "?" not in pattern:
# For exact patterns, add variants
excluded_files.add(pattern)
excluded_files.add(f"{pattern}.rst")
excluded_files.add(f"{pattern}.txt")
excluded_files.add(pattern.replace("-", "_"))
excluded_files.add(pattern.replace("_", "-"))
# Filter remaining files # Find files that haven't been processed yet
remaining_files = sorted( remaining_source_files = [
[ f for f in all_source_files if f.resolve() not in processed_paths
name
for name in txt_files
if name not in added_files
and name not in excluded_files
and not any(
self.collector._match_exclude_pattern(name, pattern)
for pattern in exclude_patterns
)
] ]
# Sort the remaining files for consistent ordering
remaining_source_files.sort()
if remaining_source_files:
logger.info(
f"Found {len(remaining_source_files)} additional files not in"
f" toctree"
) )
if remaining_files:
logger.info(f"Adding remaining files: {remaining_files}") for file_path in remaining_source_files:
for file_stem in remaining_files: # Extract docname from path by removing the source and link suffixes
file_path = txt_files[file_stem] rel_path = str(file_path.relative_to(sources_dir))
content, line_count = self._read_source_file(file_path, file_stem) docname = None
# Try each source suffix to find which one this file uses
for src_suffix in source_suffixes:
# Avoid duplicate extensions when suffixes match
if src_suffix == source_link_suffix:
combined_suffix = src_suffix
else:
combined_suffix = f"{src_suffix}{source_link_suffix}"
if rel_path.endswith(combined_suffix):
docname = rel_path[: -len(combined_suffix)] # Remove suffix
break
if docname is None:
continue
# Skip excluded docnames
if exclude_patterns and any(
self.collector._match_exclude_pattern(docname, pattern)
for pattern in exclude_patterns
):
logger.debug(f"sphinx-llms-txt: Skipping excluded file: {docname}")
continue
# Read and process the file
content, line_count = self._read_source_file(file_path, docname)
# Check if adding this file would exceed the maximum line count # Check if adding this file would exceed the maximum line count
if max_lines is not None and total_line_count + line_count > max_lines: if max_lines is not None and total_line_count + line_count > max_lines:
break break
# Double-check that this file should be included if content:
should_include = True logger.debug(f"sphinx-llms-txt: Adding remaining file: {docname}")
file_stem = file_path.stem
exclude_patterns = self.config.get("llms_txt_exclude")
if exclude_patterns:
# Check stem against exclusion patterns
if any(
self.collector._match_exclude_pattern(file_stem, pattern)
for pattern in exclude_patterns
):
logger.debug(
"sphinx-llms-txt: Final exclusion check removed remaining"
f" file: {file_stem}"
)
should_include = False
if content and should_include:
content_parts.append(content) content_parts.append(content)
total_line_count += line_count total_line_count += line_count
# Process code files at the end if configured
if not abort_due_to_max_lines:
code_file_parts, processed_file_paths = self._process_code_files()
code_files_line_count = sum(
part.count("\n") + 1 for part in code_file_parts
)
# Check if adding code files would exceed the maximum line count
max_lines = self.config.get("llms_txt_full_max_size")
if (
max_lines is not None
and total_line_count + code_files_line_count > max_lines
):
logger.warning(
f"sphinx-llms-txt: Adding code files would exceed max line limit "
f"({max_lines}). Current: {total_line_count}, "
f"Code files: {code_files_line_count}. Skipping code files."
)
else:
# Add source code files section if there are any code files
if code_file_parts:
section_header = self._create_code_files_section_header(
processed_file_paths
)
content_parts.append(section_header)
content_parts.extend(code_file_parts)
# Add line count for the section header too
total_line_count += (
code_files_line_count + section_header.count("\n") + 1
)
# Check if line limit was exceeded before creating the file # Check if line limit was exceeded before creating the file
max_lines = self.config.get("llms_txt_full_max_size") max_lines = self.config.get("llms_txt_full_max_size")
if abort_due_to_max_lines or ( if abort_due_to_max_lines or (
max_lines is not None and total_line_count > max_lines max_lines is not None and total_line_count > max_lines
): ):
logger.warning( logger.warning(
f"sphinx-llm-txt: Max line limit ({max_lines}) exceeded:" f"sphinx-llms-txt: Max line limit ({max_lines}) exceeded:"
f" {total_line_count} > {max_lines}. " f" {total_line_count} > {max_lines}. "
f"Not creating llms-full.txt file." f"Not creating llms-full.txt file."
) )
@@ -325,5 +487,329 @@ class LLMSFullManager:
return content_str, line_count + 1 return content_str, line_count + 1
except Exception as e: except Exception as e:
logger.error(f"sphinx-llm-txt: Error reading source file {file_path}: {e}") logger.error(f"sphinx-llms-txt: Error reading source file {file_path}: {e}")
return "", 0 return "", 0
def _get_source_suffixes(self):
"""Get all valid source file suffixes from Sphinx configuration.
Returns:
list: List of source file suffixes (e.g., ['.rst', '.md', '.txt'])
"""
if not self.app:
return [".rst"] # Default fallback
source_suffix = self.app.config.source_suffix
if isinstance(source_suffix, dict):
return list(source_suffix.keys())
elif isinstance(source_suffix, list):
return source_suffix
else:
return [source_suffix] # String format
def _process_code_files(self) -> Tuple[List[str], List[Path]]:
"""Process code files specified in llms_txt_code_files configuration.
Supports include/exclude patterns with +:/- : prefixes:
- '+:pattern' = include files matching pattern
- '-:pattern' = exclude files matching pattern
- 'pattern' (no prefix) = ignored (no special handling)
Returns:
Tuple of (formatted code block strings, list of processed file paths)
"""
code_file_patterns = self.config.get("llms_txt_code_files", [])
if not code_file_patterns:
return [], []
# Parse patterns into include and exclude lists
include_patterns = []
exclude_patterns = []
for pattern in code_file_patterns:
if pattern.startswith("-:"):
exclude_patterns.append(pattern[2:]) # Remove the '-:' prefix
elif pattern.startswith("+:"):
include_patterns.append(pattern[2:]) # Remove the '+:' prefix
else:
# No prefix = log warning about ignored pattern
logger.warning(
f"sphinx-llms-txt: Code file pattern '{pattern}' ignored."
f"Use '+:{pattern}' to include or '-:{pattern}' to exclude."
)
# If no include patterns specified, nothing to process
if not include_patterns:
return [], []
code_parts = []
processed_files = set()
all_matching_files = set()
# First, collect all files matching include patterns
for pattern in include_patterns:
# Resolve pattern relative to source directory
if self.srcdir:
pattern_path = Path(self.srcdir) / pattern
else:
pattern_path = Path(pattern)
# Use glob to find matching files
matching_files = glob.glob(str(pattern_path), recursive=True)
for file_path_str in matching_files:
file_path = Path(file_path_str)
if file_path.is_file(): # Only add files, not directories
all_matching_files.add(file_path.resolve())
# Filter out files matching exclude patterns
filtered_files = set()
for file_path in all_matching_files:
should_exclude = False
for exclude_pattern in exclude_patterns:
# Resolve exclude pattern relative to source directory
if self.srcdir:
exclude_pattern_path = Path(self.srcdir) / exclude_pattern
else:
exclude_pattern_path = Path(exclude_pattern)
# Check if this file matches the exclude pattern
exclude_matches = glob.glob(str(exclude_pattern_path), recursive=True)
if str(file_path) in exclude_matches:
should_exclude = True
logger.debug(
f"sphinx-llms-txt: Excluding code file: {file_path} "
f"(matched pattern: {exclude_pattern})"
)
break
if not should_exclude:
filtered_files.add(file_path)
# Sort files for consistent ordering
sorted_files = sorted(filtered_files)
for file_path in sorted_files:
# Skip if already processed (shouldn't happen with set, but safety check)
if file_path in processed_files:
continue
try:
# Read the file content
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
content = f.read()
# Get language identifier
language = _get_language_from_extension(file_path)
# Get relative path from source directory for title
if self.srcdir:
try:
title = file_path.relative_to(Path(self.srcdir))
# Strip base path if configured,
# or auto-detect from git root
base_path = self.config.get("llms_txt_code_base_path")
if base_path is None:
# Auto-detect: try to make path relative to git root
git_root = _get_git_root(Path(self.srcdir))
if git_root:
try:
# Get srcdir relative to git root
srcdir_relative = Path(self.srcdir).relative_to(
git_root
)
# Calculate relative path from srcdir to
# git root
if srcdir_relative != Path("."):
# Count directory levels to go up
up_levels = len(srcdir_relative.parts)
base_path = "../" * up_levels
else:
base_path = None
except ValueError:
base_path = None
if base_path:
title_str = str(title)
if title_str.startswith(base_path):
title = Path(title_str[len(base_path) :])
except ValueError:
# File is not relative to srcdir, use filename
title = file_path.name
else:
title = file_path.name
# Format as code block with equals underline
title_str = str(title)
equals_line = "=" * len(title_str)
# Indent the content for reStructuredText code-block directive
indented_content = "\n".join(
f" {line}" if line.strip() else ""
for line in content.splitlines()
)
code_block = f"""
{title_str}
{equals_line}
.. code-block:: {language}
{indented_content}"""
code_parts.append(code_block)
processed_files.add(file_path)
logger.debug(f"sphinx-llms-txt: Added code file: {title}")
except Exception as e:
logger.warning(
f"sphinx-llms-txt: Error reading code file {file_path}: {e}"
)
continue
return code_parts, sorted(processed_files)
def _create_code_files_section_header(self, file_paths: List[Path] = None) -> str:
"""Create the section header for source code files.
Args:
file_paths: List of file paths that were added to generate tree view
Returns:
String containing the section header with title, underlines, description,
and file tree
"""
section_title = "Source Code Files"
star_line = "*" * len(section_title)
description = "This section contains source code files from the project repository. These files are included to provide implementation context and technical details that complement the documentation above." # noqa: E501
header = f"""
{star_line}
{section_title}
{star_line}
{description}"""
# Add file tree if file paths are provided
if file_paths:
tree_display = self._generate_file_tree(file_paths)
header += f"""
**Files included:**
.. code-block:: text
{tree_display}"""
return header
def _generate_file_tree(self, file_paths: List[Path]) -> str:
"""Generate a tree-like representation of file paths.
Args:
file_paths: List of file paths to display in tree format
Returns:
String containing indented tree representation of the files
"""
if not file_paths:
return ""
# Convert to relative paths if possible and create tree structure
tree_data = {}
for file_path in sorted(file_paths):
# Get relative path from source directory for display
if self.srcdir:
try:
rel_path = file_path.relative_to(Path(self.srcdir))
# Apply base path stripping logic similar to code processing
base_path = self.config.get("llms_txt_code_base_path")
if base_path is None:
# Auto-detect: try to make path relative to git root
git_root = _get_git_root(Path(self.srcdir))
if git_root:
try:
# Get srcdir relative to git root
srcdir_relative = Path(self.srcdir).relative_to(
git_root
)
# Calculate relative path from srcdir to git root
if srcdir_relative != Path("."):
# Count directory levels to go up
up_levels = len(srcdir_relative.parts)
base_path = "../" * up_levels
else:
base_path = None
except ValueError:
base_path = None
if base_path:
rel_path_str = str(rel_path)
if rel_path_str.startswith(base_path):
rel_path = Path(rel_path_str[len(base_path) :])
except ValueError:
# File is not relative to srcdir, use filename
rel_path = Path(file_path.name)
else:
rel_path = Path(file_path.name)
# Build nested dictionary structure
parts = rel_path.parts
current = tree_data
for part in parts[:-1]: # All but the last part (directories)
if part not in current:
current[part] = {}
current = current[part]
# Add the file (last part)
if parts:
current[parts[-1]] = None # None indicates it's a file
# Convert tree structure to string representation
lines = []
self._format_tree_node(tree_data, lines, "", True)
# Indent each line for reStructuredText code block
indented_lines = [f" {line}" for line in lines]
return "\n".join(indented_lines)
def _format_tree_node(
self, node: dict, lines: List[str], prefix: str, is_root: bool
):
"""Recursively format tree nodes into lines with proper tree characters.
Args:
node: Dictionary representing the tree structure
lines: List to append formatted lines to
prefix: Current prefix for indentation and tree characters
is_root: Whether this is the root level (no tree characters)
"""
if not node:
return
items = sorted(node.items())
for i, (name, subtree) in enumerate(items):
is_last = i == len(items) - 1
if is_root:
# Root level - no tree characters
current_prefix = ""
next_prefix = ""
else:
# Use tree characters
current_prefix = prefix + ("└── " if is_last else "├── ")
next_prefix = prefix + (" " if is_last else "")
lines.append(current_prefix + name)
# Recursively handle subdirectories
if subtree is not None: # It's a directory
self._format_tree_node(subtree, lines, next_prefix, False)
+48 -6
View File
@@ -94,8 +94,14 @@ class DocumentProcessor:
if not base_url: if not base_url:
return path return path
# Ensure base URL ends with slash
if not base_url.endswith("/"): if not base_url.endswith("/"):
base_url += "/" base_url += "/"
# Remove leading slash from path to avoid double slashes
if path.startswith("/"):
path = path[1:]
return f"{base_url}{path}" return f"{base_url}{path}"
def _is_absolute_or_url(self, path: str) -> bool: def _is_absolute_or_url(self, path: str) -> bool:
@@ -137,8 +143,41 @@ class DocumentProcessor:
prefix = match.group(1) # The entire directive prefix including whitespace prefix = match.group(1) # The entire directive prefix including whitespace
path = match.group(3).strip() # The path argument path = match.group(3).strip() # The path argument
# Only process relative paths, not absolute paths or URLs # Handle URLs and data URIs - leave unchanged
if not self._is_absolute_or_url(path): if path.startswith(("http://", "https://", "data:")):
return match.group(0)
# For ALL paths, check if image exists in _images first
# Extract filename from the path
filename = os.path.basename(path)
# Check if image exists in _images directory
# First determine the build directory from source_path
build_dir = None
if "_sources" in str(source_path):
# Extract build directory (parent of _sources)
path_parts = str(source_path).split("_sources/")
if len(path_parts) > 1:
build_dir = path_parts[0].rstrip("/")
# If we can determine the build directory, check if image exists in _images
if build_dir:
images_path = os.path.join(build_dir, "_images", filename)
if os.path.exists(images_path):
# Image exists in _images, use _images path
full_path = f"/_images/{filename}"
# Add base URL if configured
full_path = self._add_base_url(full_path, base_url)
return f"{prefix}{full_path}"
# Image doesn't exist in _images, handle based on path type
# Handle absolute paths (starting with /) - add base URL if configured
if path.startswith("/"):
# Add base URL to absolute paths if configured
full_path = self._add_base_url(path, base_url)
return f"{prefix}{full_path}"
# Handle relative paths with original logic for backward compatibility
# Special case for test files # Special case for test files
if is_test: if is_test:
# Add subdir/ prefix to match test expectations # Add subdir/ prefix to match test expectations
@@ -168,9 +207,7 @@ class DocumentProcessor:
elif rel_doc_dir: elif rel_doc_dir:
# Join with the original path to form full path relative # Join with the original path to form full path relative
# to srcdir # to srcdir
full_path = os.path.normpath( full_path = os.path.normpath(os.path.join(rel_doc_dir, path))
os.path.join(rel_doc_dir, path)
)
else: else:
full_path = path full_path = path
@@ -180,7 +217,12 @@ class DocumentProcessor:
# Return the updated directive with the full path # Return the updated directive with the full path
return f"{prefix}{full_path}" return f"{prefix}{full_path}"
# If we couldn't resolve the path or it's already absolute, return unchanged # Fallback for relative paths - add base URL if configured
else:
full_path = self._add_base_url(path, base_url)
return f"{prefix}{full_path}"
# If we couldn't resolve the path, return unchanged
return match.group(0) return match.group(0)
# Replace directive paths in the content # Replace directive paths in the content
+12 -5
View File
@@ -3,7 +3,7 @@ File writer module for sphinx-llms-txt.
""" """
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List from typing import Any, Dict, List, Tuple, Union
from sphinx.application import Sphinx from sphinx.application import Sphinx
from sphinx.util import logging from sphinx.util import logging
@@ -42,19 +42,19 @@ class FileWriter:
) )
return True return True
except Exception as e: except Exception as e:
logger.error(f"sphinx-llm-txt: Error writing combined sources file: {e}") logger.error(f"sphinx-llms-txt: Error writing combined sources file: {e}")
return False return False
def write_verbose_info_to_file( def write_verbose_info_to_file(
self, self,
page_order: List[str], page_order: Union[List[str], List[Tuple[str, str]]],
page_titles: Dict[str, str], page_titles: Dict[str, str],
total_line_count: int = 0, total_line_count: int = 0,
) -> bool: ) -> bool:
"""Write summary information to the llms.txt file. """Write summary information to the llms.txt file.
Args: Args:
page_order: Ordered list of document names page_order: Ordered list of document names or (docname, suffix) tuples
page_titles: Dictionary mapping docnames to titles page_titles: Dictionary mapping docnames to titles
total_line_count: Total number of lines in the combined content total_line_count: Total number of lines in the combined content
@@ -88,6 +88,8 @@ class FileWriter:
if description: if description:
# Trim leading and trailing whitespace # Trim leading and trailing whitespace
description = description.strip() description = description.strip()
if description:
# Only add blockquote if description is not empty
# Replace newlines with newline + blockquote marker to maintain # Replace newlines with newline + blockquote marker to maintain
# blockquote formatting # blockquote formatting
description = description.replace("\n", "\n> ") description = description.replace("\n", "\n> ")
@@ -100,7 +102,12 @@ class FileWriter:
if not base_url.endswith("/"): if not base_url.endswith("/"):
base_url += "/" base_url += "/"
for docname in page_order: for item in page_order:
# Handle both old format (str) and new format (tuple)
if isinstance(item, tuple):
docname, _ = item
else:
docname = item
title = page_titles.get(docname, docname) title = page_titles.get(docname, docname)
f.write(f"- [{title}]({base_url}{docname}.html)\n") f.write(f"- [{title}]({base_url}{docname}.html)\n")
+660
View File
@@ -334,3 +334,663 @@ def test_write_verbose_info_with_baseurl(tmp_path):
assert "- [Home Page](https://example.org/index.html)" in content assert "- [Home Page](https://example.org/index.html)" in content
assert "- [About Us](https://example.org/about.html)" in content assert "- [About Us](https://example.org/about.html)" in content
def test_get_source_suffixes_with_dict():
"""Test _get_source_suffixes method with dict source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with dict source_suffix
class MockApp:
class Config:
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert set(suffixes) == {".rst", ".md", ".txt"}
def test_get_source_suffixes_with_list():
"""Test _get_source_suffixes method with list source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with list source_suffix
class MockApp:
class Config:
source_suffix = [".rst", ".md"]
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst", ".md"]
def test_get_source_suffixes_with_string():
"""Test _get_source_suffixes method with string source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with string source_suffix
class MockApp:
class Config:
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"]
def test_get_source_suffixes_no_app():
"""Test _get_source_suffixes method with no app set."""
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"] # Default fallback
def test_html_sourcelink_suffix_default():
"""Test html_sourcelink_suffix defaults to .txt when no app is set."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with default .txt suffix
test_file = f"{sources_dir}/index.rst.txt"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .txt as the default suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed (check if output file exists)
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_custom():
"""Test html_sourcelink_suffix uses custom value from Sphinx config."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with custom html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = "source"
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with custom .source suffix
test_file = f"{sources_dir}/index.rst.source"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .source as the custom suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_with_dot():
"""Test html_sourcelink_suffix adds dot if missing."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with html_sourcelink_suffix without leading dot
class MockApp:
class Config:
html_sourcelink_suffix = "src" # No leading dot
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with .src suffix (dot should be added automatically)
test_file = f"{sources_dir}/index.rst.src"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it adds the dot and finds the file
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_mixed_source_file_formats():
"""Test handling of mixed source file formats (.rst, .md, .txt)."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with multiple source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create test source files with different formats
files_to_create = [
f"{sources_dir}/page1.rst.txt",
f"{sources_dir}/page2.md.txt",
f"{sources_dir}/page3.txt.txt",
]
for test_file in files_to_create:
with open(test_file, "w") as f:
f.write(f"Content for {os.path.basename(test_file)}")
# Mock env with all documents
class MockEnv:
all_docs = {"page1": None, "page2": None, "page3": None}
titles = {
"page1": type("TitleNode", (), {"astext": lambda: "Page 1"})(),
"page2": type("TitleNode", (), {"astext": lambda: "Page 2"})(),
"page3": type("TitleNode", (), {"astext": lambda: "Page 3"})(),
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("page1")
# Test that all file formats are found and processed
manager.combine_sources(outdir, srcdir)
# Verify the output file was created and contains content from all formats
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# Should contain content from all three files
assert "Content for page1.rst.txt" in content
assert "Content for page2.md.txt" in content
assert "Content for page3.txt.txt" in content
def test_source_suffix_detection_priority():
"""Test source suffix detection tries formats in correct order for docnames."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with ordered source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = [".rst", ".md"] # rst has priority over md
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create both .rst and .md versions of the same document
# Only create files for the specific docname "index"
rst_file = f"{sources_dir}/index.rst.txt"
md_file = f"{sources_dir}/index.md.txt"
with open(rst_file, "w") as f:
f.write("RST content for index")
with open(md_file, "w") as f:
f.write("Markdown content for index")
# Mock env with only the index document
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Index Page"})()
}
toctree_includes = {"index": []}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test the priority behavior
manager.combine_sources(outdir, srcdir)
# Check that output file was created
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# The system should prefer RST over MD for the "index" docname
# But since both files exist and the second phase adds remaining files,
# both will be included. The test verifies that RST appears first
# (indicating it was found first in the priority order)
assert "RST content for index" in content
# Find positions to verify order
rst_pos = content.find("RST content for index")
md_pos = content.find("Markdown content for index")
# RST should come before MD (due to priority in toctree processing)
assert rst_pos < md_pos, "RST content should appear before MD content"
def test_summary_default_uses_first_paragraph():
"""
Test that summary defaults to first paragraph of root document when not configured.
"""
from docutils import nodes
from docutils.frontend import OptionParser
from docutils.parsers.rst import Parser
from docutils.utils import new_document
from sphinx_llms_txt import build_finished, doctree_resolved
# Create a proper document with settings
settings = OptionParser(components=(Parser,)).get_default_values()
doctree = new_document("<rst-doc>", settings)
title = nodes.title(text="Test Title")
paragraph = nodes.paragraph(
text="This is the first paragraph that should be used as summary."
)
doctree.append(title)
doctree.append(paragraph)
# Mock Sphinx app
class MockApp:
class Config:
master_doc = "index"
llms_txt_summary = None # Not configured
llms_txt_file = True
llms_txt_filename = "llms.txt"
llms_txt_title = None
llms_txt_full_file = True
llms_txt_full_filename = "llms-full.txt"
llms_txt_full_max_size = None
llms_txt_directives = []
llms_txt_exclude = []
llms_txt_code_files = []
llms_txt_code_base_path = None
html_baseurl = ""
config = Config()
outdir = "/tmp/build"
srcdir = "/tmp/source"
class Env:
titles = {
"index": type("TitleNode", (), {"astext": lambda self: "Test Title"})()
}
env = Env()
app = MockApp()
# Reset the global state
import sphinx_llms_txt
sphinx_llms_txt._root_first_paragraph = ""
# Call doctree_resolved to extract the first paragraph
doctree_resolved(app, doctree, "index")
# Verify the first paragraph was extracted
assert (
sphinx_llms_txt._root_first_paragraph
== "This is the first paragraph that should be used as summary."
)
# Mock the manager methods to avoid actual file operations
original_combine_sources = sphinx_llms_txt._manager.combine_sources
sphinx_llms_txt._manager.combine_sources = lambda outdir, srcdir: None
# Call build_finished and verify the summary is set correctly
build_finished(app, None)
# Check that the summary was properly configured
assert (
sphinx_llms_txt._manager.config["llms_txt_summary"]
== "This is the first paragraph that should be used as summary."
)
# Restore original method
sphinx_llms_txt._manager.combine_sources = original_combine_sources
def test_code_files_include_exclude_patterns(tmp_path):
"""Test the +/- pattern syntax for llms_txt_code_files configuration."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
cache_dir = docs_dir / "__pycache__"
cache_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
(docs_dir / "guide.rst").write_text("Guide RST content")
(docs_dir / "backup.bak").write_text("Backup file content")
(cache_dir / "compiled.pyc").write_text("Compiled Python")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with include/exclude patterns
config = {
"llms_txt_code_files": [
"+:docs/**/*.rst", # Include all RST files in docs
"-:docs/**/__pycache__/**", # Exclude pycache files
"-:docs/**/*.bak", # Exclude backup files
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Verify we have the expected number of files
assert len(code_parts) == 2, f"Expected 2 files, got {len(code_parts)}"
# Extract file titles from code blocks
titles = []
for part in code_parts:
lines = part.strip().split("\n")
if lines:
titles.append(lines[0])
# Verify expected files are included
assert "docs/example.rst" in titles
assert "docs/guide.rst" in titles
# Verify excluded files are not present
content = "\n".join(code_parts)
assert "backup.bak" not in content
assert "__pycache__" not in content
assert "compiled.pyc" not in content
def test_code_files_exclude_only_patterns(tmp_path):
"""Test that exclude-only patterns result in no files being included."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with only exclude patterns
config = {
"llms_txt_code_files": [
"-:docs/**/*.rst", # Only exclude pattern, no includes
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should have no files with exclude-only patterns
assert len(code_parts) == 0, "Should have no files with exclude-only patterns"
def test_code_files_no_prefix_patterns(tmp_path):
"""Test that patterns without prefix are ignored."""
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
(docs_dir / "backup.bak").write_text("Backup file content")
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with no prefix (should be ignored)
config = {
"llms_txt_code_files": [
"docs/**/*.rst", # No prefix = ignored
"+:docs/**/*.rst", # Include RST files
"-:docs/**/*.bak", # Exclude backup files
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should include RST files (from +: pattern) and exclude BAK files (from -: pattern)
assert len(code_parts) == 1, "Should include RST files and exclude BAK files"
content = "\n".join(code_parts)
assert "Example RST content" in content
assert "backup.bak" not in content
def test_code_files_ignored_patterns(tmp_path, caplog):
"""Test that patterns without +: or -: prefix log a warning and are ignored."""
from unittest.mock import patch
from sphinx_llms_txt.manager import LLMSFullManager
# Create test directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
docs_dir = src_dir / "docs"
docs_dir.mkdir()
# Create test files
(docs_dir / "example.rst").write_text("Example RST content")
# Use a mock to capture the warning message directly
captured_warnings = []
def capture_warning(message, *args, **kwargs):
captured_warnings.append(message)
# Patch the logger to capture warnings
with patch("sphinx_llms_txt.manager.logger.warning", side_effect=capture_warning):
# Create manager and set source directory
manager = LLMSFullManager()
manager.srcdir = str(src_dir)
# Test configuration with only no-prefix patterns (should result in no files)
config = {
"llms_txt_code_files": [
"docs/**/*.rst", # No prefix = ignored with warning
]
}
manager.set_config(config)
# Process code files
code_parts, _ = manager._process_code_files()
# Should have no files since the pattern without prefix is ignored
assert (
len(code_parts) == 0
), "Should have no files when only using patterns without prefix"
# Check that a warning was logged
assert (
len(captured_warnings) == 1
), f"Expected 1 warning, got {len(captured_warnings)}"
assert (
"Code file pattern 'docs/**/*.rst' ignored." in captured_warnings[0]
), f"Warning message should contain expected text. Got: {captured_warnings[0]}"
+156 -3
View File
@@ -102,7 +102,7 @@ def test_process_path_directives_with_html_baseurl(tmp_path):
def test_process_path_directives_absolute_urls(tmp_path): def test_process_path_directives_absolute_urls(tmp_path):
"""Test that absolute URLs are not modified.""" """Test that absolute URLs are not modified but absolute paths get base URL."""
# Create a processor # Create a processor
config = { config = {
"llms_txt_directives": [], "llms_txt_directives": [],
@@ -127,10 +127,17 @@ def test_process_path_directives_absolute_urls(tmp_path):
with open(source_file, "w", encoding="utf-8") as f: with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content) f.write(source_content)
# Process the directives (should remain unchanged) # Process the directives
processed_content = processor._process_path_directives(source_content, source_file) processed_content = processor._process_path_directives(source_content, source_file)
assert processed_content == source_content # Expected: URLs and data URIs unchanged, absolute paths get base URL
expected_content = (
".. image:: https://othersite.com/images/test.png\n"
".. image:: https://example.com/docs/absolute/path/image.png\n"
".. image:: data:image/png;base64,iVBORw0KG...\n"
)
assert processed_content == expected_content
def test_process_path_directives_custom_directives(tmp_path): def test_process_path_directives_custom_directives(tmp_path):
@@ -251,3 +258,149 @@ def test_process_content_end_to_end(tmp_path):
) )
assert processed_content == expected_content assert processed_content == expected_content
def test_process_path_directives_images_directory(tmp_path):
"""Test that _images directory paths are handled correctly."""
# Create a processor with base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "https://example.com/docs",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create _sources directory to mimic Sphinx output
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Create a source file with various _images directory paths
source_content = (
"Some content.\n"
".. image:: _images/test.png\n" # Relative _images should become /_images
".. image:: /_images/absolute.png\n" # Absolute _images should get base URL
".. figure:: _images/figure.png\n" # Test with figure directive too
" :alt: A test figure\n"
".. image:: images/normal.png\n" # Normal relative path should be unchanged
)
# Create source file in sources directory to simulate Sphinx build output
source_file = sources_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: _images paths should be converted and get base URL
expected_content = (
"Some content.\n"
".. image:: https://example.com/docs/_images/test.png\n"
".. image:: https://example.com/docs/_images/absolute.png\n"
".. figure:: https://example.com/docs/_images/figure.png\n"
" :alt: A test figure\n"
".. image:: https://example.com/docs/images/normal.png\n"
)
assert processed_content == expected_content
def test_process_path_directives_images_directory_no_baseurl(tmp_path):
"""
Test that _images directory paths work correctly without base URL.
Only converts when image exists.
"""
# Create a processor without base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create _sources directory to mimic Sphinx output
build_dir = tmp_path / "build"
build_dir.mkdir()
sources_dir = build_dir / "_sources"
sources_dir.mkdir()
# Create _images directory and one test image
images_dir = build_dir / "_images"
images_dir.mkdir()
(images_dir / "test.png").write_text("fake image content")
# Note: absolute.png is not created, so it won't be converted
# Create a source file with _images directory paths
source_content = (
".. image:: _images/test.png\n" # Should become /_images (image exists)
".. image:: /_images/absolute.png\n" # Should stay unchanged (absolute path)
)
# Create source file in sources directory to simulate Sphinx build output
source_file = sources_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: only test.png gets converted because it exists in _images
expected_content = (
".. image:: /_images/test.png\n" # Converted because image exists
".. image:: /_images/absolute.png\n" # Absolute path unchanged
)
assert processed_content == expected_content
def test_process_path_directives_all_absolute_paths_get_baseurl(tmp_path):
"""Test that all absolute paths (starting with /) get base URL prepended."""
# Create a processor with base URL
config = {
"llms_txt_directives": [],
"html_baseurl": "https://mysite.com/docs/",
}
processor = DocumentProcessor(config)
# Create source directory structure
src_dir = tmp_path / "src"
src_dir.mkdir()
processor.srcdir = str(src_dir)
# Create a source file with various absolute paths
source_content = (
".. image:: /static/images/logo.png\n"
".. figure:: /assets/diagrams/flow.svg\n"
".. image:: /media/photos/team.jpg\n"
" :alt: Team photo\n"
".. image:: relative/path.png\n" # This should still get normal processing
)
# Create source file
source_file = src_dir / "page.txt"
with open(source_file, "w", encoding="utf-8") as f:
f.write(source_content)
# Process the directives
processed_content = processor._process_path_directives(source_content, source_file)
# Expected: All absolute paths get base URL prepended
expected_content = (
".. image:: https://mysite.com/docs/static/images/logo.png\n"
".. figure:: https://mysite.com/docs/assets/diagrams/flow.svg\n"
".. image:: https://mysite.com/docs/media/photos/team.jpg\n"
" :alt: Team photo\n"
".. image:: https://mysite.com/docs/relative/path.png\n"
)
assert processed_content == expected_content