Compare commits

...
9 Commits
12 changed files with 753 additions and 67 deletions
+18
View File
@@ -1,6 +1,24 @@
Changelog Changelog
========= =========
0.3.1
-----
- Fix issue when ``source_suffix`` equals ``source_link_suffix``
`#29 <https://github.com/jdillard/sphinx-llms-txt/pull/29>`_
0.3.0
-----
- Use first paragraph as default for ``llms_txt_summary``
`#22 <https://github.com/jdillard/sphinx-llms-txt/pull/22>`_
0.2.4
-----
- Support source file suffix detection
`#21 <https://github.com/jdillard/sphinx-llms-txt/pull/21>`_
0.2.3 0.2.3
----- -----
+2
View File
@@ -3,7 +3,9 @@
A Sphinx extension that generates a summary `llms.txt` file and a single combined documentation `llms-full.txt` file. A Sphinx extension that generates a summary `llms.txt` file and a single combined documentation `llms-full.txt` file.
[![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt) [![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg)](https://anaconda.org/conda-forge/sphinx-llms-txt)
[![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt) [![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt)
[![Parallel Safe](https://img.shields.io/badge/parallel%20safe-true-brightgreen)](#)
## Documentation ## Documentation
+1 -1
View File
@@ -146,7 +146,7 @@ Integration Examples
Complete Configuration Example Complete Configuration Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's a complete example showing multiple :ref:`configuration-values`: Here's a complete example showing multiple :doc:`configuration-values`:
.. code-block:: python .. code-block:: python
+5 -1
View File
@@ -81,7 +81,11 @@ html_theme = "furo"
# further. For a list of options available for each theme, see the # further. For a list of options available for each theme, see the
# documentation. # documentation.
# #
html_theme_options = {} html_theme_options = {
"source_repository": "https://github.com/jdillard/sphinx-llms-txt/",
"source_branch": "main",
"source_directory": "docs/source/",
}
html_baseurl = "https://sphinx-llms-txt.readthedocs.org/" html_baseurl = "https://sphinx-llms-txt.readthedocs.org/"
+2 -2
View File
@@ -67,8 +67,8 @@ Project Configuration Values
.. confval:: llms_txt_summary .. confval:: llms_txt_summary
- **Type**: string or ``None`` - **Type**: string
- **Default**: ``None`` - **Default**: The first paragraph in the root document, else an empty string
- **Description**: Optional, but recommended, summary description for ``llms.txt``. - **Description**: Optional, but recommended, summary description for ``llms.txt``.
See :ref:`custom_summary`. See :ref:`custom_summary`.
+1 -1
View File
@@ -31,7 +31,7 @@ Once added, the extension will automatically generate the LLMs.txt files during
See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**. See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**.
How It Works How It Works
----------- ------------
During the Sphinx build process: During the Sphinx build process:
+13 -1
View File
@@ -3,7 +3,7 @@ Sphinx llms.txt Generator
A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Markdown, and a single combined documentation ``llms-full.txt`` file, written in reStructuredText. A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Markdown, and a single combined documentation ``llms-full.txt`` file, written in reStructuredText.
|PyPI version| |PyPI version| |Conda Version| |Downloads| |Parallel Safe| |GitHub Stars|
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
@@ -20,3 +20,15 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar
.. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg .. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg
:target: https://pypi.python.org/pypi/sphinx-llms-txt :target: https://pypi.python.org/pypi/sphinx-llms-txt
:alt: Latest PyPi Version :alt: Latest PyPi Version
.. |Conda Version| image:: https://img.shields.io/conda/vn/conda-forge/sphinx-llms-txt.svg
:target: https://anaconda.org/conda-forge/sphinx-llms-txt
:alt: Latest Conda Version
.. |Downloads| image:: https://static.pepy.tech/badge/sphinx-llms-txt/month
:target: https://pepy.tech/project/sphinx-llms-txt
:alt: PyPi Downloads per month
.. |Parallel Safe| image:: https://img.shields.io/badge/parallel%20safe-true-brightgreen
:target: #
:alt: Parallel read/write safe
.. |GitHub Stars| image:: https://img.shields.io/github/stars/jdillard/sphinx-llms-txt?style=social
:target: https://github.com/jdillard/sphinx-llms-txt
:alt: GitHub Repository stars
+23 -4
View File
@@ -12,7 +12,7 @@ from .manager import LLMSFullManager
from .processor import DocumentProcessor from .processor import DocumentProcessor
from .writer import FileWriter from .writer import FileWriter
__version__ = "0.2.3" __version__ = "0.3.1"
# Export classes needed by tests # Export classes needed by tests
__all__ = [ __all__ = [
@@ -25,9 +25,14 @@ __all__ = [
# Global manager instance # Global manager instance
_manager = LLMSFullManager() _manager = LLMSFullManager()
# Store root document first paragraph
_root_first_paragraph = ""
def doctree_resolved(app: Sphinx, doctree, docname: str): def doctree_resolved(app: Sphinx, doctree, docname: str):
"""Called when a docname has been resolved to a document.""" """Called when a docname has been resolved to a document."""
global _root_first_paragraph
# Extract title from the document # Extract title from the document
title = None title = None
# findall() returns a generator, convert to list to check if it has elements # findall() returns a generator, convert to list to check if it has elements
@@ -38,6 +43,14 @@ def doctree_resolved(app: Sphinx, doctree, docname: str):
if title: if title:
_manager.update_page_title(docname, title) _manager.update_page_title(docname, title)
# Extract first paragraph from root document
if docname == app.config.master_doc:
for node in doctree.traverse(nodes.paragraph):
first_para = node.astext()
if first_para:
_root_first_paragraph = first_para
break
def build_finished(app: Sphinx, exception): def build_finished(app: Sphinx, exception):
"""Called when the build is finished.""" """Called when the build is finished."""
@@ -47,12 +60,17 @@ def build_finished(app: Sphinx, exception):
_manager.set_master_doc(app.config.master_doc) _manager.set_master_doc(app.config.master_doc)
_manager.set_app(app) _manager.set_app(app)
# Get the summary - use configured value or extracted first paragraph
summary = app.config.llms_txt_summary
if summary is None:
summary = _root_first_paragraph
# Set up configuration # Set up configuration
config = { config = {
"llms_txt_file": app.config.llms_txt_file, "llms_txt_file": app.config.llms_txt_file,
"llms_txt_filename": app.config.llms_txt_filename, "llms_txt_filename": app.config.llms_txt_filename,
"llms_txt_title": app.config.llms_txt_title, "llms_txt_title": app.config.llms_txt_title,
"llms_txt_summary": app.config.llms_txt_summary, "llms_txt_summary": summary,
"llms_txt_full_file": app.config.llms_txt_full_file, "llms_txt_full_file": app.config.llms_txt_full_file,
"llms_txt_full_filename": app.config.llms_txt_full_filename, "llms_txt_full_filename": app.config.llms_txt_full_filename,
"llms_txt_full_max_size": app.config.llms_txt_full_max_size, "llms_txt_full_max_size": app.config.llms_txt_full_max_size,
@@ -91,9 +109,10 @@ def setup(app: Sphinx) -> Dict[str, Any]:
app.connect("doctree-resolved", doctree_resolved) app.connect("doctree-resolved", doctree_resolved)
app.connect("build-finished", build_finished) app.connect("build-finished", build_finished)
# Reset manager for each build # Reset manager and root paragraph for each build
global _manager global _manager, _root_first_paragraph
_manager = LLMSFullManager() _manager = LLMSFullManager()
_root_first_paragraph = ""
return { return {
"version": __version__, "version": __version__,
+98 -12
View File
@@ -3,7 +3,7 @@ Document collector module for sphinx-llms-txt.
""" """
import fnmatch import fnmatch
from typing import Any, Dict, List from typing import Any, Dict, List, Tuple
from sphinx.environment import BuildEnvironment from sphinx.environment import BuildEnvironment
from sphinx.util import logging from sphinx.util import logging
@@ -19,6 +19,7 @@ class DocumentCollector:
self.master_doc: str = None self.master_doc: str = None
self.env: BuildEnvironment = None self.env: BuildEnvironment = None
self.config: Dict[str, Any] = {} self.config: Dict[str, Any] = {}
self.app = None
def set_master_doc(self, master_doc: str): def set_master_doc(self, master_doc: str):
"""Set the master document name.""" """Set the master document name."""
@@ -37,8 +38,79 @@ class DocumentCollector:
"""Set configuration options.""" """Set configuration options."""
self.config = config self.config = config
def get_page_order(self) -> List[str]: def set_app(self, app):
"""Get the correct page order from the toctree structure.""" """Set the Sphinx application reference."""
self.app = app
def _get_source_suffixes(self):
"""Get all valid source file suffixes from Sphinx configuration.
Returns:
list: List of source file suffixes (e.g., ['.rst', '.md', '.txt'])
"""
if not self.app:
return [".rst"] # Default fallback
source_suffix = self.app.config.source_suffix
if isinstance(source_suffix, dict):
return list(source_suffix.keys())
elif isinstance(source_suffix, list):
return source_suffix
else:
return [source_suffix] # String format
def _get_docname_suffix(self, docname: str, sources_dir) -> str:
"""
Determine the source suffix for a given docname by checking which
file exists.
Args:
docname: The document name to check
sources_dir: Path to the _sources directory
Returns:
The source suffix if found, or None if no matching file exists
"""
if not sources_dir or not sources_dir.exists():
return None
# Get the source link suffix from Sphinx config
source_link_suffix = ""
if self.app and hasattr(self.app.config, "html_sourcelink_suffix"):
source_link_suffix = self.app.config.html_sourcelink_suffix
# Handle empty string case specially
if source_link_suffix == "":
source_link_suffix = "" # Keep it empty
elif not source_link_suffix.startswith("."):
source_link_suffix = "." + source_link_suffix
# Get the source file suffixes from Sphinx config
source_suffixes = self._get_source_suffixes()
# Try to find the source file with any of the valid source suffixes
for src_suffix in source_suffixes:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
candidate_file = sources_dir / f"{docname}{src_suffix}"
else:
candidate_file = (
sources_dir / f"{docname}{src_suffix}{source_link_suffix}"
)
if candidate_file.exists():
return src_suffix
return None
def get_page_order(self, sources_dir=None) -> List[Tuple[str, str]]:
"""Get the correct page order from the toctree structure.
Args:
sources_dir: Optional path to _sources directory for suffix detection
Returns:
List of tuples (docname, source_suffix) in toctree order
"""
if not self.env or not self.master_doc: if not self.env or not self.master_doc:
return [] return []
@@ -52,9 +124,12 @@ class DocumentCollector:
visited.add(docname) visited.add(docname)
# Add the current document # Add the current document with its suffix
if docname not in page_order: if docname not in [doc for doc, _ in page_order]:
page_order.append(docname) suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
# Check for toctree entries in this document # Check for toctree entries in this document
try: try:
@@ -101,22 +176,33 @@ class DocumentCollector:
# Add any remaining documents not in the toctree (sorted) # Add any remaining documents not in the toctree (sorted)
if hasattr(self.env, "all_docs"): if hasattr(self.env, "all_docs"):
processed_docnames = {doc for doc, _ in page_order}
remaining = sorted( remaining = sorted(
[doc for doc in self.env.all_docs.keys() if doc not in page_order] [
doc
for doc in self.env.all_docs.keys()
if doc not in processed_docnames
]
) )
page_order.extend(remaining) for docname in remaining:
suffix = None
if sources_dir:
suffix = self._get_docname_suffix(docname, sources_dir)
page_order.append((docname, suffix))
return page_order return page_order
def filter_excluded_pages(self, page_order: List[str]) -> List[str]: def filter_excluded_pages(
self, page_order: List[Tuple[str, str]]
) -> List[Tuple[str, str]]:
"""Filter out excluded pages from the page order.""" """Filter out excluded pages from the page order."""
exclude_patterns = self.config.get("llms_txt_exclude") exclude_patterns = self.config.get("llms_txt_exclude")
if exclude_patterns: if exclude_patterns:
return [ return [
page (docname, suffix)
for page in page_order for docname, suffix in page_order
if not any( if not any(
self._match_exclude_pattern(page, pattern) self._match_exclude_pattern(docname, pattern)
for pattern in exclude_patterns for pattern in exclude_patterns
) )
] ]
+97 -33
View File
@@ -56,6 +56,7 @@ class LLMSFullManager:
def set_app(self, app: Sphinx): def set_app(self, app: Sphinx):
"""Set the Sphinx application reference.""" """Set the Sphinx application reference."""
self.app = app self.app = app
self.collector.set_app(app)
if self.writer: if self.writer:
self.writer.app = app self.writer.app = app
@@ -69,23 +70,7 @@ class LLMSFullManager:
self.processor = DocumentProcessor(self.config, srcdir) self.processor = DocumentProcessor(self.config, srcdir)
self.writer = FileWriter(self.config, outdir, self.app) self.writer = FileWriter(self.config, outdir, self.app)
# Get the correct page order # Find sources directory first so we can pass it to get_page_order
page_order = self.collector.get_page_order()
if not page_order:
logger.warning(
"Could not determine page order, skipping llms-full creation"
)
return
# Apply exclusion filter if configured
page_order = self.collector.filter_excluded_pages(page_order)
# Determine output file name and location
output_filename = self.config.get("llms_txt_full_filename")
output_path = Path(outdir) / output_filename
# Find sources directory
sources_dir = None sources_dir = None
possible_sources = [ possible_sources = [
Path(outdir) / "_sources", Path(outdir) / "_sources",
@@ -104,6 +89,22 @@ class LLMSFullManager:
) )
return return
# Get the correct page order with source suffixes
page_order = self.collector.get_page_order(sources_dir)
if not page_order:
logger.warning(
"Could not determine page order, skipping llms-full creation"
)
return
# Apply exclusion filter if configured
page_order = self.collector.filter_excluded_pages(page_order)
# Determine output file name and location
output_filename = self.config.get("llms_txt_full_filename")
output_path = Path(outdir) / output_filename
# Log discovered files and page order # Log discovered files and page order
logger.debug(f"sphinx-llms-txt: Page order (after exclusion): {page_order}") logger.debug(f"sphinx-llms-txt: Page order (after exclusion): {page_order}")
@@ -115,8 +116,19 @@ class LLMSFullManager:
# Create a mapping from docnames to source files # Create a mapping from docnames to source files
docname_to_file = {} docname_to_file = {}
# Process each docname in the page order # Get the source link suffix from Sphinx config
for docname in page_order: source_link_suffix = (
self.app.config.html_sourcelink_suffix if self.app else ".txt"
)
# Handle empty string case specially
if source_link_suffix == "":
source_link_suffix = "" # Keep it empty
elif not source_link_suffix.startswith("."):
source_link_suffix = "." + source_link_suffix
# Process each (docname, suffix) in the page order
for docname, src_suffix in page_order:
# Skip excluded pages # Skip excluded pages
if exclude_patterns and any( if exclude_patterns and any(
self.collector._match_exclude_pattern(docname, pattern) self.collector._match_exclude_pattern(docname, pattern)
@@ -124,15 +136,28 @@ class LLMSFullManager:
): ):
continue continue
# Construct expected source file path directly from docname # Build the source file path directly using the known suffix
source_file = sources_dir / f"{docname}.rst.txt" if src_suffix:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
source_file = sources_dir / f"{docname}{src_suffix}"
expected_suffix = src_suffix
else:
source_file = (
sources_dir / f"{docname}{src_suffix}{source_link_suffix}"
)
expected_suffix = f"{src_suffix}{source_link_suffix}"
if source_file.exists(): if source_file.exists():
docname_to_file[docname] = source_file docname_to_file[docname] = source_file
else: else:
logger.warning( logger.warning(
f"sphinx-llm-txt: Source file not found for: {docname}. Expected" f"sphinx-llms-txt: Source file not found for: {docname}."
f" at {source_file}" f"Expected: {docname}{expected_suffix}"
)
else:
logger.warning(
f"sphinx-llms-txt: No source suffix determined for: {docname}"
) )
# Generate content # Generate content
@@ -144,7 +169,7 @@ class LLMSFullManager:
max_lines = self.config.get("llms_txt_full_max_size") max_lines = self.config.get("llms_txt_full_max_size")
abort_due_to_max_lines = False abort_due_to_max_lines = False
for docname in page_order: for docname, _ in page_order:
if docname in docname_to_file: if docname in docname_to_file:
file_path = docname_to_file[docname] file_path = docname_to_file[docname]
content, line_count = self._read_source_file(file_path, docname) content, line_count = self._read_source_file(file_path, docname)
@@ -179,14 +204,23 @@ class LLMSFullManager:
total_line_count += line_count total_line_count += line_count
else: else:
logger.warning( logger.warning(
f"sphinx-llm-txt: Source file not found for: {docname}. Check that" f"sphinx-llms-txt: Source file not found for: {docname}. Check that"
f" the file exists at _sources/{docname}.rst.txt" f" file exists at _sources/{docname}[suffix]{source_link_suffix}"
) )
# Add any remaining files (in alphabetical order) that aren't in the page order # Add any remaining files (in alphabetical order) that aren't in the page order
if not abort_due_to_max_lines: if not abort_due_to_max_lines:
# Get all .rst.txt files in the _sources directory # Get all source files in the _sources directory using configured suffixes
all_source_files = list(sources_dir.glob("**/*.rst.txt")) source_suffixes = self._get_source_suffixes()
all_source_files = []
for src_suffix in source_suffixes:
# Avoid duplicate extensions when source_suffix == source_link_suffix
if src_suffix == source_link_suffix:
glob_pattern = f"**/*{src_suffix}"
else:
glob_pattern = f"**/*{src_suffix}{source_link_suffix}"
all_source_files.extend(sources_dir.glob(glob_pattern))
processed_paths = set(file.resolve() for file in docname_to_file.values()) processed_paths = set(file.resolve() for file in docname_to_file.values())
# Find files that haven't been processed yet # Find files that haven't been processed yet
@@ -204,11 +238,23 @@ class LLMSFullManager:
) )
for file_path in remaining_source_files: for file_path in remaining_source_files:
# Extract docname from path by removing the .rst.txt extension # Extract docname from path by removing the source and link suffixes
rel_path = str(file_path.relative_to(sources_dir)) rel_path = str(file_path.relative_to(sources_dir))
if rel_path.endswith(".rst.txt"): docname = None
docname = rel_path[:-8] # Remove .rst.txt extension
# Try each source suffix to find which one this file uses
for src_suffix in source_suffixes:
# Avoid duplicate extensions when suffixes match
if src_suffix == source_link_suffix:
combined_suffix = src_suffix
else: else:
combined_suffix = f"{src_suffix}{source_link_suffix}"
if rel_path.endswith(combined_suffix):
docname = rel_path[: -len(combined_suffix)] # Remove suffix
break
if docname is None:
continue continue
# Skip excluded docnames # Skip excluded docnames
@@ -237,7 +283,7 @@ class LLMSFullManager:
max_lines is not None and total_line_count > max_lines max_lines is not None and total_line_count > max_lines
): ):
logger.warning( logger.warning(
f"sphinx-llm-txt: Max line limit ({max_lines}) exceeded:" f"sphinx-llms-txt: Max line limit ({max_lines}) exceeded:"
f" {total_line_count} > {max_lines}. " f" {total_line_count} > {max_lines}. "
f"Not creating llms-full.txt file." f"Not creating llms-full.txt file."
) )
@@ -304,5 +350,23 @@ class LLMSFullManager:
return content_str, line_count + 1 return content_str, line_count + 1
except Exception as e: except Exception as e:
logger.error(f"sphinx-llm-txt: Error reading source file {file_path}: {e}") logger.error(f"sphinx-llms-txt: Error reading source file {file_path}: {e}")
return "", 0 return "", 0
def _get_source_suffixes(self):
"""Get all valid source file suffixes from Sphinx configuration.
Returns:
list: List of source file suffixes (e.g., ['.rst', '.md', '.txt'])
"""
if not self.app:
return [".rst"] # Default fallback
source_suffix = self.app.config.source_suffix
if isinstance(source_suffix, dict):
return list(source_suffix.keys())
elif isinstance(source_suffix, list):
return source_suffix
else:
return [source_suffix] # String format
+12 -5
View File
@@ -3,7 +3,7 @@ File writer module for sphinx-llms-txt.
""" """
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List from typing import Any, Dict, List, Tuple, Union
from sphinx.application import Sphinx from sphinx.application import Sphinx
from sphinx.util import logging from sphinx.util import logging
@@ -42,19 +42,19 @@ class FileWriter:
) )
return True return True
except Exception as e: except Exception as e:
logger.error(f"sphinx-llm-txt: Error writing combined sources file: {e}") logger.error(f"sphinx-llms-txt: Error writing combined sources file: {e}")
return False return False
def write_verbose_info_to_file( def write_verbose_info_to_file(
self, self,
page_order: List[str], page_order: Union[List[str], List[Tuple[str, str]]],
page_titles: Dict[str, str], page_titles: Dict[str, str],
total_line_count: int = 0, total_line_count: int = 0,
) -> bool: ) -> bool:
"""Write summary information to the llms.txt file. """Write summary information to the llms.txt file.
Args: Args:
page_order: Ordered list of document names page_order: Ordered list of document names or (docname, suffix) tuples
page_titles: Dictionary mapping docnames to titles page_titles: Dictionary mapping docnames to titles
total_line_count: Total number of lines in the combined content total_line_count: Total number of lines in the combined content
@@ -88,6 +88,8 @@ class FileWriter:
if description: if description:
# Trim leading and trailing whitespace # Trim leading and trailing whitespace
description = description.strip() description = description.strip()
if description:
# Only add blockquote if description is not empty
# Replace newlines with newline + blockquote marker to maintain # Replace newlines with newline + blockquote marker to maintain
# blockquote formatting # blockquote formatting
description = description.replace("\n", "\n> ") description = description.replace("\n", "\n> ")
@@ -100,7 +102,12 @@ class FileWriter:
if not base_url.endswith("/"): if not base_url.endswith("/"):
base_url += "/" base_url += "/"
for docname in page_order: for item in page_order:
# Handle both old format (str) and new format (tuple)
if isinstance(item, tuple):
docname, _ = item
else:
docname = item
title = page_titles.get(docname, docname) title = page_titles.get(docname, docname)
f.write(f"- [{title}]({base_url}{docname}.html)\n") f.write(f"- [{title}]({base_url}{docname}.html)\n")
+474
View File
@@ -334,3 +334,477 @@ def test_write_verbose_info_with_baseurl(tmp_path):
assert "- [Home Page](https://example.org/index.html)" in content assert "- [Home Page](https://example.org/index.html)" in content
assert "- [About Us](https://example.org/about.html)" in content assert "- [About Us](https://example.org/about.html)" in content
def test_get_source_suffixes_with_dict():
"""Test _get_source_suffixes method with dict source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with dict source_suffix
class MockApp:
class Config:
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert set(suffixes) == {".rst", ".md", ".txt"}
def test_get_source_suffixes_with_list():
"""Test _get_source_suffixes method with list source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with list source_suffix
class MockApp:
class Config:
source_suffix = [".rst", ".md"]
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst", ".md"]
def test_get_source_suffixes_with_string():
"""Test _get_source_suffixes method with string source_suffix."""
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with string source_suffix
class MockApp:
class Config:
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"]
def test_get_source_suffixes_no_app():
"""Test _get_source_suffixes method with no app set."""
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
suffixes = manager._get_source_suffixes()
assert suffixes == [".rst"] # Default fallback
def test_html_sourcelink_suffix_default():
"""Test html_sourcelink_suffix defaults to .txt when no app is set."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
manager = LLMSFullManager()
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with default .txt suffix
test_file = f"{sources_dir}/index.rst.txt"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .txt as the default suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed (check if output file exists)
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_custom():
"""Test html_sourcelink_suffix uses custom value from Sphinx config."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with custom html_sourcelink_suffix
class MockApp:
class Config:
html_sourcelink_suffix = "source"
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with custom .source suffix
test_file = f"{sources_dir}/index.rst.source"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it uses .source as the custom suffix
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_html_sourcelink_suffix_with_dot():
"""Test html_sourcelink_suffix adds dot if missing."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with html_sourcelink_suffix without leading dot
class MockApp:
class Config:
html_sourcelink_suffix = "src" # No leading dot
source_suffix = ".rst"
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create a test source file with .src suffix (dot should be added automatically)
test_file = f"{sources_dir}/index.rst.src"
with open(test_file, "w") as f:
f.write("Test content")
# Mock env with minimal required attributes
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Test Title"})()
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test that it adds the dot and finds the file
manager.combine_sources(outdir, srcdir)
# Verify the file was found and processed
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
def test_mixed_source_file_formats():
"""Test handling of mixed source file formats (.rst, .md, .txt)."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with multiple source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = {".rst": None, ".md": None, ".txt": None}
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create test source files with different formats
files_to_create = [
f"{sources_dir}/page1.rst.txt",
f"{sources_dir}/page2.md.txt",
f"{sources_dir}/page3.txt.txt",
]
for test_file in files_to_create:
with open(test_file, "w") as f:
f.write(f"Content for {os.path.basename(test_file)}")
# Mock env with all documents
class MockEnv:
all_docs = {"page1": None, "page2": None, "page3": None}
titles = {
"page1": type("TitleNode", (), {"astext": lambda: "Page 1"})(),
"page2": type("TitleNode", (), {"astext": lambda: "Page 2"})(),
"page3": type("TitleNode", (), {"astext": lambda: "Page 3"})(),
}
toctree_includes = {}
manager.set_env(MockEnv())
manager.set_master_doc("page1")
# Test that all file formats are found and processed
manager.combine_sources(outdir, srcdir)
# Verify the output file was created and contains content from all formats
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# Should contain content from all three files
assert "Content for page1.rst.txt" in content
assert "Content for page2.md.txt" in content
assert "Content for page3.txt.txt" in content
def test_source_suffix_detection_priority():
"""Test source suffix detection tries formats in correct order for docnames."""
import tempfile
from sphinx_llms_txt.manager import LLMSFullManager
# Mock Sphinx app with ordered source suffixes
class MockApp:
class Config:
html_sourcelink_suffix = ".txt"
source_suffix = [".rst", ".md"] # rst has priority over md
config = Config()
manager = LLMSFullManager()
manager.set_app(MockApp())
manager.set_config(
{
"llms_txt_full_filename": "test.txt",
"llms_txt_exclude": [],
"llms_txt_directives": [],
}
)
# Create a temporary directory structure
with tempfile.TemporaryDirectory() as tmpdir:
outdir = f"{tmpdir}/build"
srcdir = f"{tmpdir}/source"
sources_dir = f"{outdir}/_sources"
# Create directories
import os
os.makedirs(sources_dir, exist_ok=True)
os.makedirs(srcdir, exist_ok=True)
# Create both .rst and .md versions of the same document
# Only create files for the specific docname "index"
rst_file = f"{sources_dir}/index.rst.txt"
md_file = f"{sources_dir}/index.md.txt"
with open(rst_file, "w") as f:
f.write("RST content for index")
with open(md_file, "w") as f:
f.write("Markdown content for index")
# Mock env with only the index document
class MockEnv:
all_docs = {"index": None}
titles = {
"index": type("TitleNode", (), {"astext": lambda: "Index Page"})()
}
toctree_includes = {"index": []}
manager.set_env(MockEnv())
manager.set_master_doc("index")
# Test the priority behavior
manager.combine_sources(outdir, srcdir)
# Check that output file was created
output_file = f"{outdir}/test.txt"
assert os.path.exists(output_file)
with open(output_file, "r") as f:
content = f.read()
# The system should prefer RST over MD for the "index" docname
# But since both files exist and the second phase adds remaining files,
# both will be included. The test verifies that RST appears first
# (indicating it was found first in the priority order)
assert "RST content for index" in content
# Find positions to verify order
rst_pos = content.find("RST content for index")
md_pos = content.find("Markdown content for index")
# RST should come before MD (due to priority in toctree processing)
assert rst_pos < md_pos, "RST content should appear before MD content"
def test_summary_default_uses_first_paragraph():
"""
Test that summary defaults to first paragraph of root document when not configured.
"""
from docutils import nodes
from docutils.frontend import OptionParser
from docutils.parsers.rst import Parser
from docutils.utils import new_document
from sphinx_llms_txt import build_finished, doctree_resolved
# Create a proper document with settings
settings = OptionParser(components=(Parser,)).get_default_values()
doctree = new_document("<rst-doc>", settings)
title = nodes.title(text="Test Title")
paragraph = nodes.paragraph(
text="This is the first paragraph that should be used as summary."
)
doctree.append(title)
doctree.append(paragraph)
# Mock Sphinx app
class MockApp:
class Config:
master_doc = "index"
llms_txt_summary = None # Not configured
llms_txt_file = True
llms_txt_filename = "llms.txt"
llms_txt_title = None
llms_txt_full_file = True
llms_txt_full_filename = "llms-full.txt"
llms_txt_full_max_size = None
llms_txt_directives = []
llms_txt_exclude = []
html_baseurl = ""
config = Config()
outdir = "/tmp/build"
srcdir = "/tmp/source"
class Env:
titles = {
"index": type("TitleNode", (), {"astext": lambda self: "Test Title"})()
}
env = Env()
app = MockApp()
# Reset the global state
import sphinx_llms_txt
sphinx_llms_txt._root_first_paragraph = ""
# Call doctree_resolved to extract the first paragraph
doctree_resolved(app, doctree, "index")
# Verify the first paragraph was extracted
assert (
sphinx_llms_txt._root_first_paragraph
== "This is the first paragraph that should be used as summary."
)
# Mock the manager methods to avoid actual file operations
original_combine_sources = sphinx_llms_txt._manager.combine_sources
sphinx_llms_txt._manager.combine_sources = lambda outdir, srcdir: None
# Call build_finished and verify the summary is set correctly
build_finished(app, None)
# Check that the summary was properly configured
assert (
sphinx_llms_txt._manager.config["llms_txt_summary"]
== "This is the first paragraph that should be used as summary."
)
# Restore original method
sphinx_llms_txt._manager.combine_sources = original_combine_sources