diff --git a/CHANGELOG.rst b/CHANGELOG.rst index ea56c66..74da444 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -4,6 +4,8 @@ Changelog 0.5.0 ----- +- Add :ref:`block_level_ignore` and :ref:`page_level_ignore` + `#33 `_ - Add :confval:`llms_txt_full_size_policy` configuration option to control behavior when :confval:`llms_txt_full_max_size` is exceeded. `#35 `_ diff --git a/docs/source/advanced-configuration.rst b/docs/source/advanced-configuration.rst index 403b7dc..d457944 100644 --- a/docs/source/advanced-configuration.rst +++ b/docs/source/advanced-configuration.rst @@ -124,6 +124,13 @@ This ensures that paths in your custom directives are properly resolved in the g Excluding Content ^^^^^^^^^^^^^^^^^ +There are several ways to exclude content from the generated ``llms-full.txt`` file: + +.. _global_exclusion: + +Global Page Exclusion +~~~~~~~~~~~~~~~~~~~~~~ + You can exclude specific pages from being included in the generated files: .. code-block:: python @@ -135,6 +142,67 @@ You can exclude specific pages from being included in the generated files: ] This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption. +It can also be used to reduce the size of llms-full.txt. + +.. _page_level_ignore: + +Page-Level Ignore Metadata +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can exclude individual pages by adding metadata at the top of any reStructuredText file: + +.. code-block:: restructuredtext + + :llms-txt-ignore: true + + Page Title + ========== + + This entire page will be excluded from llms-full.txt + +When this metadata is present, the entire page is skipped during processing. + +.. _block_level_ignore: + +Block-Level Ignore Directives +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can exclude specific sections within a page using ignore directives: + +.. code-block:: restructuredtext + + Page Title + ========== + + This content will be included in llms-full.txt. + + .. llms-txt-ignore-start + + This content will be excluded from llms-full.txt. + + Section To Ignore + ----------------- + + This entire section and any nested content will be ignored. + + .. code-block:: python + + # This code block will also be ignored + def ignored_function(): + pass + + .. llms-txt-ignore-end + + This content will be included again. + +Block-level ignores can be useful for: + +- Removing internal notes or TODOs +- Hiding implementation details while keeping user-facing documentation + +.. note:: + - Multiple ignore blocks can be used within the same file + - Ignore directives work with any indentation level .. _including_code_files: diff --git a/docs/source/configuration-values.rst b/docs/source/configuration-values.rst index 5e8904d..0eef2b5 100644 --- a/docs/source/configuration-values.rst +++ b/docs/source/configuration-values.rst @@ -89,7 +89,7 @@ Project Configuration Values - **Type**: list of strings - **Default**: ``[]`` - - **Description**: A list of pages to ignore. + - **Description**: A list of pages to ignore using glob patterns. See :ref:`excluding_content`. .. versionadded:: 0.2.1 diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst index 9227df2..8011e15 100644 --- a/docs/source/getting-started.rst +++ b/docs/source/getting-started.rst @@ -1,11 +1,6 @@ Getting Started =============== -Demo ----- - -You can see this Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as a simple example. - Installation ------------ @@ -15,6 +10,12 @@ Directly install via ``pip`` by using: pip install sphinx-llms-txt +Or with ``conda`` via ``conda-forge``: + +.. code:: + + conda install -c conda-forge sphinx-llms-txt + Usage ----- @@ -26,25 +27,12 @@ Add the extension to your Sphinx configuration (``conf.py``): 'sphinx_llms_txt', ] -Once added, the extension will automatically generate the LLMs.txt files during the build process. +After the HTML finishes building, **sphinx-llms-txt** will output the location of the output files:: + + sphinx-llms-txt: Created /path/to/_build/html/llms-full.txt with 45 sources and 6879 lines + sphinx-llms-txt: created /path/to/_build/html/llms.txt + + +.. tip:: Make sure to confirm the accuracy of the output files after installs and upgrades. See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**. - -How It Works ------------- - -During the Sphinx build process: - -1. **Content Collection**: Scans all of your documentation's ``_source`` pages and collects their content -2. **Directive Processing**: Resolves ``include`` directives by automatically incorporating their content -3. **Path Resolution**: Transforms relative paths in directives to full paths -4. **Output Generation**: Creates two optional files: - - - ``llms.txt``: A concise summary of your documentation, in Markdown - - ``llms-full.txt``: A comprehensive version with all documentation content, in reStructuredText - -5. **Content Filtering**: Allows you to exclude specific pages from the generated files - - -.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt -.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt diff --git a/docs/source/index.rst b/docs/source/index.rst index 04fb43b..649cccb 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -5,6 +5,25 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar |PyPI version| |Conda Version| |Downloads| |Parallel Safe| |GitHub Stars| +Demo +---- + +You can see this Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as a simple example. + +Highlights +---------- + +1. **Content Collection**: Quickly gathers content from _sources, without needing a separate build +2. **Directive Processing**: Resolves ``include`` directives by automatically incorporating their content +3. **Path Resolution**: Transforms relative paths in directives to full paths +4. **Output Generation**: Creates two optional files: + + - ``llms.txt``: A concise summary of your documentation, in Markdown + - ``llms-full.txt``: A comprehensive version with all documentation content, in reStructuredText + +5. **Content Filtering**: Allows you to exclude specific pages or sections +6. **Source Code**: Allows you to include specific source code files + .. toctree:: :maxdepth: 2 @@ -15,6 +34,8 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar changelog +.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt +.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt .. _Sphinx: http://sphinx-doc.org/ .. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg diff --git a/sphinx_llms_txt/__init__.py b/sphinx_llms_txt/__init__.py index e201b97..c09f172 100644 --- a/sphinx_llms_txt/__init__.py +++ b/sphinx_llms_txt/__init__.py @@ -33,6 +33,13 @@ def doctree_resolved(app: Sphinx, doctree, docname: str): """Called when a docname has been resolved to a document.""" global _root_first_paragraph + # Check for llms-txt-ignore metadata at the page level + if hasattr(app.env, "metadata") and docname in app.env.metadata: + metadata = app.env.metadata[docname] + if metadata.get("llms-txt-ignore", "").lower() in ("true", "1", "yes"): + _manager.mark_page_ignored(docname) + return + # Extract title from the document title = None # findall() returns a generator, convert to list to check if it has elements diff --git a/sphinx_llms_txt/manager.py b/sphinx_llms_txt/manager.py index b31dcea..1f97da5 100644 --- a/sphinx_llms_txt/manager.py +++ b/sphinx_llms_txt/manager.py @@ -5,7 +5,7 @@ Main manager module for sphinx-llms-txt. import glob import subprocess from pathlib import Path -from typing import Any, Dict, List, Optional, Tuple +from typing import Any, Dict, List, Optional, Tuple, Union from sphinx.application import Sphinx from sphinx.environment import BuildEnvironment @@ -129,6 +129,7 @@ class LLMSFullManager: self.srcdir: Optional[str] = None self.outdir: Optional[str] = None self.app: Optional[Sphinx] = None + self.ignored_pages: set = set() def set_master_doc(self, master_doc: str): """Set the master document name.""" @@ -144,6 +145,27 @@ class LLMSFullManager: """Update the title for a page.""" self.collector.update_page_title(docname, title) + def mark_page_ignored(self, docname: str): + """Mark a page as ignored due to llms-txt-ignore metadata.""" + self.ignored_pages.add(docname) + + def _filter_ignored_pages( + self, page_order: Union[List[str], List[Tuple[str, str]]] + ) -> Union[List[str], List[Tuple[str, str]]]: + """Filter out ignored pages from page_order.""" + filtered_pages = [] + for item in page_order: + # Handle both old format (str) and new format (tuple) + if isinstance(item, tuple): + docname, _ = item + else: + docname = item + + if docname not in self.ignored_pages: + filtered_pages.append(item) + + return filtered_pages + def set_config(self, config: Dict[str, Any]): """Set configuration options.""" self.config = config @@ -286,6 +308,11 @@ class LLMSFullManager: should_abort_early = size_policy_action in ["skip", "note"] for docname, _ in page_order: + # Skip pages marked as ignored + if docname in self.ignored_pages: + logger.debug(f"sphinx-llms-txt: Skipping ignored page: {docname}") + continue + if docname in docname_to_file: file_path = docname_to_file[docname] content, line_count = self._read_source_file(file_path, docname) @@ -383,6 +410,13 @@ class LLMSFullManager: if docname is None: continue + # Skip pages marked as ignored + if docname in self.ignored_pages: + logger.debug( + f"sphinx-llms-txt: Skipping ignored remaining file: {docname}" + ) + continue + # Skip excluded docnames if exclude_patterns and any( self.collector._match_exclude_pattern(docname, pattern) @@ -468,8 +502,11 @@ class LLMSFullManager: logger.info(f"sphinx-llms-txt: Skipping {filename} generation") # Log summary information if requested if self.config.get("llms_txt_file"): + filtered_page_order = self._filter_ignored_pages(page_order) self.writer.write_verbose_info_to_file( - page_order, self.collector.page_titles, total_line_count + filtered_page_order, + self.collector.page_titles, + total_line_count, ) return elif action == "note": @@ -478,8 +515,11 @@ class LLMSFullManager: # Log summary information if requested if self.config.get("llms_txt_file"): + filtered_page_order = self._filter_ignored_pages(page_order) self.writer.write_verbose_info_to_file( - page_order, self.collector.page_titles, total_line_count + filtered_page_order, + self.collector.page_titles, + total_line_count, ) return elif action == "keep": @@ -496,8 +536,9 @@ class LLMSFullManager: # Log summary information if requested if success and self.config.get("llms_txt_file"): + filtered_page_order = self._filter_ignored_pages(page_order) self.writer.write_verbose_info_to_file( - page_order, self.collector.page_titles, total_line_count + filtered_page_order, self.collector.page_titles, total_line_count ) def _read_source_file(self, file_path: Path, docname: str) -> Tuple[str, int]: diff --git a/sphinx_llms_txt/processor.py b/sphinx_llms_txt/processor.py index 9badbbb..f04ddfd 100644 --- a/sphinx_llms_txt/processor.py +++ b/sphinx_llms_txt/processor.py @@ -44,7 +44,10 @@ class DocumentProcessor: Returns: Processed content with directives properly resolved """ - # First process include directives + # First process llms-txt-ignore blocks + content = self._process_ignore_blocks(content) + + # Then process include directives content = self._process_includes(content, source_path) # Then process path directives (image, figure, etc.) @@ -337,3 +340,36 @@ class DocumentProcessor: # Replace all includes with their content processed_content = include_pattern.sub(replace_include, content) return processed_content + + def _process_ignore_blocks(self, content: str) -> str: + """Process llms-txt-ignore-start/end blocks by removing their content. + + Args: + content: The source content to process + + Returns: + Processed content with ignore blocks removed + """ + # Process ignore blocks iteratively to handle nested cases correctly + while True: + # Pattern to match ignore blocks - handles whitespace and indentation + ignore_pattern = re.compile( + r"^\s*\.\.\s+llms-txt-ignore-start\s*\n" # Start directive line + r"(.*?)" # Content to ignore (non-greedy) + r"^\s*\.\.\s+llms-txt-ignore-end\s*$", # End directive line + re.MULTILINE | re.DOTALL, + ) + + # Find and remove one ignore block at a time + match = ignore_pattern.search(content) + if not match: + break + + # Remove the matched block + content = content[: match.start()] + content[match.end() :] + + # Clean up any extra blank lines that might be left + # Replace multiple consecutive newlines with at most 2 newlines + processed_content = re.sub(r"\n\n\n+", "\n\n", content) + + return processed_content diff --git a/tests/roots/basic/index.rst b/tests/roots/basic/index.rst index 0893a95..0138141 100644 --- a/tests/roots/basic/index.rst +++ b/tests/roots/basic/index.rst @@ -8,6 +8,8 @@ Welcome to Test Project's documentation! page1 page2 page_with_include + page_ignored_metadata + page_with_ignore_blocks Indices and tables ================== diff --git a/tests/roots/basic/page_ignored_metadata.rst b/tests/roots/basic/page_ignored_metadata.rst new file mode 100644 index 0000000..dce4eab --- /dev/null +++ b/tests/roots/basic/page_ignored_metadata.rst @@ -0,0 +1,16 @@ +:llms-txt-ignore: true + +Page Ignored by Metadata +======================== + +This page should not appear in llms-full.txt because of the metadata directive. + +Section 1 +--------- + +This content should be completely ignored. + +Section 2 +--------- + +This content should also be ignored. \ No newline at end of file diff --git a/tests/roots/basic/page_with_ignore_blocks.rst b/tests/roots/basic/page_with_ignore_blocks.rst new file mode 100644 index 0000000..dfb825c --- /dev/null +++ b/tests/roots/basic/page_with_ignore_blocks.rst @@ -0,0 +1,39 @@ +Page With Ignore Blocks +======================= + +This content should appear in llms-full.txt. + +.. llms-txt-ignore-start + +This content should be ignored and not appear in llms-full.txt. + +Section Ignored +--------------- + +This section should also be ignored. + +.. llms-txt-ignore-end + +This content after the ignore block should appear in llms-full.txt. + +Another Section +--------------- + +This content should definitely appear. + +.. llms-txt-ignore-start + +Another ignored block with multiple lines. + +- Item 1 (ignored) +- Item 2 (ignored) + +.. code-block:: python + + # This code should be ignored + def ignored_function(): + pass + +.. llms-txt-ignore-end + +Final content that should appear. \ No newline at end of file diff --git a/tests/test_ignore_features.py b/tests/test_ignore_features.py new file mode 100644 index 0000000..af44b6a --- /dev/null +++ b/tests/test_ignore_features.py @@ -0,0 +1,220 @@ +"""Tests for llms-txt ignore features.""" + +from pathlib import Path + +from sphinx_llms_txt import DocumentProcessor + + +def test_process_ignore_blocks(): + """Test that ignore blocks are properly removed from content.""" + processor = DocumentProcessor({}, None) + + content = """This content should remain. + +.. llms-txt-ignore-start + +This content should be removed. + +Section Ignored +--------------- + +This section should also be removed. + +.. llms-txt-ignore-end + +This content should remain after the ignore block. + +.. llms-txt-ignore-start + +Another ignored block. +Multiple lines here. + +.. llms-txt-ignore-end + +Final content that should remain.""" + + processed = processor._process_ignore_blocks(content) + + # Check that ignored content is removed + assert "This content should be removed." not in processed + assert "Section Ignored" not in processed + assert "Another ignored block." not in processed + assert "Multiple lines here." not in processed + + # Check that non-ignored content remains + assert "This content should remain." in processed + assert "This content should remain after the ignore block." in processed + assert "Final content that should remain." in processed + + +def test_process_ignore_blocks_with_indentation(): + """Test that ignore blocks work with different indentation levels.""" + processor = DocumentProcessor({}, None) + + content = """Section Title +============= + +Normal content. + + .. llms-txt-ignore-start + + Indented ignored content. + More indented content. + + .. llms-txt-ignore-end + +Back to normal content.""" + + processed = processor._process_ignore_blocks(content) + + # Check that ignored content is removed + assert "Indented ignored content." not in processed + assert "More indented content." not in processed + + # Check that non-ignored content remains + assert "Section Title" in processed + assert "Normal content." in processed + assert "Back to normal content." in processed + + +def test_process_ignore_blocks_multiple(): + """Test that multiple ignore blocks are handled correctly.""" + processor = DocumentProcessor({}, None) + + content = """Start content. + +.. llms-txt-ignore-start + +First ignore block. + +.. llms-txt-ignore-end + +Middle content that should remain. + +.. llms-txt-ignore-start + +Second ignore block. + +.. llms-txt-ignore-end + +End content.""" + + processed = processor._process_ignore_blocks(content) + + # Check that ignored content is removed + assert "First ignore block." not in processed + assert "Second ignore block." not in processed + + # Check that non-ignored content remains + assert "Start content." in processed + assert "Middle content that should remain." in processed + assert "End content." in processed + + +def test_build_with_ignore_features(basic_sphinx_app): + """Test building HTML documentation with ignore features.""" + app = basic_sphinx_app + app.build() + + # Check if the output file was created + output_file = Path(app.outdir) / "test-llms-full.txt" + assert output_file.exists(), f"Output file {output_file} does not exist" + + # Read the content of the output file + content = output_file.read_text() + + # Check that page with metadata ignore is completely excluded + assert "Page Ignored by Metadata" not in content + assert "This page should not appear in llms-full.txt" not in content + + # Check that page with ignore blocks has the right content + assert "Page With Ignore Blocks" in content + assert "This content should appear in llms-full.txt." in content + assert "This content after the ignore block should appear" in content + assert "Another Section" in content + assert "Final content that should appear." in content + + # Check that ignored block content is not present + assert "This content should be ignored and not appear" not in content + assert "Section Ignored" not in content + assert "Another ignored block with multiple lines." not in content + assert "Item 1 (ignored)" not in content + assert "def ignored_function():" not in content + + +def test_manager_mark_page_ignored(): + """Test that manager can mark pages as ignored.""" + from sphinx_llms_txt import LLMSFullManager + + manager = LLMSFullManager() + + # Initially no pages are ignored + assert len(manager.ignored_pages) == 0 + + # Mark a page as ignored + manager.mark_page_ignored("test_page") + + # Check that page is in ignored set + assert "test_page" in manager.ignored_pages + assert len(manager.ignored_pages) == 1 + + # Mark another page as ignored + manager.mark_page_ignored("another_page") + + # Check both pages are ignored + assert "test_page" in manager.ignored_pages + assert "another_page" in manager.ignored_pages + assert len(manager.ignored_pages) == 2 + + +def test_process_ignore_blocks_empty_blocks(): + """Test that empty ignore blocks are handled correctly.""" + processor = DocumentProcessor({}, None) + + content = """Content before. + +.. llms-txt-ignore-start + +.. llms-txt-ignore-end + +Content after.""" + + processed = processor._process_ignore_blocks(content) + + # Check that content remains + assert "Content before." in processed + assert "Content after." in processed + + # Check that we don't have excessive newlines + lines = processed.strip().split("\n") + non_empty_lines = [line for line in lines if line.strip()] + assert len(non_empty_lines) == 2 + + +def test_ignore_metadata_affects_both_files(basic_sphinx_app): + """Test that :llms-txt-ignore: true affects both files.""" + app = basic_sphinx_app + # Enable both llms.txt and llms-full.txt file generation + app.config.llms_txt_file = True + app.config.llms_txt_filename = "test-llms.txt" + app.build() + + # Check if both output files were created + llms_full_file = Path(app.outdir) / "test-llms-full.txt" + llms_summary_file = Path(app.outdir) / "test-llms.txt" + + assert llms_full_file.exists(), f"Output file {llms_full_file} does not exist" + assert llms_summary_file.exists(), f"Output file {llms_summary_file} does not exist" + + # Read the content of both files + llms_full_content = llms_full_file.read_text() + llms_summary_content = llms_summary_file.read_text() + + # Check that page with metadata ignore is excluded from llms-full.txt + assert "Page Ignored by Metadata" not in llms_full_content + assert "This page should not appear in llms-full.txt" not in llms_full_content + + # Check that page with metadata ignore is also excluded from llms.txt + # This should NOT contain a link to the ignored page + assert "Page Ignored by Metadata" not in llms_summary_content + assert "page_ignored_metadata.html" not in llms_summary_content