Add support for page and block level ignores (#33)

docs/source/advanced-configuration.rst

@@ -124,6 +124,13 @@ This ensures that paths in your custom directives are properly resolved in the g
 Excluding Content
 ^^^^^^^^^^^^^^^^^
 
+There are several ways to exclude content from the generated ``llms-full.txt`` file:
+
+.. _global_exclusion:
+
+Global Page Exclusion
+~~~~~~~~~~~~~~~~~~~~~~
+
 You can exclude specific pages from being included in the generated files:
 
 .. code-block:: python
@@ -135,6 +142,67 @@ You can exclude specific pages from being included in the generated files:
    ]
 
 This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption.
+It can also be used to reduce the size of llms-full.txt.
+
+.. _page_level_ignore:
+
+Page-Level Ignore Metadata
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can exclude individual pages by adding metadata at the top of any reStructuredText file:
+
+.. code-block:: restructuredtext
+
+   :llms-txt-ignore: true
+
+   Page Title
+   ==========
+
+   This entire page will be excluded from llms-full.txt
+
+When this metadata is present, the entire page is skipped during processing.
+
+.. _block_level_ignore:
+
+Block-Level Ignore Directives
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can exclude specific sections within a page using ignore directives:
+
+.. code-block:: restructuredtext
+
+   Page Title
+   ==========
+
+   This content will be included in llms-full.txt.
+
+   .. llms-txt-ignore-start
+
+   This content will be excluded from llms-full.txt.
+
+   Section To Ignore
+   -----------------
+
+   This entire section and any nested content will be ignored.
+
+   .. code-block:: python
+
+      # This code block will also be ignored
+      def ignored_function():
+          pass
+
+   .. llms-txt-ignore-end
+
+   This content will be included again.
+
+Block-level ignores can be useful for:
+
+- Removing internal notes or TODOs
+- Hiding implementation details while keeping user-facing documentation
+
+.. note::
+   - Multiple ignore blocks can be used within the same file
+   - Ignore directives work with any indentation level
 
 .. _including_code_files:
 

docs/source/configuration-values.rst

@@ -89,7 +89,7 @@ Project Configuration Values
 
    - **Type**: list of strings
    - **Default**: ``[]``
-   - **Description**: A list of pages to ignore.
+   - **Description**: A list of pages to ignore using glob patterns.
      See :ref:`excluding_content`.
 
    .. versionadded:: 0.2.1

docs/source/getting-started.rst

@@ -1,11 +1,6 @@
 Getting Started
 ===============
 
-Demo
-----
-
-You can see this Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as a simple example.
-
 Installation
 ------------
 
@@ -15,6 +10,12 @@ Directly install via ``pip`` by using:
 
     pip install sphinx-llms-txt
 
+Or with ``conda`` via ``conda-forge``:
+
+.. code::
+
+   conda install -c conda-forge sphinx-llms-txt
+
 Usage
 -----
 
@@ -26,25 +27,12 @@ Add the extension to your Sphinx configuration (``conf.py``):
         'sphinx_llms_txt',
     ]
 
-Once added, the extension will automatically generate the LLMs.txt files during the build process.
+After the HTML finishes building, **sphinx-llms-txt** will output the location of the output files::
+
+    sphinx-llms-txt: Created /path/to/_build/html/llms-full.txt with 45 sources and 6879 lines
+    sphinx-llms-txt: created /path/to/_build/html/llms.txt
+
+
+.. tip:: Make sure to confirm the accuracy of the output files after installs and upgrades.
 
 See :doc:`advanced-configuration` for more information about how to use **sphinx-llms-txt**.
-
-How It Works
-------------
-
-During the Sphinx build process:
-
-1. **Content Collection**: Scans all of your documentation's ``_source`` pages and collects their content
-2. **Directive Processing**: Resolves ``include`` directives by automatically incorporating their content
-3. **Path Resolution**: Transforms relative paths in directives to full paths
-4. **Output Generation**: Creates two optional files:
-
-   - ``llms.txt``: A concise summary of your documentation, in Markdown
-   - ``llms-full.txt``: A comprehensive version with all documentation content, in reStructuredText
-
-5. **Content Filtering**: Allows you to exclude specific pages from the generated files
-
-
-.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt
-.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt

docs/source/index.rst

@@ -5,6 +5,25 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar
 
 |PyPI version| |Conda Version| |Downloads| |Parallel Safe| |GitHub Stars|
 
+Demo
+----
+
+You can see this Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as a simple example.
+
+Highlights
+----------
+
+1. **Content Collection**: Quickly gathers content from _sources, without needing a separate build
+2. **Directive Processing**: Resolves ``include`` directives by automatically incorporating their content
+3. **Path Resolution**: Transforms relative paths in directives to full paths
+4. **Output Generation**: Creates two optional files:
+
+   - ``llms.txt``: A concise summary of your documentation, in Markdown
+   - ``llms-full.txt``: A comprehensive version with all documentation content, in reStructuredText
+
+5. **Content Filtering**: Allows you to exclude specific pages or sections
+6. **Source Code**: Allows you to include specific source code files
+
 .. toctree::
    :maxdepth: 2
 
@@ -15,6 +34,8 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar
    changelog
 
 
+.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt
+.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt
 .. _Sphinx: http://sphinx-doc.org/
 
 .. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg