From 8f4d2c07c67a09ed450a372768f18044b49d446c Mon Sep 17 00:00:00 2001 From: Jared Dillard Date: Sun, 18 May 2025 21:10:49 -0700 Subject: [PATCH] Update docs and README (#17) * Move readme content to index.rst * Clean up project name * Add advanced configuration --- README.md | 82 +----------- docs/source/advanced-configuration.rst | 170 +++++++++++++++++++++++++ docs/source/conf.py | 2 +- docs/source/configuration-values.rst | 27 ++-- docs/source/getting-started.rst | 24 ++++ docs/source/index.rst | 20 +-- 6 files changed, 212 insertions(+), 113 deletions(-) create mode 100644 docs/source/advanced-configuration.rst diff --git a/README.md b/README.md index 7691780..1d44062 100644 --- a/README.md +++ b/README.md @@ -5,87 +5,9 @@ A Sphinx extension that generates a summary `llms.txt` file, written in Markdown [![PyPI version](https://img.shields.io/pypi/v/sphinx-llms-txt.svg)](https://pypi.python.org/pypi/sphinx-llms-txt) [![Downloads](https://static.pepy.tech/badge/sphinx-llms-txt/month)](https://pepy.tech/project/sphinx-llms-txt) -## Installation +## Documentation -```bash -pip install sphinx-llms-txt -``` - -## Usage - -1. Add the extension to your Sphinx configuration (`conf.py`): - -```python -extensions = [ - 'sphinx_llms_txt', -] -``` - -## Configuration Options - -### `llms_txt_full_file` - -- **Type**: boolean -- **Default**: `'True'` -- **Description**: Whether to write the single output file - -### `llms_txt_full_filename` - -- **Type**: string -- **Default**: `'llms-full.txt'` -- **Description**: Name of the single output file - -### `llms_txt_full_max_size` - -- **Type**: integer or `None` -- **Default**: `None` (no limit) -- **Description**: Sets a maximum line count for `llms_txt_full_filename`. - If exceeded, the file is skipped and a warning is shown, but the build still completes. - -### `llms_txt_file` - -- **Type**: boolean -- **Default**: `True` -- **Description**: Whether to write the summary information file - -### `llms_txt_filename` - -- **Type**: string -- **Default**: `llms.txt` -- **Description**: Name of the summary information file - -### `llms_txt_directives` - -- **Type**: list of strings -- **Default**: `[]` -- **Description**: List of custom directive names to process for path resolution. - -### `llms_txt_title` - -- **Type**: string or `None` -- **Default**: `None` -- **Description**: Overrides the Sphinx project name as the heading in `llms.txt`. - -### `llms_txt_summary` - -- **Type**: string or `None` -- **Default**: `None` -- **Description**: Optional, but recommended, summary description for `llms.txt`. - -### `llms_txt_exclude` - -- **Type**: list of strings -- **Default**: `[]` -- **Description**: A list of pages to ignore (e.g., `["page1", "page_with_*"]`). - -## Features - -- Creates `llms.txt` and `llms-full.txt` -- Automatically add content from `include` directives -- Resolves relative paths in directives like `image` and `figure` to use full paths - - Ability to add list of custom directives with `llms_txt_directives` - - Optionally, prepend a base URL using Sphinx's `html_baseurl` -- Ability to exclude pages +See [sphinx-llms-txt documentation](https://sphinx-llms-txt.readthedocs.io/en/latest/index.html) for installation and configuration instructions. ## License diff --git a/docs/source/advanced-configuration.rst b/docs/source/advanced-configuration.rst new file mode 100644 index 0000000..bc77e21 --- /dev/null +++ b/docs/source/advanced-configuration.rst @@ -0,0 +1,170 @@ +Advanced Configuration +====================== + +This page covers advanced configuration options for the sphinx-llms-txt extension. + +.. _customizing_llms_files: + +Customizing the LLMs Files +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +By default, the extension generates two files: + +1. ``llms.txt`` - A summary file in Markdown format +2. ``llms-full.txt`` - A complete documentation file in reStructuredText format + +You can customize these files in several ways: + +.. _changing_filenames: + +Changing Filenames +~~~~~~~~~~~~~~~~~~ + +You can change the default filenames by setting these values in your ``conf.py``: + +.. code-block:: python + + llms_txt_filename = "custom-summary.txt" + llms_txt_full_filename = "custom-docs.txt" + +.. _disabling_file_generation: + +Disabling File Generation +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you only want one of the files, you can disable generation of the other: + +.. code-block:: python + + # Disable summary file + llms_txt_file = False + + # Disable full documentation file + llms_txt_full_file = False + +.. _custom_summary: + +Adding a Custom Summary +~~~~~~~~~~~~~~~~~~~~~~~ + +The summary file can include a custom description of your project: + +.. code-block:: python + + llms_txt_summary = """ + This documentation explains how to use MyProject to build amazing + applications. The project provides a comprehensive API for handling + data processing and visualization. + """ + +.. note:: The summary can span multiple lines and will be properly formatted in the output file. + +.. _custom_title: + +Custom Title +~~~~~~~~~~~~ + +By default, the project name from Sphinx is used as the title in ``llms.txt``. You can override this: + +.. code-block:: python + + llms_txt_title = "My Custom Project Documentation" + +.. _handling_large_documentation: + +Handling Large Documentation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For very large documentation sets, generating the full documentation file might exceed reasonable size limits. +You can set a maximum line count: + +.. code-block:: python + + llms_txt_full_max_size = 10000 # Maximum 10,000 lines + +If the generated file would exceed this limit, the extension will skip its generation and show a warning, allowing the build to complete. + +.. tip:: Use :ref:`excluding_content` to remove less relevant pages. + +.. _custom_directive_handling: + +Custom Directive Handling +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. _path_resolution: + +Path Resolution +~~~~~~~~~~~~~~~ + +The extension resolves paths in the common directives ``[ 'image', 'figure']`` by default. +You can add custom directives to this list: + +.. code-block:: python + + llms_txt_directives = [ + "my-custom-image-directive", + "another-directive-with-paths", + ] + +This ensures that paths in your custom directives are properly resolved in the generated files. + +.. _excluding_content: + +Excluding Content +^^^^^^^^^^^^^^^^^ + +You can exclude specific pages from being included in the generated files: + +.. code-block:: python + + llms_txt_exclude = [ + "search", # Exclude the search page + "genindex", # Exclude the index page + "private_*", # Exclude all pages starting with 'private_' + ] + +This is useful for excluding auto-generated pages, indexes, or content that isn't relevant for LLM consumption. + +.. _using_html_baseurl: + +Using HTML Base URL +^^^^^^^^^^^^^^^^^^^ + +If you want to include absolute URLs for resources in your documentation, you can use Sphinx's built-in ``html_baseurl`` configuration: + +.. code-block:: python + + html_baseurl = "https://example.com/docs/" + +When this option is set, all resolved paths in directives will be prefixed with this URL, creating absolute paths in the generated files. + +.. _integration_examples: + +Integration Examples +^^^^^^^^^^^^^^^^^^^^ + +Complete Configuration Example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Here's a complete example showing multiple configuration options: + +.. code-block:: python + + # File names and generation options + llms_txt_filename = "ai-summary.txt" + llms_txt_full_filename = "ai-full-docs.txt" + llms_txt_full_max_size = 50000 + + # Content customization + llms_txt_title = "Project Documentation for AI Assistants" + llms_txt_summary = """ + This is a comprehensive documentation set for our project. + It includes API references, usage examples, and tutorials. + """ + + # Path handling + html_baseurl = "https://docs.example.com/" + llms_txt_directives = ["custom-image", "custom-include"] + + # Content filtering + llms_txt_exclude = ["search", "genindex", "404", "private_*"] diff --git a/docs/source/conf.py b/docs/source/conf.py index 1195b2b..9d7d960 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -12,7 +12,7 @@ import subprocess # -- Project information ----------------------------------------------------- -project = "Sphinx llms.txt Generator" +project = "sphinx-llms-txt" copyright = "Jared Dillard" author = "Jared Dillard" llms_txt_summary = """ diff --git a/docs/source/configuration-values.rst b/docs/source/configuration-values.rst index 9b268ec..6fd522b 100644 --- a/docs/source/configuration-values.rst +++ b/docs/source/configuration-values.rst @@ -5,7 +5,8 @@ Project Configuration Values - **Type**: boolean - **Default**: ``True`` - - **Description**: Whether to write the single output file + - **Description**: Whether to write the single output file. + See :ref:`disabling_file_generation`. .. versionadded:: 0.1.0 @@ -13,7 +14,8 @@ Project Configuration Values - **Type**: string - **Default**: ``'llms-full.txt'`` - - **Description**: Name of the single output file + - **Description**: Name of the single output file. + See :ref:`changing_filenames`. .. versionadded:: 0.1.0 @@ -23,6 +25,7 @@ Project Configuration Values - **Default**: ``None`` (no limit) - **Description**: Sets a maximum line count for ``llms_txt_full_filename``. If exceeded, the file is skipped and a warning is shown, but the build still completes. + See :ref:`handling_large_documentation`. .. versionadded:: 0.2.0 @@ -30,7 +33,8 @@ Project Configuration Values - **Type**: boolean - **Default**: ``True`` - - **Description**: Whether to write the summary information file + - **Description**: Whether to write the summary information file. + See :ref:`disabling_file_generation`. .. versionadded:: 0.2.0 @@ -38,7 +42,8 @@ Project Configuration Values - **Type**: string - **Default**: ``llms.txt`` - - **Description**: Name of the summary information file + - **Description**: Name of the summary information file. + See :ref:`changing_filenames`. .. versionadded:: 0.2.0 @@ -47,6 +52,7 @@ Project Configuration Values - **Type**: list of strings - **Default**: ``[]`` (empty list) - **Description**: List of custom directive names to process for path resolution. + See :ref:`path_resolution`. .. versionadded:: 0.1.0 @@ -55,6 +61,7 @@ Project Configuration Values - **Type**: string or ``None`` - **Default**: ``None`` - **Description**: Overrides the Sphinx project name as the heading in ``llms.txt``. + See :ref:`custom_title`. .. versionadded:: 0.2.0 @@ -63,6 +70,7 @@ Project Configuration Values - **Type**: string or ``None`` - **Default**: ``None`` - **Description**: Optional, but recommended, summary description for ``llms.txt``. + See :ref:`custom_summary`. .. versionadded:: 0.2.0 @@ -70,14 +78,7 @@ Project Configuration Values - **Type**: list of strings - **Default**: ``[]`` - - **Description**: A list of pages to ignore (e.g., ``["page1", "page_with_*"]``). + - **Description**: A list of pages to ignore. + See :ref:`excluding_content`. .. versionadded:: 0.2.1 - -.. confval:: llms_txt_rm_directives - - - **Type**: boolean - - **Default**: ``False`` - - **Description**: Whether to remove all directives from the output files. - - .. versionadded:: 0.2.3 diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst index c88a055..c796419 100644 --- a/docs/source/getting-started.rst +++ b/docs/source/getting-started.rst @@ -1,6 +1,11 @@ Getting Started =============== +Demo +---- + +You can see this Sphinx project's `llms.txt`_ and `llms-full.txt`_ files as a simple example. + Installation ------------ @@ -22,3 +27,22 @@ Add the extension to your Sphinx configuration (``conf.py``): ] Once added, the extension will automatically generate the LLMs.txt files during the build process. + +How It Works +----------- + +During the Sphinx build process: + +1. **Content Collection**: Scans all of your documentation's ``_source`` pages and collects their content +2. **Directive Processing**: Resolves ``include`` directives by automatically incorporating their content +3. **Path Resolution**: Transforms relative paths in directives to full paths +4. **Output Generation**: Creates two optional files: + + - ``llms.txt``: A concise summary of your documentation, in Markdown + - ``llms-full.txt``: A comprehensive version with all documentation content, in reStructuredText + +5. **Content Filtering**: Allows you to exclude specific pages from the generated files + + +.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt +.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt diff --git a/docs/source/index.rst b/docs/source/index.rst index 8545323..09281ed 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,31 +9,13 @@ A `Sphinx`_ extension that generates a summary ``llms.txt`` file, written in Mar :maxdepth: 2 getting-started + advanced-configuration configuration-values contributing changelog -Features --------- - -Sphinx LLMs.txt provides the following features: - -- Creates ``llms.txt`` and ``llms-full.txt`` -- Automatically add content from ``include`` directives -- Resolves relative paths in directives like ``image`` and ``figure`` to use full paths - - Ability to add list of custom directives with ``llms_txt_directives`` - - Optionally, prepend a base URL using Sphinx's ``html_baseurl`` -- Ability to exclude pages - -Example -------- - -You can see this Sphinx projects `llms.txt`_ and `llms-full.txt`_ files as a simple example. - .. _Sphinx: http://sphinx-doc.org/ -.. _llms.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms.txt -.. _llms-full.txt: https://sphinx-llms-txt.readthedocs.io/en/latest/llms-full.txt .. |PyPI version| image:: https://img.shields.io/pypi/v/sphinx-llms-txt.svg :target: https://pypi.python.org/pypi/sphinx-llms-txt