ArcPy – How to Generate Python Toolbox Documentation from Pyt Files

Question

Python toolboxes are a great way to create custom, GUI toolboxes for ArcGIS without actually interacting with the GUI (since everything can be defined within the pyt file). Regrettably (and incomprehensibly) documenting a python toolbox does require the GUI (see Documenting a tool in a Python toolbox). This help page also states:

For Python toolboxes, the documentation for the toolbox and tools are stored in .xml files that are associated with the toolbox and tools by name.

In a question on this topic the user @dmahr stated in his answer:

Fortunately, Python supports for reading and writing XML, so you should be able to dynamically edit the help text from a separate script.

How would something like this be achieved? Since my toolbox is constantly changing, I'd like code and documentation to live in the same file.

EDIT:

As @user2856 commented, the first step would be to look at the xml file generated when documenting the tool via the GUI. This step should have been part of the question, so here's the xml for the tool named "Tool" (saved with the name New Python Toolbox.Tool.pyt.xml:

<?xml version="1.0"?>
  -<metadata xml:lang="de">
    -<Esri>
      <CreaDate>20190401</CreaDate>
      <CreaTime>10300800</CreaTime>
      <ArcGISFormat>1.0</ArcGISFormat>
      <SyncOnce>TRUE</SyncOnce>
      <ModDate>20190401</ModDate>
      <ModTime>14352000</ModTime>
      -<scaleRange>
      <minScale>150000000</minScale>
      <maxScale>5000</maxScale>
      </scaleRange>
    </Esri>
    -<tool xmlns="" toolboxalias="" displayname="Tool" name="Tool">
      <arcToolboxHelpPath>c:\program files\arcgis\pro\Resources\Help\gp</arcToolboxHelpPath>
      -<parameters>
      -<param displayname="Input Features" name="in_features" expression="in_features" datatype="Feature Layer" direction="Input" type="Required">
          <dialogReference><DIV STYLE="text-align:Left;"><DIV><P><SPAN>this is the in_features dialog explenation </SPAN></P></DIV></DIV></dialogReference>
          <pythonReference><DIV STYLE="text-align:Left;"><DIV><P><SPAN>and here is the scripting explenation</SPAN></P></DIV></DIV></pythonReference>
        </param>
      </parameters>
      <summary><DIV STYLE="text-align:Left;"><DIV><P><SPAN>This is a nice summary</SPAN></P></DIV></DIV></summary>
      <usage><DIV STYLE="text-align:Left;"><DIV><P><SPAN>This is how this tool should be used</SPAN></P></DIV></DIV></usage>
    </tool>
    -<dataIdInfo>
      -<idCitation>
          <resTitle>Tool</resTitle>
        </idCitation>
        -<searchKeys>
          <keyword>Here are some tags</keyword>
        </searchKeys>
    </dataIdInfo>
    -<distInfo>
      -<distributor>
        -<distorFormat>
          <formatName>ArcToolbox Tool</formatName>
        </distorFormat>
      </distributor>
    </distInfo>
    -<mdHrLv>
      <ScopeCd value="005"/>
    </mdHrLv>
    <mdDateSt Sync="TRUE">20190401</mdDateSt>
  </metadata>

Answer 1

This was bothering me probably more than it should have, and after getting to know python a little bit I've found that the problem is pretty easy to solve. Since I'm mostly an R user, these lines of python code might not be very pythonic. Feel free to comment / edit:

EDIT: Since the whole script is fairly long I've added everything to a Github Repo: https://github.com/ratnanil/PythonToolboxDocumentation

Step 1: Adding attributes within .pyt File

Add the documentation to the .pyt File as attributes to the objects. For example, adding adding a "Dialog Reference" for an individual Parameter can be created like so:

param0 = arcpy.Parameter(
    displayName="Input workspace",
    name="in_workspace",
    datatype="DEWorkspace",
    parameterType="Required",
    direction="Input")

param0.dialogref = "Enter explanatory information about this tool here"

You are free to choose your own names for the attributes (eg dialogref), just make sure you are not overwriting existing names (name, direction, value etc.).

Step 2: Create a python Script to extract the tool's documentation and write xml Files

This file will hold the code to extract all the documentation from the .pyt File and write the xml files based on this information.

import shutil
import importlib
import os
import datetime
import xml.etree.cElementTree as ET

# enter the filename of pyt-File WITHOUT extension
pythontoolbox = "mytoolbox" 

# this function copies the .pyt- to a .py-file (and deletes a previous version if it exists). We need this because I have not found a way to directly import a .pyt-file
def copypyt(pythontoolbox):
    pyfile = pythontoolbox+".py"
    pytfile = pythontoolbox + ".pyt"
    if os.path.isfile(pyfile):
        os.remove(pyfile)
    shutil.copy(pytfile, pyfile)

# run the above function to create a .py Copy of the .pyt-File
copypyt(pythontoolbox)

# imports the toolbox based on the "pythontoolbox"-Variable. The module is first stored in a variable for later use
import_toolbox = importlib.import_module(pythontoolbox)
import_toolbox

# since the module is stored in a variable, we can reload it in case the pyt file experienced changes during the session
importlib.reload(import_toolbox)

# import all the tools from the toolbox
toolbox_tools = import_toolbox.Toolbox().tools
# we could also import the toolbox label and alias, but we don't need in this minimal example
# toolbox_label = import_toolbox.Toolbox().label
# toolbox_alias = import_toolbox.Toolbox().alias

# Get today's date 
today = datetime.datetime.now().strftime("%Y%m%d")

# the following for-loop creates an .xml file for each tool, including all metadata extracted from the pyt file
for tool_i in toolbox_tools:

    # creates the first level tag named "metadata"
    metadata = ET.Element("metadata")
    # creates a sublevel tag named "esri"
    esri = ET.SubElement(metadata, "Esri")

    # Add todays date to the xml Tag "CreaDate", but maybe the creation date of pyt would be more appropriate
    ET.SubElement(esri, "CreaDate").text = today

    # I don't care too much about some of the elements in the xml file. In this example, I've included them none the less but filled with some dummy data. these are marked with a # *
    ET.SubElement(esri, "CreaTime").text = today             # *
    ET.SubElement(esri, "ArcGISFormat").text = "1.0"         # *
    ET.SubElement(esri, "SyncOnce").text = "TRUE"            # *
    ET.SubElement(esri, "ModDate").text = today              # *
    ET.SubElement(esri, "ModTime").text = today              # *
    scaleRange = ET.SubElement(esri, "scaleRange")
    ET.SubElement(scaleRange, "minScale").text = "150000000" # *
    ET.SubElement(scaleRange, "maxScale").text = "5000"

    tool_name = tool_i.__name__
    tool_fullname = str(tool_i).replace("<class '", "").replace("'>", "")
    tool_label = tool_i().label
    tool_description = tool_i().description
    tool_canRunInBackground = tool_i().canRunInBackground = False
    tool_category = tool_i().category

    tool = ET.SubElement(metadata,
                         "tool",
                         xmlns="",
                         name=tool_name,
                         displayname=tool_label,
                         toolboxalias="")

    ET.SubElement(tool,"arcToolboxHelpPath").text = "c:\\program files\\arcgis\\pro\\Resources\\Help\\gp" # *

    parameters = ET.SubElement(tool, "parameters")

    # this nested loop writes the documentation for all parameters
    for param_i in tool_i().parameters:
        param_displayName = param_i.displayName
        param_name = param_i.name
        param_datatype = param_i.datatype
        param_parameterType = param_i.parameterType
        param_direction = param_i.direction
        param_value = param_i.value

        # For all custom parameters, include a failsafe in case some attributes were note defined for some tools
        hasattr(toolbox_tools[1]().parameters[1],"pythonref")
        if hasattr(param_i, "dialogref"):
            dialogref = param_i.dialogref
        else:
            dialogref = ""
        if hasattr(param_i, "pythonref"):
            pythonref = param_i.pythonref
        else:
            pythonref = ""

        param = ET.SubElement(parameters,
                              "param",
                              name=param_name,
                              displayname=param_displayName,
                              type=param_parameterType,
                              direction=param_direction,
                              datatype=param_datatype,
                              expression=param_name)

        reference_style_before = "<DIV STYLE=\"text-align:Left;\"><DIV><P><SPAN>"
        reference_style_after= "</SPAN></P></DIV></DIV>"
        dialogReference = ET.SubElement(param, "dialogReference").text = reference_style_before + dialogref + reference_style_after
        pythonReference = ET.SubElement(param, "pythonReference").text = reference_style_before + pythonref + reference_style_after

    summary = ET.SubElement(tool,"summary").text = reference_style_before + tool_description + reference_style_after
    usage = ET.SubElement(tool,"usage").text = reference_style_before + "usage" + reference_style_after
    scriptExamples = ET.SubElement(tool,"scriptExamples")
    scriptExample = ET.SubElement(scriptExamples,"scriptExample")
    ET.SubElement(scriptExample,"title").text = "title for eg 1"
    ET.SubElement(scriptExample,"para").text = "some description for code sample 1"
    ET.SubElement(scriptExample,"code").text = "code sample 1"

    dataIdInfo = ET.SubElement(metadata, "dataIdInfo")
    idCitation = ET.SubElement(dataIdInfo, "idCitation")
    resTitle = ET.SubElement(idCitation, "resTitle").text = tool_label

    idCredit = ET.SubElement(dataIdInfo, "idCredit").text = "Credit whom credit is due" # *

    searchKeys = ET.SubElement(dataIdInfo, "searchKeys") 
    ET.SubElement(searchKeys, "keyword").text = "tag1, tag2, tag3" # *

    distInfo = ET.SubElement(metadata, "distInfo") # *
    distributor = ET.SubElement(distInfo, "distributor") # *
    distorFormat = ET.SubElement(distributor, "distorFormat") # *
    ET.SubElement(distorFormat, "formatName").text = "ArcToolbox Tool"

    mdHrLv = ET.SubElement(metadata, "mdHrLv")
    ET.SubElement(mdHrLv, "ScopeCd ", value = "005") # *
    mdDateSt = ET.SubElement(metadata, "mdDateSt",Sync="TRUE").text = today # *

    tree = ET.ElementTree(metadata)
    tree.write(tool_fullname+".pyt.xml")

Step 3: Create a python Script to extract the toolbox's documentation and write xml Files

Now you have created the most important part of the documentation (for each tool). In a similar manner, you can now create the documentation for the toolbox itself.