Help with Blocks Extension

[facelessuser]

Posted originally by @tiangolo in #1973 (comment)

I'm trying to build a blocks extension but I'm not sure it's the right approach. 😅 (sorry for the long post).

I want to include multiple source file examples inline in multiple tabs in a summary, from a new block (?) with a config.

Use Case

In my docs I have some sections that include external files inline. For that I use mdx-include.

Now, in many cases, I have several versions of the same file, with old syntax for Python 3.8+ and newer syntax in 3.9 and 3.10. I also have versions using typing.Annotated and without it (only in the FastAPI docs for now).

So, it's a lot of different Python example files for the same example (a matrix of combinations). Then I include them in tab blocks.

Here's an example from SQLModel: https://github.com/fastapi/sqlmodel/blob/1eb40b1f33aaebd02f03fd1986e242b65c79d573/docs/tutorial/fastapi/read-one.md#L17

Currently, I include some lines of the file (not the entire file). And then at the end, I include a details block with the entire file (in case people want to copy-paste or check the entire thing), I currently do this only in the SQLModel docs.

---

I have been wanting to refactor it, to make it much less verbose to write in Markdown and automatize including the other versions (Python 3.10, 3.8, Annotated, etc).

My idea is to have one main file shown and include only the specific lines for that file, the highlighted lines only for that file, and then have a details block below including tabs with all the versions, with the entire files.

I imagine it could be something like this:

/// include
src=docs_src/tutorial/fastapi/read_one/tutorial001
lines=1-2
lines=59-65
hl_lines=59-60 62 65
///

Then the (block?) extension would look for that source file to include, check what versions are there, include inline the highest version available of the file (e.g. Python 3.10 with Annotated), only with the lines specified, add a comment in between the included lines like # Code here omitted 👈, and add the highlighted lines.

I tried for a few days to follow the tutorials and read all the docs for extensions with the Markdown package, the pymdown blocks docs, and the docs for xml.etree.ElementTree, a bit of the internal code for mdx_include and pymdownx blocks. But I have been struggling a lot to figure out how to do it all, in particular, how to modify the content, and also make it processable by the other blocks later (tab, details). 😅

Maybe my whole approach is wrong, because maybe don't need a block that wraps content, only has metadata that is then used to include new content on the fly. I also see that mdx_include is a preprocessor instead of a block processor and maybe that's what I should be doing instead of trying to do it with the block processors, not really sure.

Side note, do you do any freelance work @facelessuser? Would you be willing to help me out start this? If so, please contact me on my email (I didn't find a way to reach out to you). 🤓

[facelessuser]

@tiangolo I want to make sure I understand what is being asked. You want to have a single file with multiple examples, but have an include block that will grab multiple examples (specified by lines) and then highlight specific lines?

So, this:

/// include
src=docs_src/tutorial/fastapi/read_one/tutorial001
lines=1-2
lines=59-65
hl_lines=59-60  62  65
///

Might generate:

```
1. code
2. code
```

```hl_lines="59-60 62 65"
59. Code
...
65. More code
```

Is that what is being asked?

[facelessuser]

Okay, the problem with Blocks extension in this context is that it runs after the preprocessor. It's handled alongside normal blocks like paragraphs, etc. mdx-include is run as a preprocessor and so is SuperFences, so a Blocks extensions would come after. So even if you generate all this stuff with Blocks, the preprocessor that normally run on content will not be run on this content. This means fences won't be parsed, includes won't be parsed, some basic preprocessing that is expected won't be applied.

Just off the top of my head, you might be able to handle some of this special, inlined into your logic:

You could likely construct and manually run mdx-include on specially formatted lines to return your file content., Then format code to run through the SuperFences formatter.

md.preprocessors['fenced_code_block'].extension.superfences[0]['formatter'](
    src=my_code,
    class_name="highlight",
    language='my-language',
    md=md,
    options=options,
    **kwargs
)

And inject the formatted code into Markdown specifying tabs and such...then let the Blocks extension run that content through the normal Markdown processor....

I think it could actually work.

[tiangolo]

Get it, thank you! That gives me a lot of clarity, it seems I was trying to use the wrong mechanisms for my problem. 😅

I'll try with these ideas. Maybe it's even simpler to write a preprocessor that runs before mdx-include and SuperFences and prepares the markdown for them to process. 🤔

[facelessuser]

Maybe it's even simpler to write a preprocessor

Depending on your needs, maybe. For a robust solution, it can get quite tricky. You need to handle your content embedded in blockquotes, not process it in block html at times unless the md_in_html is enabled for that tag, make sure you account for indentation, but recognize when the indentation is intentional for code blocks. There's a lot of context that isn't directly available in a preprocessor step.

If only you are using it, and you are aware of the weakness you have not addressed, it can be fine though and you can cut corners, but if you are providing something robust, there's a lot to think about, which is why SuperFences is so complicated. Python Markdown will strip out extraneous new lines, but in code blocks you want those as they can have meaning, so fenced code must be handled as a preprocessor. I have to store content and then restore it in cases where we were too greedy and converted a code block that was in an indented code block, or in HTML that wasn't meant to be processed. Preprocessors are easiest to handle if you don't care about the context of the document outside of what you are preprocessing, like maybe templating variables, etc.

Anyway, some stuff to think about.

[tiangolo]

Awesome, thanks for all the input! 🙇

And yeah, as it's a very constrained use case, I don't need to support anyone else using it, I can probably do it in a simple/dirty preprocessor.

[facelessuser]

Figured that might be the case, this can make things quite a bit easier.