Metanorma updates

[opoudjis]

The Metanorma software stack is updated every two weeks. Updates consist of:

• A release of the gem metanorma-cli, https://github.com/metanorma/metanorma-cli/releases
• Release of all relevant updated gems that are tied to Metanorma, including the various Metanorma flavours specific to each supported SDO, and supporting gems. A list of all Metanorma and Relaton (bibliographic) repositories, including test documents and templates, can be found in https://github.com/metanorma/metanorma-release
• Update to the Metanorma docker image, https://github.com/metanorma/metanorma-docker, for those users using Docker instead of the command line to run Metanorma

Releases are kicked off on a Monday evening (Australian Eastern time 🇦🇺 ) , and can take a while to run to completion—especially as Metanorma is tested against a representative set of documents from all supported flavours as a precondition to release. There are also often issues in software dependencies which get in the way of testing successfully. For that reason, we will be posting to this thread only after each release propagates successfully through to metanorma-docker, to indicate to our users when the next release has successfully gone live.

We will include in this thread a summary of the updates made in the latest release. The release notes for each gem point to the GitHub issues that have motivated each update.

[opoudjis]

metanorma-cli version 1.9.6 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Dealing with Nokogiri update being more fussy about reading XML into other XML documents (affects SVG files)
  • Debugging handling of UnitsML formulas and of some mathematical formula rendering in Word
  • Significantly more efficient processing of numbers as mathematical expressions (which forces uniform number formatting for numbers entered as stem:[....])
  • More efficient generation of cross-references within documents, to deal with very large documents
  • Fixed inconsistent use of comma in cross-references to list entries (Clause 1, a); Clause 1, a) i) : comma in the latter was missing)
  • Debugged preprocessing of URL links in Literal blocks (to prevent character substitutions distorting the URLs)
• Document collections
  • Work leading up to generation of document collections in Word (multiple documents included in the one Word document)
  • Debugged hyperlinking between documents in a document collection
  • Debugged HTML index page for a document collection consisting of a single document, broken up into one HTML page per clause (sectionsplit)
  • Improved detection of document identifiers in a document collection, based on individual document metadata
• ISO
  • Corrected format for cross-referencing of annex subclauses to match preferred ISO style (Annex A, A.1.2, but Clause A.1)
  • Automatically infer the historical (pre-2014) PDF formatting to be applied to documents based on the copyright year attribute
  • Applied the ISO House style for localisation of numbers (identified by putting them in stem:[....])
  • Remove notifications of style violations for numbers being issued for every single number in a document
  • Stopped validating the language script used in documents (using Chinese no longer gives a warning)
• IETF
  • Use em-dash instead of en-dash after term modified
• IEEE
  • Debugged citation of miscellaneous bibliographic types (e.g. websites, theses): comma, not full stop, after title

The successful release of Metanorma on Docker will be announced separately.

[opoudjis]

metanorma-docker 1.9.6 has been released.

[opoudjis]

metanorma-cli version 1.9.7 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Allow unnumbered clauses in main section of document
  • Exclude automatically inserted Table of Contents heading in Word output, if no Table of Contents is supplied
  • Allow empty coverpage to be specified in Word output
• Document collections
  • Support Word output for document collections, putting multiple Metanorma documents into a single Word document
  • Support collection-specific cover pages
• JIS
  • Support Plateau documents

The successful release of Metanorma on Docker will be announced separately.

[opoudjis]

metanorma-docker 1.9.7 has been released.

[opoudjis]

metanorma-cli version 1.9.8 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Enable hover anchor links for section headings in HTML
  • Process lists, captions, notes as a single division of content in HTML, DOC output
• Document collections
  • Interleave individual documents with groupings of documents in collection manifests, instead of requiring subtrees of documents to all be the same depth
  • Include external URLs in collection manifest for file references, to redirect links to documents
  • Fix insertion of anchors for indirect anchors in collections, to work with unitary documents as well as section-split documents
• ISO
  • Include definition list title in the post-processing of table keys, so that "Key" title is rendered correctly
• NIST
  • Integrate pubid-nist gem, for uniform rendering of NIST identifiers in document metadata
• OGC
  • Label sourcecode and pseudocode as Listings in output
• Plateau
  • Create separate gem for MLIT-Plateau documents

The successful release of Metanorma on Docker will be announced separately.

[opoudjis]

metanorma-docker 1.9.8 has been released.

[opoudjis]

This fortnight's release of the Metanorma stack has been cancelled, as I am undertaking massive refactoring of collections processing. There will be a release in a fortnight, and I will be endeavouring to have that refactoring done by then.

[opoudjis]

metanorma-cli version 1.10.0 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Introduce hover-over highlighting and hyperlinks for clause headings in HTML output, so that links to particular clauses are easy to discover
  • Allow attachments to be specified for documents, and hyperlinks to attachments to be generated
• Compilation
  • Change --no-prefix, --no-install-fonts flags to --prefix, --install-fonts
  • Raise error flag on exit for illegal options
  • Temporarily remove namespace from Collection XML until external object mapper (Shale) enhanced to deal with namespaces better
• Requirements
  • If no anchor is specified for a requirement under Modspec, generate an anchor based on the requirement identifier
• Collections
  • Simplify and streamline collection manifest structure
  • Allow collection manifests to process files outside the current working directory
  • Allow nested collection manifest files
  • Provide external script hooks for collection manifest processing, to alter file identifiers, file locations, and manifest preprocessing
  • Use external object mapper (Shale) to process collection manifests as a class
  • Temporarily remove namespace from Collection XML until Shale enhanced to deal with namespaces better
• ISO
  • Do not display "Warning for WDs and CDs" for DIS, FDIS documents
• ITU
  • Support justification annex in ITU contributions
• OGC
  • Generate hover-over links for requirement headings as well as clause headings, to allow easy lookup of hyperlinks
• Plateau
  • Initial public release, all basic functionality now in place

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

Windows and Snap releases failed. Re-run in progress

[ronaldtse]

@CAMOBAP @maxirmx says that Tebako Windows is ready. I wonder if it makes sense speeding up migration of packed-mn. Just in case the fix for the current Windows release isn't easy.

[CAMOBAP]

Both Snap and chocolatey are released for now

But it's a good idea to migrate packed-mn to tebako from aibika just for a consistency reason.

Thanks for letting me know, I will do it in the scope of metanorma/packed-mn#228

[maxirmx]

@CAMOBAP let me test it first
Metanorma is really big and it may create some special issues.

[CAMOBAP]

metanorma-docker 1.10.0 has been released.

[opoudjis]

metanorma-cli version 1.10.1 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Use Plurimath number formatter, supporting scientific notation
  • Allow attachments to be specified for documents, and hyperlinks to attachments to be generated
• Collections
  • Debugging of SVG mapping to collection document destinations
  • Drop use of temp files for section-splitting in collections (one clause per HTML page)
  • Ensure Semantic XML is recompiled to update file links for section-splitting
• Plateau
  • Distinct clause labels for level 4-6 clauses

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

+

• The snap package is released https://snapcraft.io/metanorma
• The chocolatey package is released https://community.chocolatey.org/packages/metanorma/1.10.1
• The brew package is released metanorma/homebrew-metanorma@d97fce5

Docker release is blocked by metanorma/metanorma-docker#188 . Investigation in progress

[CAMOBAP]

• docker is released https://hub.docker.com/u/metanorma

[opoudjis]

This fortnight's release of the Metanorma stack will be postponed to the week of July 8, as I am going on holiday.

[opoudjis]

metanorma-cli version 1.10.2 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Fix HTML Table of Contents sidebar
  • Process number formatting configurations, introduce number:[] macro to support number formatting
  • Bug fix in naming of folder containing document attachments for output rendering
  • Change fontist option in compilation from no_install_fonts to install_fonts
  • Rationalise placement of predefined introductory text for Normative References and Terms and Definitions
  • Abort if empty table detected in markup
• ISO
  • Allow normative references as subclause of the main clause for 1980s document format, with correct placement of predefined introductory text
• IEEE
  • Consistent insertion of "Abstract:" heading in Word
• IHO
  • Process bibliography entered as subclause

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Docker: released hud.docker.com
• Chocolatey: https://community.chocolatey.org/packages/metanorma/1.10.2 - waiting for automated review
• Snap: https://github.com/metanorma/metanorma-snap/actions/runs/9851430058 - release in progress
• Brew: https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.2 - released

[CAMOBAP]

Snap is released https://snapcraft.io/metanorma

[opoudjis]

metanorma-cli version 1.10.3 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Debug number formatting, using significant digits, and rendering exponents in scientific notation correctly
  • Recognise cross-references in document abstracts
  • Improve processing of passthrough lines in document preprocessing
  • Set maximum permitted size for images when encoded as Data URIs (by default)
  • Capture Relaton error messages for fetching bibliographic data into the document error log
  • Allow Symbols and abbreviations as a title recognised automatically as a Symbols clause header
• Bibliography Rendering
  • Add URI to rendering of "misc" document type
  • Do not use italics or quotation marks to delimit titles in Japanese by default
  • Introduce internationalisation of document stages
• Internationalisation
  • Internationalise NO ID tag for missing document identifier in bibliography; provide Japanese for available from for ISO and inheriting flavours (including JIS)
  • Allow internationalisation of Metanorma flavours to annul any rendering of a term inherited from Metanorma
  • Do not convert line break to space within Chinese/Japanese/Korean paragraph text
  • Force smart quotes when quoted Latin text is next to Chinese/Japanese/Korean text without an intervening space
• Collections
  • Introduce bilingual text rendering as multiple columns (experimental)
• IEEE
  • Suppress NO ID for references missing a document identifier
  • Suppress ordinal identifiers in Normative References
  • Update Standards boilerplate text for 2024
• OGC
  • Use open source fonts
• IHO
  • Fix placement and numbering of References within the Overview clause
• JIS
  • Display table keys
  • Do not assume all definition lists in a table are table keys
  • Render subfigure and nested requirement cross-references correctly in Japanese
• Plateau
  • Insert Table of Contents after abstract
  • Add sources on paragraphs and lists
  • Indent all paragraphs in Japanese text
  • Unnumbered document history annex

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Docker - released - https://hub.docker.com/u/metanorma
• Chocolatey - pending automated review - https://community.chocolatey.org/packages/metanorma/1.10.3
• Snap - released - https://snapcraft.io/metanorma
• Brew - released - https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.3

[opoudjis]

metanorma-cli version 1.10.4 & 1.10.5 contain the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

ALERT: There is currently a problem with accessing the TeXGyreChorus font; as a result, Metanorma cannot process BIPM documents.

• General
  • Debugged handling of line-breaks in Metanorma (introduced in 1.10.3): carriage returns introduced by Asciidoc are normally treated as space in most scripts, but need to be stripped in Chinese/Japanese/Korean. The approach taken was not working for sourcecode embedded within examples; that has now been addressed. This has resulted in the hotfix of 1.10.5.
• Attachments
  • Debugged processing of document attachments, so that they can be processed when the document to be compiled is outside the working directory.
  • Debugged processing of document attachments, to exempt them from text substitution (which was occasionally distorting their Bin64 encoding).
  • Debugged processing of large document attachments, which were being truncated at 10 MB.
• Internationalisation
  • Improved stripping of carriage returns in Chinese/Japanese/Korean text, when followed by non-text or blank text in the next line.
  • Updated processing of multilingual metadata about publisher of document.
  • Internationalised Modspec requirements model for Japanese.
• IEEE
  • Cite documents other than standards in Normative References using Author-Date referencing.
• JIS
  • Support participant tables processing as prefatory metadata
  • Debugged rendering of investigative organisations and committees in document boilerplate
  • Treat JIS standards like ISO standards in bibliographic rendering: do not provide additional publisher details
• Plateau
  • Correct metadata about MLIT as publisher
  • Sort references in the ordering required by MLIT
  • Move revision history to front of document
  • Update the document identifier type from JIS to PLATEAU
  • Process commentaries, rendering them at the end of the document
  • Remove Word as an available output format
  • Do not label figures embedded inside other assets, such as tables
  • Adopt Modspec as default requirements model

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Docker is released https://hub.docker.com/layers/metanorma/metanorma/ubuntu-1.10.5/images/sha256-2f844149a4dd848548dcfa68d51f29a8a6d967c48ef49184f52c1804c65f08e8?context=explore
• Snap is released https://snapcraft.io/metanorma
• Chocolatey automation review in progress https://community.chocolatey.org/packages/metanorma/1.10.5

Other deliverables are on the way, to be updated...

[opoudjis]

metanorma-cli version 1.10.6 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Allow unnumbered annexes
  • Allow collapsible snippets in HTML output
  • Debug processing of hyperlinks whose text is itself a hyperlink
  • Debug escaping hyperlink from substitution insertion when it is quoted
  • Debug smart quotes processing surrounding a single XML link
  • Escape XML escapes like & in passthrough blocks
  • Improved error reporting of duplicate terms
  • Register plugins for LutamlGmlDictionary and LutamlEaDiagram
  • Support Plateau as a flavour loaded by default
• Requirements
  • Allow requirements to be allocated to different classes, so that they can be labelled and numbered separately than the default of Requirement, Recommendation, Permission
• Internationalisation
  • Internationalise requirements for Japanese
• Attachments
  • Ease processing burden by introducing linebreaks into the XML encoding of attachments
  • Allow arbitrary folder name for attachments in cross-referencing
• Bibliography
  • Remove duplicate bibliographic items, as they may be inserted by bibliographic modules independently of each other
• Generic, CSA, CC, IHO, Ribose, BIPM
  • Resolve customize YAML filepaths in the YAML configuration file relative to the configuration file location
  • Configure Fontist fonts manifest in configuration file
  • Configure pdf_stylesheet in configuration file
• ISO
  • Internationalise date_printing used in legacy ISO cover pages
• IEEE
  • Debug Tables of Content generation in Word for tables and figures
  • Optimise sorting of bibliography
• OGC
  • Move OGC rendering colours to Presentation XML, in order to enable shared formatting between formats
  • Put bibliography in front of annexes in Engineering Reports
  • Support style attributes in clauses for Engineering Reports, to indicate specific clause types
• JIS
  • Debug failure to populate cover page of Word output
  • Implement sorting order of references
• Plateau
  • Bring list and clause numbering in line with ISO
  • Sort both normative refs and bibliography according to JIS sort order

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Docker is released - https://hub.docker.com/r/metanorma/metanorma/tags?name=latest
• Chocolatey is released - https://community.chocolatey.org/packages/metanorma/1.10.6
• Snap is released - https://snapcraft.io/metanorma
• Brew is released - https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.6

[opoudjis]

metanorma-cli version 1.10.7 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Format numbers in formulas
• Internationalisation
  • More Japanese internationalisation terms
  • Mark up boilerplate for terms and normative references in internationalisation config files as Asciidoc, not XML
• Attachments
  • Prioritise attachment URLs over citation URLs in attachment linking
  • Process unembedded attachments
• Generic, CSA, CC, IHO, Ribose, BIPM
  • Allow overriding of output formats in configuration
  • Resolve path of :customize: config file correctly
• ISO
  • Internationalise website
• JIS
  • Strip final period in references
  • Strip italics and smallcaps in references template
• Plateau
  • Use JIS, not ISO rendering of references

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

docker, snap, brew and choco are released

[opoudjis]

metanorma-cli version 1.10.8 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Do not numbers in formulas by default
  • Debug rendering of amendments, and of person/organisation identifiers within bibliographic metadata
  • Document Semantic and Presentation XML grammar
• Internationalisation
  • Support locale in internationalisation of bibliographic metadata
• ISO
  • Validate that figures, tables, formulae and annexes are cross-referenced at least once in documents
• OGC
  • Make figures and sourcecode collapsible in HTML

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Doker https://hub.docker.com/r/metanorma/metanorma/tags?page_size=&ordering=&name=1.10.8
• Chocolatey https://community.chocolatey.org/packages/metanorma/1.10.8
• Snapcraft https://snapcraft.io/metanorma
• Brew https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.8

[opoudjis]

metanorma-cli version 1.10.9 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Add large_notation, number_sign options to number formatting
  • Allow number formatting on inline-rendered MathML expressions
  • Strip space after inline-rendered MathML expressions
  • Do not strip space before element in block in multiline text, when final line of block does not end in carriage return
  • Shrink font used for superscripts, subscripts in HTML
  • Allow nested tables within other assets (e.g. examples) to be labelled
  • Filter on error log by severity or category
  • Add LutamlKlassTableBlockMacro, LutamlEaXmiPreprocessor plugins
• Collections
  • Document collections schema
  • Sanitise title of sections in sectionsplit mode
  • Ensure document attachments are moved to destination folder when parsing collections
• IHO
  • Updated copyright statement
  • Hierarchical numbering of assets within clauses
  • Align clause reference rendering with IHO S-102
  • Remove period after clause number

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Doker https://hub.docker.com/r/metanorma/metanorma/tags?page_size=&ordering=&name=1.10.9
• Chocolatey https://community.chocolatey.org/packages/metanorma/1.10.9
• Snapcraft https://snapcraft.io/metanorma (on the way)
• Brew https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.9

[opoudjis]

metanorma-cli version 1.10.10 contains the following changes; more details can be found in the individual gem release pages. PDF rendering updates are not tracked separately.

• General
  • Optimise document includes validation
  • Restore UnitsML markup
  • Add abstracts, classification, and images to spans-formatted data in bibliographic entries
  • Allow embedded document headers to be processed as different flavours from the current flavour
  • Introduce consistent naming of modules with titlecase (Metanorma::Ieee not Metanorma::IEEE)
• Collections
  • Optimise collections processing of links between documents in the collection
  • Introduce recompile-xml directive, to make recompilation of XML documents in the collection optional
  • Optimise sectionsplit by removing the Semantic XML of clauses other than the current clause from Presentation XML
  • Add flavor directive with validation, to indicate which Metanorma flavour to compile the collection manifest with
• ISO
  • Support ISO Addendum document type
  • Add the has-draft relation to indicate the latest prepublication draft of the current published document
• NIST
  • Move document boilerplate from XML and code to external Asciidoc, as with the rest of Metanorma

The successful release of Metanorma on Docker will be announced separately.

[CAMOBAP]

• Docker on the way (workflow restarted due to network issue)
• Chocolatey https://community.chocolatey.org/packages/metanorma/1.10.10
• Snapcraft https://snapcraft.io/metanorma - ready
• Brew https://github.com/metanorma/homebrew-metanorma/releases/tag/v1.10.10