Implementation badges to signal support and compliance

[Relequestual]

JSON Schema has a problem. New users find it hard to pick a library. Not because there aren’t enough to choose from, but because developers are unclear what versions of JSON Schema are supported, and how compliant the implementation is in reality.

When we receive a PR to add an implementation to our listing, they must also provide the versions of JSON Schema they support. We break up the listings, only showing those that support draft-07 and up on the main page, relegating implementations that only support older versions to a legacy page.

Even though knowing which version of JSON Schema an implementation supports is required, often the library does not self declare which version it supports in its own readme or documentation. Developers should be able to pick an implementation with a level of confidence in compliance with the specification as this ensures the level of interoperability they require.

Let’s try the simplest idea that has the highest gain to effort ratio. A simple badge.

https://shields.io is a long-time provider of many ready-made and dynamic badges for repository readme files. These are most well known for showing build status, warding off would-be users from projects which have a broken build without a fix for years.

I propose we create a number of ready to use badges for implementations to add to their readme, with varying levels of detail. We can investigate a number of improvements should the basic approach be successful and useful.

Here’s some examples of the most basic badge we could enable maintainers to construct.

(may be better using bar | )
(https://img.shields.io/badge/JSON%20Schema%20Version%20Support:-draft--07,%202019--09,%202020--12-green)

These would be self assertions.

We can shorten these by using the JSON Schema icon, which is supported by using a data URI base64 encoded image.

We can make these badges link to anywhere we like by default, although users could change them, or any of the contents, if they so wished. We can have the left side and right side go to different locations.

We could create a small webapp to allow these badges to be built easily by implementers, making them more likely to use it, and allow us to set style and link expectations.

It is also possible using the shields.io service to make badges dynamic. Although the test suite doesn’t publish releases as such, we could use commit hash, and include that in the badge URL, and compare it against the latest commit, to check how up to date the test suite is.

[jdesrosiers]

How exactly would these badges work? Is it honor system, or would it be backed up by actual verifiable test results? If it's not backed by anything, there's no difference from what we have now, which is people making claims on their readme that even if honest are probably out of date.

We'll have to think carefully about what we want those badges to say. "Partial compliance" can mean a lot of things. Many implementations (including mine) have reasons to make harmless deviations from the spec. I think calling my implementation partially compliant wouldn't be appropriate especially considering it would have the same status as an implementation with missing features or major bugs. Even including a percentage of tests passed wouldn't be quite right. We have several tests that cover behaviors that are technically part of the spec, but would never happen in a real schema. Implementations that fail those tests shouldn't have the same badge results as an implementation that fails the same number of tests, because of missing features or significant bugs.

One way to address this could be to label each test with a severity label (severe, minor, trivial) and report on the number of each type of failure. Another approach could be feature based. So, if an implementation passes everything, but doesn't support dynamic references or external references or whatever, the badge would be specific about those missing features and users could be confident in the implementation if they don't rely on those features. But this is getting complicated already, but there might not be an uncomplicated way to do this.right.

[Relequestual]

How exactly would these badges work? Is it honor system, or would it be backed up by actual verifiable test results? If it's not backed by anything, there's no difference from what we have now, which is people making claims on their readme that even if honest are probably out of date.

Initially an honor system, later with verifiable tests.
The problem currently is most implementations don't actually list thier supported versions. And, I would bet most people don't go to the JSON Schema site to look for an implementation, and find it via the appropriate package management service, such as npm.

We'll have to think carefully about what we want those badges to say. "Partial compliance" can mean a lot of things. Many implementations (including mine) have reasons to make harmless deviations from the spec. I think calling my implementation partially compliant wouldn't be appropriate...

Interesting. I'd be interested to hear which of those tests you don't pass and consider harmless deviations. The readme says it supports all the required features, which to me reads as fully compliant. If there are required tests which cover beyond required features, that's an issue.
(This is the exact reason I raised this as a discussion. This is good discussion!)

One way to address this could be to label each test with a severity label (severe, minor, trivial) and report on the number of each type of failure. Another approach could be feature based. So, if an implementation passes everything, but doesn't support dynamic references or external references or whatever, the badge would be specific about those missing features and users could be confident in the implementation if they don't rely on those features. But this is getting complicated already, but there might not be an uncomplicated way to do this right.

To me, the level of compliance is a lesser issue than just getting implementations to self identify their intended compliance/dialect suport. We know we need to extend the capabilities of the test suite, and I believe @karenetheridge has put quite a lot of thought into it (I'm sure others have too, just this is what I remember off the top of my head).

One other thought on compliance: You can assert test suite compliance, but which VERSION of the test suite? We don't publish versions currently last time I checked.

[karenetheridge]

I would like implementations, at a minimum, to clearly list in their own documentation what spec versions they support, and where their deviations are (and maybe the implementations page can have links to that specific part of their docs too). We'll probably have to write an article giving guidance for this -- e.g. anything in the spec that's a MUST is mandatory (and we have tests for most of these things, but some are outside the scope of the test suite, at least for now), and anything that's a SHOULD should also be documented if it is deviated against also, with explanations why that decision was made.

One other thought on compliance: You can assert test suite compliance, but which VERSION of the test suite? We don't publish versions currently last time I checked.

This is a good point. "compliant with the test suite as of <date>" would be good, when we start collecting this data, and maybe links to the results - e.g. https://github.com/karenetheridge/JSON-Schema-Modern/blob/master/t/results/draft2020-12-acceptance.txt

[jdesrosiers]

I'd be interested to hear which of those tests you don't pass and consider harmless deviations.

We have tests that check that core keywords like $id and $anchor are only treated like keywords when they appear in a sub-schema. For example, an $anchor in an enum value is not actually an anchor. Not handling this properly is harmless because there's no good reason in a real-life schema to have an $anchor in an enum.

When the $id and $anchor keywords are being processed, I don't have a way to know whether an object is a sub-schema or just an object. Fixing this in a way that doesn't hard-code to specific dialects is not worth the complexity it adds or the effort to make it happen. Not for something that never happens in real schemas.

Another example is that $id changed in draft 2019-09 from being just a base URI change to being an embedded schema. I treat it as an embedded schema in all dialects. This means that I don't support JSON Pointers crossing schema boundaries in older drafts even tho it's allowed. Not only is there no reason to ever write a schema that does that, it's always a bad idea. So, again, not supporting something that never happens in real schemas is harmless.

[karenetheridge]

Not handling this properly is harmless

I disagree. Schemas could be contained in the examples keyword, and these should definitely not be treated as real schemas. Identifiers found in these faux schemas can interfere or conflict with real schemas, causing errors at parse time (duplicate identifiers) or at runtime (do you really want your $ref going into an examples section, rather than its intended location)? Not only did this not "never happen", it happened to me last year with another implementation and was one of the factors involved in me deciding to write my own.

I found it fairly straightforward to handle this by knowing what keyword is being processed while traversing recursively through the schema (which is done before evaluation to find $schema, $id, $anchor and $recursiveAnchor keywords). For example, when the $defs keyword is found, we know that the values two levels below that are schemas, so each of those structures are traversed knowing that they're schemas; for not or items, the schema is one level deep; for prefixItems the schemas are in the array items, etc.

(As a corollary, this made it possible to add support for callback functions on specific keywords, which made implementing an OpenAPI document handler trivially easy -- while evaluating the openapi document against its schema, a callback for $dynamicRef collects all the places where the destination uri reference is #meta, and then I can traverse those identified schema locations for embedded identifiers; likewise a callback on $ref with a target of #/$defs/operation identifies all operation locations, making it easy to collect all operationIds for later use by the application.)

This means that I don't support JSON Pointers crossing schema boundaries in older drafts even tho it's allowed

Do we have tests for these? I thought we took them out because the spec said that it wasn't mandatory to support this.

[jdesrosiers]

Not handling this properly is harmless

I disagree. Schemas could be contained in the examples keyword

I still can't imagine this being a problem. What reason would there be to have a schema in examples that has $ids that conflict with the $ids in the schema? I would love to hear more about your real-life case that you encountered, but I don't want to derail this discussion any more than we already have. I think we can at least agree that it's extremely rare. My point was that treating two implementation that each fail three tests as the same is not helpful. If one of those implementations fails tests for cases that almost never happen and the other fails tests for common functionality, there's a huge difference in the level of compliance of those two implementations despite their test results being similar.

This means that I don't support JSON Pointers crossing schema boundaries in older drafts even tho it's allowed

Do we have tests for these? I thought we took them out because the spec said that it wasn't mandatory to support this.

We took them out for 2019-09 and up. They are still there for older drafts.

[Julian]

There are some existing thoughts on this in json-schema-org/JSON-Schema-Test-Suite#314.

[Relequestual]

Issue json-schema-org/JSON-Schema-Test-Suite#314 is closed in favour of bowtie-json-schema/bowtie#211 which is linked to bowtie-json-schema/bowtie#55

I'm closing this Discussion as we have a solution to add badges based on actual running tests.
It may be a little work to obtain the badge, but a little hard work never hurt anyone, right?

[Relequestual]

I think we need a new issue to track adding documentation to a JSON Schema org controlled location regarding Bowtie and badging for implementations, once bowtie-json-schema/bowtie#211 is resolved.