Alternatives to protocols

[mynomoto]

I started to think about this problem with the idea of using protocols but I'm not sure anymore.

One alternative is to create a library based on Polylith components (@jeroenvandijk suggested that on slack). I like this alternative because we can have all the implementations over the same repository and make sure that they behave functionally the same. If you want you can always add a specific library and use it but you could use only one version of the polylith based library if we are sneaky using versions. (a version could be something like "1.0.0-cheshire" for a json client based on cheshire, "1.0.0-jsonista" and so on for other libraries)

So each implementation would be using the same namespace but each published library would have only one of the implementations because of how we can use polylith. Changing the implementation would be as easy as changing the version.

If one dreams big we could think about catching exceptions and throwing a common exception so user code would not have to change at all when changing implementations.

WDYT?

[borkdude]

I don't have a lot of experience with polylith, but I don't want to include any other dependencies to solve this problem: it should also work out of the box in babashka for example. Keeping it bare-bones is my preference.

[mynomoto]

I think the main point we disagree is that I think this should be centralized and you think it should be decentralized.

The ideia of one namespace having multiple implementations only makes sense imo if this is centralized. Otherwise protocols, multimethods or a way to register seem better options.

I think that for the implementation to be swappable it should have the same behavior including throwing the same exceptions or else we would be setting users up to be surprised. I'm not sure if this is possible/practical but it would be great to have. I will try to write a version of the centralized idea soon to find if it is a crazy idea at least.

[borkdude]

Wether or not it's centralized or not: could be, I'm not disagreeing necessarily, but ultimately, like say with clj-kondo configs/hooks, the maintenance of implementations belong to the library authors, not the project that defines the interface. So there should also be a way to provide implementations "out of band".

[borkdude]

Of course the interfaces should be good enough to accommodate several libraries, that's where I see the most benefit of test-driving the implementations, in addition to providing them for others.

[borkdude]

Providing an out of band implementation for clojure.tools.logging is done by timbre as follows:

https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre/tools/logging.clj

What I like about that approach:

Using timbre as a tools logging provider is explicit. There is absolutely no confusion which provider "wins", compared to the "who is first on the classpath" approach.

(taoensso.timbre.tools.logging/use-timbre)

What I currently don't like about the specific mechanism in tools.logging:

• the reliance on alter-var-root: this could give problems with direct linking and also isn't cross-platform (CLJS). Using an explicit registration function (so the internals of the "framework" can decide how functions get registered) seems an improvement.

• usage of protocols leads to either reify or defprotocol. We could just use maps and functions. An explicit registration function can do some sanity checks on the map with functions.

Both of these problems are addressed in the below approach:

#3 (comment)

[borkdude]

Another point: as long as the registration function is explicit, it doesn't even matter if we use protocols or whatever, this can be decided on a case by case basis I guess: whatever makes sense for the tool at hand.

[borkdude]

Another approach could just be to not use protocols or multimethod but a light-weight plugin system.

This library defines functions that delegate to an implementation.

(ns clj-easy.tools.misc.json)

(def ^:private write-str-impl (volatile! nil))

(defn write-str [s] (@write-str-impl s))

(defn register-json [{:keys [write-str]}]
  (vswap! write-str-impl write-str))

When loading an implementation, it can register itself:

(ns clj-foobar.library (:require [clj-easy.tools.misc.json :as tmj] [cheshire.core]))
(tmj/register-json {:write-str cheshire.core/generate-str})

The clj-foobar.library namespace registers cheshire as the implementation. We could provide this namespace so it can be used by app users and library users for testing only.

We could generalize this to a keywordized system, like (reg! :clj-easy.tools.misc.json/write-str cheshire/generate-str) as well.

We could also warn and check if multiple libraries register implementations for the same thing, which imo should not happen in practice (since then one of the libraries on the classpath does not play by the rules that only end-users should provide implementations).

[borkdude]

I agree in general that we should avoid using protocols on the API surface. The API surface of each tool can just be a registration function. I've committed that as an example. The tests now look like:

(ns clj-easy.tools.misc.json-test
  (:require [clj-easy.tools.json :as json]
            [clj-easy.tools.json.cheshire :as tools.cheshire]
            [clj-easy.tools.json.data-json :as tools.data.json]
            [clojure.test :refer [deftest is testing]]))

(tools.cheshire/reg-provider!)
(tools.data.json/reg-provider!)

(deftest read-str-test
  (doseq [provider '[cheshire/cheshire org.clojure/data.json]]
    (json/choose-provider! provider)
    (testing "read json"
      (is (= [1 2 3] (json/read-str "[1, 2, 3]")))
      (is (= {:a 1} (json/read-str "{\"a\": 1}")))
      (is (= {:a 1} (json/read-str "{\"a\": 1}" {:key-fn keyword})))
      (is (= {"a" 1} (json/read-str "{\"a\": 1}" {:key-fn str}))))
    (testing "write json"
      (is (= [1 2 3] (json/read-str (json/write-str [1 2 3]))))
      (is (= {:a 1} (json/read-str (json/write-str {:a 1})))))))

Note that there is no problem with having multiple providers on the classpath: registration and choosing a provider is explicit.
When choosing a provider while there was already one chosen previously, a warning is emitted:

$ clojure -M:test

Running tests in #{"test"}

Testing clj-easy.tools.misc.json-test
WARNING: json provider was already set to: cheshire/cheshire and is now set to: org.clojure/data.json

Ran 1 tests containing 2 assertions.
0 failures, 0 errors.

Thoughts, @mynomoto ?

[bsless]

Coming over from other discussions I've opened, I have an objection to registering. I can see it blow up in my face in some creative way in the future, the joys of side effects.
Alternative solution? Forgive me for uttering the phrase, but "factory"?
(def parser (json/provider :cheshire) where the provider returns a function.
I'm not a fan of global registries and definitely wouldn't want to introduce a volatile read on the code path for no reason.

[borkdude]

I don't understand where is that API used from and how. What does json/provider do? I think it's not a very good UX if this means you would have to define the read and write vars yourself again to wrap those functions. Since how can libraries program against a contract this way?

[jeroenvandijk]

The current implementation for json looks good to me. Since parsing json and writing json is stateless it is unlikely that you have to switch the provider more than once (in a normal program).

For http things are more complicated as it is more likely you will swap the implementation and it needs to be threadsafe (e.g. in test environments of the users)

Some remarks:

• Maybe the read and write function could be wrapped in a function that throws when it is nil to prevent a cryptic nullpointer exception when it is not set
• Should the provider be selected the moment you register it, so it is only one step of requiring the namespace? I wonder what the use case is of choosing a provider other than in test environments?
• Choosing a provider could also have an extra check if the provider exists to catch errors early on

[borkdude]

It looks like you're reacting to @bsless's comment here, but assuming you are talking about the current JSON impl: good points.

I think it's up to the library that offers the implementation if it will register on namespace require, we can certainly do so for our default implementations, so it will be one step less.
You could even "choose-provider" automatically on namespace require. Then the require of the impl namespace does both things, so it will all be automatic.

The NPEs: sure that's easy to solve. You can even check more things in the "choose-provider" function, e.g. the existence of certain keys.

[jeroenvandijk]

Yeah sorry, indeed about the current implementation. You seemed to have continued after the discussion with @bsless, but it was a bit confusing. And I still need to get used to this discussion interface as well :)

Agree about the default implementations. And more specific checks will be helpful indeed.

[borkdude]

@bsless commented about the use of volatile!:

I think in pretty much all cases I can think of: JSON, HTTP Requests, etc, reading a volatile is neglectible compared to the rest of the work done in that function. An alternative could be using alter-var-root but that doesn't play well with direct linking.

@mynomoto is trying out a polylith-like approach where there is only one clj-easy.tools.json namespace which defines the functions and it depends on your dependency on the classpath who brings that namespace. That could also work (and there would be no intermediate volatile, registration, etc) but could become confusing when you have problems with dependencies. The system I have currently for JSON allows having multiple providers on the classpath and choosing them explicitly.