Sarama for low-level Kafka protocol manipulation

[MadsRC]

I've got a use-case where I'd like to receive Kafka protocol messages, parse them and respond to them. Sarama seems like an obvious choice since it has many of the low-level primitives available for use in the source code.

However, I've found that Sarama seems to be more geared towards being a Kafka consumer/producer, and not towards acting as a broker - which is understandable since that is where the most users are. Much of the functionality needed in order to manipulate the Kafka protocol isn't available to users or Sarama due to not being exported.

By changing a few symbols to be exported, Sarama can successfully be used to decode, modify and encode Kafka protocol messages. I've got a fork with some fairly naive changes that allow for this here: https://github.com/MadsRC/sarama/commit/a86ff8547b4bd0c77d556ea2efcfc3f30c641d62

My idea would be for a new feature would be to make the Decode, Encode, request.CorrelationID and request.Body available to users of Sarama. One naive approach would be to just export them, but perhaps a better alternative is to have a low-level/protocol level subpackage of Sarama? Something that just makes the protocol primitives available for use, along with methods that allow for the encoding and decoding of these.

[puellanivis]

I would, in general, recommend against exporting protocol features in the same package that the primary API usage is located. They are kind of cross-purpose, one being a thin-ish nearly transparent wrapper of the protocol, and the other a higher-level simple to use API. And having both exported in the same package would pollute godoc information with a mix of documentation for both purposes, meaning a simple API is buried in protocol details, and protocol details are littered with a high-level API design.

[mrh-chain]

I couldn't agree more - My fork with my changes are only to serve my own needs to get my project moving along, while this discussion is intended to find a way make that functionality available (the low-level functionality) via Sarama as opposed to creating a fork or creating something from scratch.

Perhaps an acceptable solution would be, as mentioned at the end of my original post, to create a package, or perhaps a separately versioned module, for the low-level stuff (the "thin-ish nearly transparent wrapper of the protocol" as you put it) and move the functionality to that module?

[puellanivis]

This was the idea we had for pkg/sftp. To move the protocol stuff into a separate package, and then make the main package use that protocol package. It’s stalled out in actual rolling it out, as F/OSS is wont to do, but I still liked the principal of the matter.

[dnwe]

This was always sort of on the roadmap for /v2 of Sarama because currently there’s a few places where the internal protocol types are returned to the user’s client calls, which is undesirable, so we would want to replace all of those with concrete types and then move the protocol code to another package.

The protocol package itself would probably be unversioned (to allow for complete freedom of changing types/fields) though

[mrh-chain]

@dnwe that is great to hear!

Is there any place this thing, or the roadmap for /v2, is being actively discussed?

[dnwe]

Initial discussions happened on #1732 but a few other things came up that needed more urgent attention. It’s a tricky one because we sort of wanted to make as many breaking changes as possible before cutting the v2, but perhaps we should be more conservative and break things up into a series of v2, v3, v4 style staging out