OCapN Pre-standardization Group

March 2024


Called to order.

General topic for today from the end of last meeting: concrete serialization format

Jonathan proposes:

Jonathan: Issue on agenda #93, document the requirements. There’s a tension between this and suggestion from Mark and Christine that we’re almost done with this.

Christine: put something together. Mark asked if we were willing to agree on Smallcaps? The generalized version of the question is “Should we get concrete about concrete syntax”? Jessica and I have tried to take a requirements approach. We put together a presentation. I’m nervous because we got a lot out of the abstract syntax and this is an unusually emotionally charged topic.


#93 OCapN serialization Requirements / Analysis

(Christine gave presentation, the Slides are available here

Christine: There are two proposed, three approaches. SmallCaps and Syrup, also make something together that fits the requirements.


In both formats these have been addressed.

Jonathan: This seems a bit terse.

Christine: Before we move forward, how do people feel about these?

Mark: There are “Requirements” and “Additional Good Things”. Canonicalization and Range Queries are similar. They both require order preserving encoding. They are not necessary for interop but nice things. Compactness is another nice thing, likewise compatibility with off-the-shelf tools. Some are in tension with each other.

Mark: None of our use of SmallCaps and communication among Agoric parties require canonicalization. It is nice to have from our point of view. So there's logically a list of nice-to-haves and then a separate list of actual requirements. As a practical matter, for Spritely and Agoric to get started, is that it works with existing entranched stacks trying to talk OCapN. There is no one such stack between Spritely and Agoric. The reason I say that is that it obviously I am going to propose SmallCaps even without support for binary data or even a concrete proposal for how to do so—but since the rest of our stack lacks the ability to deal with binary data, interoperability can get started before we have a resolution on how to do binary data. Practically, I would like us to get started with interop expericments and I think SmallCaps is a good way to do that.

Christine: I want to reïterate some of the things Mark said and follow through with this presentation. Mark agrees abstract data types are necessary. Binary is necessary in some form.

Christine: Easy access is somewhat encompassed by the issue of existing tools. The biggest source of potential contention is canonicalization. Jessica doesn’t see it as a requirement but sees it worth working through if it is. Mark brought up range queries and canonicalization are nice to have.

Christine: Canonicalization is the only point up to debate. And range queries and integration with other tools would be nice to have. Is that a good summary of Mark's position?

Mark: Yeah, but I don’t think there’s a conflict between canonicalization and SmallCaps. I don’t think it’s a blocker.

Christine: Plus one on getting to work on interop soon. Let’s look at the approaches.

Christine: In some ways, it's more like Syrup vs. JSON than Syrup vs. Smallcaps. Baldur also mentioned CBOR.

Christine: SmallCaps has a simple textual format. It uses a base serialization format, the widest supported. jq already supports SmallCaps for processing things for developer convenience.

Christine: Syrup is effective as a binary format, takes about an hour to implement, and natively canonicalizes.

There could be a third thing and we do not.

Mark: Agoric has a third thing, "CompactOrdered", which has matured a great deal since last meeting.

Jessica: I remember asking whether Agoric was thinking of having Compact Ordered for the wire format. Is it still the case we are not thinking of it for the wire format?

Mark: We have no plans to use Compact Ordered on the wire. If it turns out that it’s the only thing OCapN can agree on, for the purposes of interop, we could use it as the wire format.

We already speak SmallCaps a lot and there are always issues to changing due to momentum with entrenched infrastructure. If we would do anything on the wire that is not SmallCaps, it would be CompactOrdered binary (henceforth "COB"). For tooling reasons, the thing we have merged is represented in text, but it is clearly a proof of concept for CompactOrderedBinary. From what we have working and tested, we know what that format would be like and we would be happy to write a spec for Compact Ordred Binary.

Christine: I would love to see a spec for COB. You said something specific I would like to capture. You said Agoric is using SmallCaps but you would be willing to move to CompactOrdered, maybe with a bridged. If you are sufficiently entrenched that you need a bridge. Are you willing to extend the bridge, you would extend to any binary format that the group comes to consensus on.

Mark: Yes, but that's almost true by definition. You’re saying there is a degree of freedom. Yes, there is.

Christine: First, the abstract types are success across the board. That’s good news. Canonicalization: again yes, up for debate. JSON canonicalization is hard but might work because of existing ECMAScript implementation details. It is speculative but might work. This touches on sorting keys, IEEE floats. We have some assurance with sorting keys. There is a challenge that one of the OCapN points is that tooling is useful across the ecosystem.

Mark: If the canonicalized JSON is still JSON and the tooling for processing SmallCaps is for processing but not producing, or if producing there can be a canonicalization step that the tool can perform, then the OCapN perspective is that canonicalization…

Christine: jq would work as a processing tool. A Python implementer would have to write a new JSON encoder from scratch.

Mark: I assume the Python requirements for IEEE are not exactly the same as JavaScript’s. In which case, the answer is yes.

Christine: It’s a good point about jq. The biggest point is how this impacts implementers. That is the point we want to make.

Richard: I want to push back on development from scratch. Once a JSON canonicalization scheme is agreed upon, that’s just a library support issue. I have a suggestion for JavaScript, Python, and Go. It is just a matter of using a vetted implementation.

Christine: There are no canonical JSON implementations across the languages I use. The selling points is that they can use the tools they already have.

Mark: Clarifying a point regarding canonicalization, we are not currently depending upon it. I'm making an assumption that I want to verify... for two-party communication, most traffic doesn't need it.

Christine: Mostly, but there's a case coming up soon that would benefit for an OCapN-over-OCapN binary relay. In the current implementation, though, because we don't sign messages for any current netlayer, canonicalization is only used for 3PH and start-session.

Mark: Signing and encryption doesn't require canonicalization, it just requires...

Christine: If you don't canonicalize, most implementations store things twice (binary for signing in addition to structured data). Cf. ActivityPub/Matrix/etc.

Kris: Relative work to create a broad ecosystem of canonicalization vs. Syrup-based messages does not strongly favor Syrup.

Christine: There's lots against Syrup later in the presentation; this is just the first substantial slide. ActivityPub was billed as "just JSON", but was actually JSON-LD. People pretty much ignored the latter, and it was de facto overruled. Any implementation built on JSON is likely to suffer the same fate.

Mark: BigInts were and are the key differentiator for us since JSON doesn't support them.

Mark: The extra encoding of Smallcaps comes out more readable than JSON-LD.

Christine: Given that failing to canonicalize requires storing twice, let's look at implementations that do canonicalize. Matrix and Secure Scuttlebutt leveraged ECMA-262 for that.

Christine: JSON doesn't support binary data, and base64 encoding results in exponential growth as it is applied at multiple layers. There was a side table proposal, but that is more complex, harder to sign, and not part of the concrete syntax. Syrup has it for free.

Christine: Textual format. JSON has it mostly for free and developers are familiar with it. Syrup has pretty much nothing.

Christine: Easy access through multiple languages. JSON comes "nearly for free", but Smallcaps is another abstraction layer with its own complexity—especially with binary data and/or canonicalization. Syrup has very few implementations except those made for OCapN, but is easy and fast to implement. However, you do need to make it all from scratch.


Christine: Smallcaps pros and cons: JSON is widespread (but you can't always use your language's JSON), and familiar (but you can't always...)

Christine: Syrup pros and cons: easy to implement, binary for free, ...

Christine: Responding to Syrup review, maps and sets can be made extensions and payloads for OCapN can be restrictive. This is similar to Smallcaps being built on JSON maps but not directly providing them.

Christine: jsyrup is an "third way" exploration that extends JSON:

Mark: Doesn't this import the same canonicalization issues?

Christine: Not if the wire format is still binary.

Christine: As for other approaches, what about other implementers?

Christine: We don't think Smallcaps is sufficient as-is.

Christine: We still have not heard a comparative review about Syrup from Agoric, but we do want that. Let's hear about Syrup + JSyrup: "yes" or "no, and why". And also from other implementers!

Christine: We need a requirements-based approach.

Mark: I want to make sure that canonicalization doesn't become a fly in the ointment. Smallcaps references capabilities by index, and capabilities get translated as messages traverse boundaries. If you can't encode capabilities canonically, is canonicalization actually serving the need?

Christine: This is a good question, but I don't think we have time for it today. Let's capture it in an issue.

Mark: [nods]

Christine: Next steps:

  1. synthesize existing conversation from presentation and dialogue at meeting
  2. Spritely proposes set of requirements based on that converstaion (noting where we have consensus and non-consensus)
  3. Write up "where is canonicalization useful" and "does c-list entries destroy canonicalization use"
  4. (Jessica, Jsyrup) Agoric reviews syrup/molasses for next meeting?
  5. Reach out to implementers and get feedback.

Mark: My concern is not whether or not capabilities are in a side table, but rather the inclusion or exclusion of capabilities from signed payloads (and if they are absent, does that remove the value of signing?).

Mark: In any canonicalization that is meaningful for security, component binary data must not be excluded from the digest. I think the same must be true for capabilities.

Christine: Everyone here is sincere about wanting to do the right thing based on analysis, which puts the group in a good position.

Mark: Cwe just get staed on interoperability experiments using Smallcaps, since you've already demonstrated an ability to convert?

Christine: I think we can get started, but should discuss next meeting "for real".

[meeting adjourns]