Internal Services
Dealer

Dealer: Bidirectional Real-Time Data

Read Pilot first. Pilot proxies websocket connections and establishes gRPC streaming connections to Dealer. The Pilot article is meant to be read before this Dealer article.

Overview

Dealer is a Go web server exposes a single gRPC bidirectional streaming API endpoint. Via this streaming RPC, Pilot can initialize a connection along with metadata that identifies the userId, scope, and permission.

Dealer will then use this information to run a few different processes:

  1. Use the userId as well as other information about the client (provided after the connection is established, by the client) to send information from NATS published by other services relevant to the user (on a best-effort basis – this is core NATS subscription, not Jetstream consumption). This takes place even if the scope is not user_. Therefore, a client connected to some other scope does not also need an active user_ scope connection.

  2. Use the scope to put clients with the same context in the same logical pub/sub space, so that PRESENCE, TYPING_INDICATOR, READ_RECIEPT, and BINARY_BLOCK_UPDATE messages received from clients get sent back down to other clients in the same scope.

  3. Prevent users with READ permission from sending BINARY_BLOCK_UPDATES. That requires WRITE permission.

All scopes are identified as user_123, room_123, block_123, or space_123, however this is immaterial to Dealer. The nature of the scope is arbitrary to Dealer, but it does care about the content of messages from the client. All messages are typed using Protobuf (a single Protobuf message type represents all possible oneof messages that can be sent to Dealer) which requires Pilot to handle all errors that arise when attempting to convert a websocket message from a client to a Protobuf message to Dealer.

API

As described above, Pilot uses the Connect method exposed by Dealer to connect, and provides metadata so Dealer can accept the connection. Missing or invalid metadata leads Dealer to immediately close the stream.

Dealer Connection Lifecycle

  1. Send stream messages to the client for a matching userId in each message received from NATS, for various subjects. As discussed later, Dealer provides very little information in each websocket message.

  2. Handle PRESENCE, READ_RECEIPT, and TYPING_INDICATOR messages from the client by 1) sending them to each other client from that scope, 2) storing the most recent in-memory, and 3) publishing presence/read receipts to NATS Jetstream every 30 seconds.

  3. When a new client connects to a scope, sending it the most recent presence updates from each other client connected to the scope.

  4. When the last client disconnects for a scope, clearing that scope from memory, after sending one final NATS Jetstream presence update.

Outbound Event Types

Dealer notifes clients of the following server-side events:

For most event types, Dealer notifies clients based on their userId, that is to say, the scope does not matter. However, in a few instances, such as BLOCK (see below) the logic Dealer follows is based on the applicable scope, not the userId.

Events are grouped below by the service responsible for publishing events to NATS which Dealer subscribes to, however that information is not actually relevant to clients of Dealer – they don't have to really understand the internal backend service-oriented architecture.

Facebox

USER_FAVORITE

  • user favorites updated for the user
  • Id provided: User
  • Dealer subscribes/unsubscribes dynamically for each connection's userId

USER

  • user record updated for the user
  • Id provided: User
  • Dealer subscribes/unsubscribes dynamically for each connection's userId

Buzzbuzz

NOTIFICATION

  • notifications to the user created or updated
  • Id provided: Notification
  • Dealer subscribes/unsubscribes dynamically for each connection's userId

Blockhead

BLOCK

  • Block created or updated where:
    • block has no parent and spaceId == scope spaceId
    • OR block has a parent, has < 10 ancestors, and spaceId = scope spaceId
    • OR block has a parent and its parent or some ancestor == scope block
  • This event is also triggered when a BlockData, UserBlockData, BlockView, or BlockResponse is created or updated. However, we don't pass the id of that record in the event. It's still just a BLOCK event.
  • Id provided: Block

SPACE_MEMBER

  • space membership created or updated for the user
  • Id provided: SpaceMembership
  • Dealer subscribes/unsubscribes dynamically for each connection's userId

SPACE

  • Space updated where current user has a membership that is active.
  • Id provided: Space

Messenger

ROOM_MEMBERSHIP

  • Room membership created or updated for the user
  • Id provided: RoomMember
  • Dealer subscribes/unsubscribes dynamically for each connection's userId

ROOM

  • Room updated where current user has a membership or where room is in a space where user has a current membership.
  • Id provided: Room

MESSAGE

  • Message sent to in room (including by the current user) where current user has a membership or where room is in a space where user has a current membership.
  • This same event is sent for changes in UserMessageHistory records, not a separate event.
  • Id provided: Message

Stagehand

AV_PRESENCE

  • Dealer subscribes to ephemeral.stagehand.presenceV1 and publishes based on parsing Stagehand's AvRoom names and determining which Pivot room or Pivot block is referenced by that name. Dealer ignores presence messages of other room types.
  • Messages are sent to Dealer clients based on the logic defined above under ROOM and BLOCK.

Schemas

Each message is extremely simple. The type enum value is one of the event types listed above, and the id corresponds to a record that can be queried from the Friend service.

{
  type: enum,
  id: string
}

The exception to this is AV_PRESENCE messages, which are thick, as they include more data about the users in an AvRoom:

{
  AvRoomType: "pivot_room" | "block",
  AvRoomId: string,
  presence: [
    {
      userId: string,
      userName: string,
      joinedAt: string,
      duration: number,
    }
  ]
}

Clients of Pilot should consider that they will receive messages from Dealer for TYPE values they don't understand and ignore those.

Block updates

Dealer consumes the Blockhead NATS subjects for both updated and created blocks and considers multiple aspects of each messange in comparing the message to the scope the client is connected for.

There are two cases to be aware of here:

  • Client is in a spaceId scope, but there is a top level block that isn't shared with them. They will still get Dealer messages on this subscription when such a block is modified, but their Friend RPCs to get the updated block will fail.

  • Client is in a blockId scope, but the actual blockId that's been modified isn't shared with them, even though an ancestor that is the scope blockId is of course shared with them (that's how they were able to join this scope). This type of tree-override in the sharing model is a UX anti-pattern and likely not going to be supported in our UI, but still, we'll still notify such clients that the block has been modified, but their Friend RPCs to get the updated block will fail.

Note that blockhead.change_feed.block.updated doesn't emit a message when richTextContent or other binary fields are edited by a user. It fires when those binaryUpdates are written to the Blockhead database via updates to a Block row, about 5 seconds later. To actually receive in real-time the binary updates, clients exchange them via Dealer.

Client-sent events

Clients can also send their own events up the Pilot websocket connection. These are listed in the Multiplayer article.

NATS

Publication

Dealer publishes presence updates to dealer.presence.userV1, .roomV1, .blockV1, and .spaceV1, depending on the scope prefix. These NATS messages are arrays - each update may have multiple user's represented it, but need not have multiple clients for the same user, Dealer deduplicates that. For all scopes presence updates are published every 30 seconds as well as when the last client for a given scope disconnects.

Dealer also publishes to dealer.read_receipt.createdV1 for read receipts

Consumption

Dealer consumes topics from each service that mutates data which should trigger the event types listed above. The publishing service needs to provide sufficient information in their NATS messages to enable Dealer to determine:

  1. The type of event
  2. The record Id that was created/updated
  3. The userIds who are relevant to the event. This is a concept that multiple services utilize in consuming NATS messages, as discussed here.

Dealer event type BLOCK does not use a userId array, so it is unnecessary for publishers to include that, at least for Dealer's purposes.

For BLOCK, the NATS message must include the blockId that was created/updated, its ancestors[] array, and the spaceId, if the block is in a space.

Databases

N/A

Temporal Workflows

N/A

Deployment

Dealer instances register with AWS Cloud Map on container launch. Dealer application code doesn't know this is happening – it's done by ECS. There is no AWS API dependency in the application code for Dealer or Pilot, but their integration is entirely dependent on Cloud Map.

For reasons of latency and due to the integration with AWS Cloud Map via ECS Service Discovery, Dealer does not have an ECS Service Connect Envoy proxy.

Security

Dealer is an internal service. It is not accessible from the internet, exposes only a gRPC API (therefore Protobuf types), and expects calling services to handle authentication, scope-authorization, and all types of rate limiting. Pilot does all those things, Dealer does none of them.

Notifications LogicFacebox