Code Wolfpack

Decentralizing the Database: Peer-to-Peer E2EE Databases with .NET and SignalR, Part 1

2021-12-24T00:00:00+00:00

Good morning and Merry Christmas, Code Wolfpack! (And a special good morning to the folks reading this from the .NET Advent Calendar!) For those who have followed along with the Code Wolfpack series on Twitch, you’ll know there has been quite a gulf between the broadcasts and the posts on here. This is no coincidence — I have been working very hard on getting things put together for the Decentralizing the Database series, which we’re kicking off today with this special edition post.

Allow me to re-introduce myself

When I first started developing Howler (a chat client similar to Discord, Slack or Matrix) for the Code Wolfpack series, it was really useful to show design patterns which reflected utility in a variety of deployment scenarios, ranging from traditional centralized deployments (Discord/Slack), federated (Matrix) to fully peer-to-peer. We designed data models which lent well to the latter, but had not yet dove into what it would take to build the underlying backing services to facilitate such an approach. As we neared the completion of the centralized and federated versions of Howler, it became clear that E2EE and the peer-to-peer version desired for the platform would be fraught with difficulty in the IaaS context, for a number of reasons:

External Forces - Cloud providers are alarmingly hostile to hosts of services which enable freedom of speech — this has been proven by platforms that have either been coerced by cloud providers to restrict content and/or been removed entirely from those providers, leaving the services scrambling to build out their own data centers.
Trusting Trust - Even if the service itself does not retain data beyond that which is necessary to make the service functional, cloud providers are not beholden to the same standard, and can and do collect data around the use of their own services which could provide at a minimum, metadata which could be used to identify users/deanonymize them when they did not consent to doing so, and further if the issuance of certificates is managed by the same provider, seamlessly insert an alternative one at the behest of a state level actor.
Infectious Design - To provide a decentralized offering alongside the centralized would still require significant build-out distinct from the latter’s present design, and ultimately “infect” the centralized platform to the point the distinction is moot.

This required a lot of soul searching — Howler was initially intended to be for educational purposes, but the pressures in the world made it clear that there was also a tremendous need for a service which can enable safe, secure, private communication amongst groups at the scale of Discord without the serious issues Discord has. Matrix, for all its fanfare finds itself frequently hamstrung by its design causing massive version/feature fragmentation, confusing setup and deployment, and flexibility ad nauseum, not to mention its group encryption (Megolm) not preserving forward secrecy like Signal’s Double-Ratchet protocol provided for the two-party case, and true decentralization still not being supported. Thus, the Howler project pivoted from being strictly an educational approach to systems design and into being a real product which serves an emerging market need — but not to be one to forego its genesis, Howler remains open source, and is developed out in the open on weekly streams (returning after the New Year).

This next section is going to lean very heavy on terms in cryptography. If you want to skip to the more general developer-oriented topic at hand, feel free to jump down to the “Back to .NET” section.

That’s great, it starts with a new SHAKE, cuckoos, stakes and multibranes

Needless to say, when we venture into the realm of multi-party private communication under adversarial conditions, the lexicon rapidly elevates to where it feels more to new and old developers alike as if one is peering into a taxonomic guide to Lovecraftian cosmic entities and finding their eyes turn black and they are suddenly repeating incantations summoning forth Cthulhu. I will try to make this as plain as possible to understand when we dive into these specific topics, but I will also try to keep these things as separated from the Howler service as possible. This first part could not avoid it, as it relates to the magic of how these concepts will impact our design choices.

As I mentioned, it was no coincidence that Howler’s data design was built around the notion of being amenable to being hosted on decentralized platforms — identifiers are deterministic and immutable, message history linkage for reply threading is deniable, and search indexability is verifiably pre-computed at the client level, permitting oblivious accumulator schemes for fast blind retrieval from external hosts. Configuration was purely a singular document store, fully segregated from the user information beyond a unique identifier for retrieval, albeit one using the user id, not that this was mandatory for the design to function. But there are deficits in the current state of the art in cryptography and decentralized trustless systems to provide what Howler needs. Let’s break it down, because each item is a lot to unpack:

Fast, secure multi-party communication with perfect forward secrecy

Signal’s Double-Ratchet Protocol answers the two-party case, but the current solution for multi-party configurations has severe tradeoffs that compromise most of the security benefits Double-Ratchet provides. These tradeoffs include a loss of future and perfect forward secrecy. Not to throw up my arms and reluctantly choose either an expensive multi-party X3DH ratchet (i.e. n-1 rounds of messages to create a new ratchet, where n is the number of users in a group) or the non-starter of Megolm’s tradeoffs, I have created a new protocol, leveraging the recent discoveries in trustless distributed key generation schemes with an adversarial majority as a precursor ratchet that provides keys for the Double-Ratchet Protocol, generalizing to what I am calling the Triple-Ratchet Protocol (publication to follow: Secure Multi-party Communication with Perfect Forward Secrecy). To support WebCrypto interfaces in the browser and greater hardware enclave compatibility, I have amended the protocol to use general elliptic curve cryptographic primitives: ECDSA, and ECDH.

Decentralized trustless private hypergraph database with oblivious queryability

The other downside to Signal and Matrix is the aforementioned metadata problem. Some approaches try to simply leverage indirection by embedding their service in Tor as a hidden service, despite Tor being effectively compromised. Message retention is also predicated on availability of ratchet state, and group size changes require transfering the ratchet state on addition for historic access to messages. Message ordering in the Megolm case is also not guaranteed, with a fractured and somewhat mutable chat state. In the Triple-Ratchet approach, ordering is preserved, but the ratchet state transference issue is still retained, and thus we must introduce a protocol for message storage which enables oblivious historic retrieval and queryability of message history without a loss to future and perfect forward secrecy. In upcoming parts of this series, I will present concept by concept the techniques which enable this, ultimately creating a CQL-compatible (per-clique) version of this database.

Decentralized Identity

As we discussed in our session on QR Authentication, key exchange itself has an adversarial component that requires a separate channel for authentication. We solved for this in the QR Code case by presenting a post-exchange value derived from the agreed key the user must confirm matches on both devices. Because the user is on both sides of this exchange, they are themselves the separate channel, via their eyes (not to mention this being predicated on trusting the software or hardware itself is not compromised, but there are significantly greater problems at hand if this is the case). Thankfully, due to the emerging world of cryptocurrencies, self-governed identities have already seen a lot of exploration, and so we will simply integrate with a number of cryptocurrencies’ common wallet APIs (starting with Ethereum/EVM-compatible and Solana) to provide this DID infrastructure.

Back to .NET

So that was a lot to take in (and it doesn’t even include all the nitty gritty details!), but with the protocol side of things squared away, we can focus on the crux of the article: Enabling the peer-to-peer communication required to support this approach with SignalR.

What is SignalR?

In short, it is a library that makes it very simple to provide real-time communication between a client and server. By adding SignalR middleware to your ASP.NET Core application, you can support websockets (and very graceful fallbacks to other transport mechanisms) with the Hub metaphor as intuitively as you build APIs with the built-in Controller/Action semantics for RESTful interfaces. Not to leave the transport negotiation as an implementation detail on the client side, there is also an accompanying library for Javascript/Typescript. But it doesn’t stop there — consider the trivial example of a basic unauthenticated non-durable chat app: no channels, nothing saved, just a user id and text for each message for those around to receive it. Once you reach the limit of a single server, the obvious answer is to scale horizontally, which would then lend towards thinking about some kind of storage to share between servers, even if just using Redis, or maybe each server will connect to each other and shuttle messages back and forth. An experienced engineer may notice this is starting to sound like a message bus. The good news is SignalR happens to support the pattern outright, with the HubLiftetimeManager abstract class forming the foundation on which a variety of bus implementations may be used. Built into SignalR, you can use Redis and Azure Service Bus as the backplane. This makes it extremely easy to scale out your messaging infrastructure so that if in this contrived example, one client connects to server A, and another to server B, both parties will be assured to see all messages from each server.

Practically Speaking

So how do you build a Hub? Extend from the Hub class, and you will have access to the protected properties which streamline the pub/sub mechanics:

public class ExampleHub : Hub
{
    private ILogger<ExampleHub> _logger;

    public ExampleHub(ILogger<ExampleHub> logger)
    {
        this._logger = logger;
    }

    public async Task JoinChannel(string channelName)
    {
        await this.Groups.AddToGroupAsync(
            this.Context.ConnectionId,
            channelName);

        await this.Clients.Group(channelName)
            .SendAsync("UserJoin", channelName, this.Context.ConnectionId);

        await this.Clients.Caller.SendAsync(
            "JoinChannelSuccess",
            channelName);
    }

    public async Task SendMessage(string channelName, string message)
    {
        await this.Clients.Group(channelName).SendAsync(
            "ReceiveMessage",
            channelName,
            this.Context.ConnectionId,
            message);
    }

    public async Task LeaveChannel(string channelName)
    {
        await this.Groups.RemoveFromGroupAsync(
            this.Context.ConnectionId,
            channelName);

        await this.Clients.Group(channelName)
            .SendAsync("UserLeave", channelName, this.Context.ConnectionId);
            
        await this.Clients.Caller.SendAsync(
            "LeaveChannelSuccess",
            channelName);
    }
}

Note that this simple example actually goes one step farther than the contrived example, adding channels as a mode of separation (obviously without checking channel membership on sends, but joining is unauthenticated anyway). Registering the Hub to be available in Startup.cs is also straightforward:

    // Amend your existing UseEndpoints to include your hub:
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
        endpoints.MapHub<ExampleHub>("/example");
    });

On the client side, add @microsoft/signalr to your package.json, and on import, connecting and handling messages/sending is this easy:

let connection = new signalR.HubConnectionBuilder()
    .withAutomaticReconnect()
    .withUrl(serverUrl + "/example").build();
connection.start();

connection.on("JoinChannelSuccess", (channelName) => {
    console.log("joined " + channelName);
});
connection.on("LeaveChannelSuccess", (channelName) => {
    console.log("left " + channelName);
});
connection.on("UserJoin", (channelName, userId) => {
    console.log("[" + channelName + "] @" + userId + " joined");
});
connection.on("UserLeave", (channelName, userId) => {
    console.log("[" + channelName + "] @" + userId + " left");
});
connection.on("ReceiveMessage", (channelName, userId, message) => {
    console.log("[" + channelName + "] @" + userId + ": " + message);
});

function sendMessage(channelName, message) {
    connection.send("SendMessage", channelName, message);
}

with that little snippet, you are immediately connected to the hub, handle all messages, and can send messages. Reconnection has sensible defaults in place (this is configurable), and authorization is pretty easy to wire up. Beautiful.

Howler and SignalR

In the federated version of Howler, we had to maintain connectivity to each logical server a user wants to participate in. Let’s revisit how that is wired up. On the UI Client side, we track state of individual federated core services connections as a map of server IDs to connection state with Redux:

export interface ServersState {
    authToken: string;
    servers: { [serverId: string]: ConnectionInfoState };
    lastRefreshedServerId: string | null;
    isLoading: boolean;
}

export interface ConnectionInfoState {
    isConnecting: boolean;
    connection?: signalR.HubConnection;
    token: string;
    userSpaces: { [userId: string]: string[] };
    error?: string;
}

And these connections are maintained via Sagas:

function* handleServerTokenRequest(request: RequestServerTokenAction) {
    let state: ServersState = yield select(s => s.servers as ServersState);
    
    if (
        state.servers[request.serverId] == null ||
        state.servers[request.serverId].token == null
    ) {
        yield put({
            type: 'REQUESTING_SERVER_TOKEN',
            serverId: request.serverId
        });
        let token: string = yield call(
            getServerToken(request.serverId, state.authToken)
        );
        yield put({
            type: 'RECEIVE_SERVER_TOKEN',
            serverId: request.serverId,
            token
        });
    }
}

function* handleConnectionRequest(request: RequestConnectionAction) {
    yield put({type: 'REQUESTING_CONNECTION', serverId: request.serverId});
    yield handleServerTokenRequest({
        type: 'REQUEST_SERVER_TOKEN',
        serverId: request.serverId
    });

    let response: {
        connection: signalR.HubConnection,
        error: any
    } = yield callServer(request.serverId, connect);
    
    if (response.error != null) {
        yield put({
            type: 'FAILED_CONNECTION',
            serverId: request.serverId,
            error: response.error
        });
    } else {
        yield put({
            type: 'RECEIVE_CONNECTION',
            serverId: request.serverId,
            connection: response.connection
        });
    }
}


export const serversSagas = {
    watchServerTokenRequests: function*() {
        yield takeLeading('REQUEST_CONNECTION', handleConnectionRequest)
        yield takeLeading('REQUEST_SERVER_TOKEN', handleServerTokenRequest)
    }
};

For a refresher course on Redux-Saga, take a look at our intro session where we discuss why and how you can integrate Redux-Saga into your React application.

Note the maintenance of server tokens and server connections involved (as there are also supplementary API calls that can server larger amounts of data outside of the websocket). As you could imagine, a user connected to many distinct servers (NB: In Howler, servers truly mean servers in a logical sense, whereas the Discord parlance of “server” is referred to as a Space.) would be fraught with many simultaneous websockets. This is less than ideal, but a necessary consequence of a federated design. By shifting to purely a peer-to-peer approach, our client does not need to keep a map of server connections at all, as it only has to maintain a single connection to the local SignalR server. We can remove all the connection/token state maintenance logic from the reducer, and just leave it as an API surface, closer to the ExampleHub, which is a nice simplification.

On that note of refactoring

Consider the historical architecture of centralized Howler:

In decentralized architectures, the notion of a server is somewhat nebulous, as every peer is a server, but the logical “server” is the collective operations of each node, with their own replicated state localized for whatever interaction is required. For Howler to move into a peer-to-peer approach, we can make some adjustments to the diagram, like so:

That looks a bit more complicated! Let’s break it down. Short of the DID integration needs for the UI, most of this is unchanged. So let’s focus strictly on the server layer:

Quite a bit more manageable. There’s some new stuff! We’ll elide over the addition of the Keys hub, that will be relevant in Part 2 where we review the Triple-Ratchet Protocol implementation. That leaves us with three boxes: the SignalR Backplane, the QCQL Program ABI, and the Quilibrium Node.

You mentioned the backplane already, but what is Quilibrium? And did you bash your keyboard in the middle?

Quilibrium is a new general purpose peer to peer platform as a service. I’ll save the general PR statement for another post, but that is the key piece of “magic” that makes this approach possible. In particular, we will leverage the Quilibrium node library to provide our aforementioned hypergraph database, no different than if you were to launch an AWS Keyspaces database, except in this scenario, each node in the network is a part of the fabric that makes the database application code run. We will provide one additional enhancement, which is the application binary interface, or ABI, to present our CQL compatibility layer, allowing us to substitute this in place of our Keyspaces servers. But more about that in another post — today, we are going to treat this no differently than if we were using our Cassandra docker container like before. That’s where the backplane comes into play.

SignalR Backplanes

The built in backplanes are fantastic for deploying alongside Redis or deeply integrating with Azure, however CQL is not supported out of the box. Rather than building out a Redis compatibility ABI alongside a CQL compatibility ABI and just using the built in Redis backplane, we can simply build towards a fully-integrated future with our overall data model. As a benefit, the greater participation in the overall network will further reduce potentially fingerprintable traffic thus enhancing privacy. All done with one very simple abstract class to implement (doc comments omitted):

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

namespace Microsoft.AspNetCore.SignalR;

/// <summary>
/// A lifetime manager abstraction for <see cref="Hub"/> instances.
/// </summary>
public abstract class HubLifetimeManager<THub> where THub : Hub
{
    // Called by the framework and not something we'd cancel, so it doesn't
    // take a cancellation token
    public abstract Task OnConnectedAsync(HubConnectionContext connection);

    // Called by the framework and not something we'd cancel, so it doesn't
    // take a cancellation token
    public abstract Task OnDisconnectedAsync(HubConnectionContext connection);

    public abstract Task SendAllAsync(
        string methodName,
        object?[] args,
        CancellationToken cancellationToken = default);

    public abstract Task SendAllExceptAsync(
        string methodName,
        object?[] args,
        IReadOnlyList<string> excludedConnectionIds,
        CancellationToken cancellationToken = default);

    public abstract Task SendConnectionAsync(
        string connectionId,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendConnectionsAsync(
        IReadOnlyList<string> connectionIds,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupAsync(
        string groupName,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupsAsync(
        IReadOnlyList<string> groupNames,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupExceptAsync(
        string groupName,
         string methodName,
         object?[] args,
         IReadOnlyList<string> excludedConnectionIds,
         CancellationToken cancellationToken = default);

    public abstract Task SendUserAsync(
        string userId,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendUsersAsync(
        IReadOnlyList<string> userIds,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task AddToGroupAsync(
        string connectionId,
         string groupName,
         CancellationToken cancellationToken = default);

    public abstract Task RemoveFromGroupAsync(
        string connectionId,
         string groupName,
         CancellationToken cancellationToken = default);
}

Of course, there’s a lot of specialized management to think about for implementing this interface well in a pattern that can scale out well. While we won’t be diving into the nuances of how our QCQL application works in this post, the detail worth noting is that a keyspace is created for every Space, and so in our case, a SignalR backplane will be need to be addressible on a per-Space basis. With that in mind, what will we do instead?

Backplane configurability will need to be contingent on the active keyspaces to scan, one stream per keyspace
Message composite keys will be bucketed into 100 millisecond partition keys, using now()

To make our example a little easier to grok, we will omit the multi-Space consideration, and opt for only one stream, one keyspace. How does this play out in the schema?

CREATE TABLE "howler_messagebus"."message"(
	"message_bucket_id" BIGINT,
    "message_timestamp" TIMESTAMP,
	"message_id" ASCII,
    "payload" BLOB,
	PRIMARY KEY("message_bucket_id", "message_timestamp"))

By concentrating our resolution to 100ms intervals, we phase our lookbacks on retrieval to small enough increments where when on a high speed connection, messages are imperceptibly instant. If of course connectivity is hindered or slow, messages will arrive when they arrive. To avoid a poor user experience, we bake in the expectation in our use of SignalR that deliverability on the backplane is not guaranteed (although invariably it would arrive), and appropriate retrievals from the channel messages table will need to occur periodically to shore up inconsistencies.

So let’s get into the CQL version of the HubLifetimeManager constructor:

    /// <summary>
    /// Initializes a new instance of the
    /// <see cref="CqlKeyspaceHubLifetimeManager{THub}"/> class.
    /// </summary>
    /// <param name="logger">
    /// The injected instance of the logger.
    /// </param>
    /// <param name="options">
    /// The <see cref="CqlOptions"/> that configures the Cassandra Session.
    /// </param>
    /// <param name="hubProtocolResolver">
    /// The injected instance of <see cref="IHubProtocolResolver"/>, gets an
    /// <see cref="IHubProtocol"/> instance to write to connections.
    /// </param>
    /// <param name="globalHubOptions">
    /// The injected global <see cref="HubOptions"/>.
    /// </param>
    /// <param name="hubOptions">
    /// The injected <typeparamref name="THub"/> specific options.
    /// </param>
    public CqlKeyspaceHubLifetimeManager(
        ILogger<CqlKeyspaceHubLifetimeManager<THub>> logger,
        IOptions<CqlOptions> options,
        IHubProtocolResolver hubProtocolResolver,
        IOptions<HubOptions>? globalHubOptions,
        IOptions<HubOptions<THub>>? hubOptions)
    {
        this._logger = logger;
        this._options = options.Value;
        
        if (globalHubOptions != null && hubOptions != null)
        {
            this._handler = new CqlMessageHandler(
                new DefaultHubMessageSerializer(
                    hubProtocolResolver,
                    globalHubOptions.Value.SupportedProtocols,
                    hubOptions.Value.SupportedProtocols));
        }
        else
        {
            var supportedProtocols = hubProtocolResolver.AllProtocols
                .Select(p => p.Name).ToList();
            this._handler = new CqlMessageHandler(
                new DefaultHubMessageSerializer(
                    hubProtocolResolver,
                    supportedProtocols,
                    null));
        }

        this._session = this._options.CreateSession();
        this.StartSubscriberAsync(this._options.Interval, this._options.Tolerance);
    }

You can see a unique CqlMessageHandler class, it is basically a CQL select/insert/delete statement generator. The DefaultHubMessageSerializer is an internal of SignalR, serializing all message formats supported by the Hub back to the client. StartSubscriberAsync? We’ll get to that momentarily. The rest of this is going to get very boilerplate following a pattern, so let’s hone in on a couple of examples:

/// <inheritdoc />
public override Task AddToGroupAsync(
    string connectionId,
    string groupName,
    CancellationToken cancellationToken = default)
{
    if (connectionId == null)
    {
        throw new ArgumentNullException(nameof(connectionId));
    }

    if (groupName == null)
    {
        throw new ArgumentNullException(nameof(groupName));
    }

    var connection = this._connections[connectionId];
    if (connection != null)
    {
        // short circuit if connection is on this server
        return this.AddGroupAsyncCore(connection, groupName);
    }

    return this.SendGroupActionAsync(
        connectionId,
        groupName,
        "add");
}

private async Task SendGroupActionAsync(
    string connectionId,
    string groupName,
    string action)
{
    var message = this._handler.WriteGroupCommand(
        this._serverName,
        action,
        groupName,
        connectionId);
    await this.PublishAsync(message);
}

RemoveFromGroupAsync, and the User-level commands all look very similar to this. What about message sending?

/// <inheritdoc />
public override Task SendGroupsAsync(
    IReadOnlyList<string> groupNames,
    string methodName,
    object?[] args,
    CancellationToken cancellationToken = default)
{
    if (groupNames == null)
    {
        throw new ArgumentNullException(nameof(groupNames));
    }

    var publishTasks = new List<Task>(groupNames.Count);

    foreach (var groupName in groupNames)
    {
        if (!string.IsNullOrEmpty(groupName))
        {
            var payload = this._handler.WriteMethodInvocation(
                groupName,
                methodName,
                args);
            publishTasks.Add(this.PublishAsync(payload));
        }
    }

    return Task.WhenAll(publishTasks);
}

private async Task PublishAsync(IStatement payload)
{
    this._session.Execute(payload)
}

Couldn’t be easier. What about receiving? Remember that StartSubscriberAsync?

public async Task StartSubscriberAsync(TimeSpan interval, int tolerance, CancellationToken cancellationToken)
{
    var tasks = new List<Task>(groupNames.Count);

    while (true)
    {
        try
        {
            var payload = await this._handler.ReadIntervalAsync(interval, Math.Min(tolerance, skippedBeats));
            var results = await this._session.ExecuteAsync(payload);

            foreach (var result in results)
            {
                tasks.Add(this.ProcessMessageAsync(result));
            }

            await Task.Delay(interval, cancellationToken);
        }
        catch (CqlException e)
        {
            this._logger.LogError(e, "An exception was encountered when reading");
            this._skippedBeats += 1;
        }
    }
}

Where ProcessMessageAsync parses through the various results and calls the appropriate routing methods. So that leaves one last part - if this is all over a decentralized CQL compatible database, how do we ensure only the messages the user is submitting are actually ones they are supposed to and not on behalf of someone else? And that, is where we start our next article: The Triple-Ratchet Algorithm. Merry Christmas!

Decentralizing the Database: Peer-to-Peer E2EE Databases with .NET and SignalR, Part 1

2021-12-19T00:00:00+00:00

Allow me to re-introduce myself

External Forces - Cloud providers are alarmingly hostile to hosts of services which enable freedom of speech — this has been proven by platforms that have either been coerced by cloud providers to restrict content and/or been removed entirely from those providers, leaving the services scrambling to build out their own data centers.
Trusting Trust - Even if the service itself does not retain data beyond that which is necessary to make the service functional, cloud providers are not beholden to the same standard, and can and do collect data around the use of their own services which could provide at a minimum, metadata which could be used to identify users/deanonymize them when they did not consent to doing so, and further if the issuance of certificates is managed by the same provider, seamlessly insert an alternative one at the behest of a state level actor.
Infectious Design - To provide a decentralized offering alongside the centralized would still require significant build-out distinct from the latter’s present design, and ultimately “infect” the centralized platform to the point the distinction is moot.

That’s great, it starts with a new SHAKE, cuckoos, stakes and multibranes

Fast, secure multi-party communication with perfect forward secrecy

Decentralized trustless private hypergraph database with oblivious queryability

Decentralized Identity

Back to .NET

What is SignalR?

Practically Speaking

So how do you build a Hub? Extend from the Hub class, and you will have access to the protected properties which streamline the pub/sub mechanics:

public class ExampleHub : Hub
{
    private ILogger<ExampleHub> _logger;

    public ExampleHub(ILogger<ExampleHub> logger)
    {
        this._logger = logger;
    }

    public async Task JoinChannel(string channelName)
    {
        await this.Groups.AddToGroupAsync(
            this.Context.ConnectionId,
            channelName);

        await this.Clients.Group(channelName)
            .SendAsync("UserJoin", channelName, this.Context.ConnectionId);

        await this.Clients.Caller.SendAsync(
            "JoinChannelSuccess",
            channelName);
    }

    public async Task SendMessage(string channelName, string message)
    {
        await this.Clients.Group(channelName).SendAsync(
            "ReceiveMessage",
            channelName,
            this.Context.ConnectionId,
            message);
    }

    public async Task LeaveChannel(string channelName)
    {
        await this.Groups.RemoveFromGroupAsync(
            this.Context.ConnectionId,
            channelName);

        await this.Clients.Group(channelName)
            .SendAsync("UserLeave", channelName, this.Context.ConnectionId);
            
        await this.Clients.Caller.SendAsync(
            "LeaveChannelSuccess",
            channelName);
    }
}

    // Amend your existing UseEndpoints to include your hub:
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
        endpoints.MapHub<ExampleHub>("/example");
    });

On the client side, add @microsoft/signalr to your package.json, and on import, connecting and handling messages/sending is this easy:

let connection = new signalR.HubConnectionBuilder()
    .withAutomaticReconnect()
    .withUrl(serverUrl + "/example").build();
connection.start();

connection.on("JoinChannelSuccess", (channelName) => {
    console.log("joined " + channelName);
});
connection.on("LeaveChannelSuccess", (channelName) => {
    console.log("left " + channelName);
});
connection.on("UserJoin", (channelName, userId) => {
    console.log("[" + channelName + "] @" + userId + " joined");
});
connection.on("UserLeave", (channelName, userId) => {
    console.log("[" + channelName + "] @" + userId + " left");
});
connection.on("ReceiveMessage", (channelName, userId, message) => {
    console.log("[" + channelName + "] @" + userId + ": " + message);
});

function sendMessage(channelName, message) {
    connection.send("SendMessage", channelName, message);
}

Howler and SignalR

export interface ServersState {
    authToken: string;
    servers: { [serverId: string]: ConnectionInfoState };
    lastRefreshedServerId: string | null;
    isLoading: boolean;
}

export interface ConnectionInfoState {
    isConnecting: boolean;
    connection?: signalR.HubConnection;
    token: string;
    userSpaces: { [userId: string]: string[] };
    error?: string;
}

And these connections are maintained via Sagas:

function* handleServerTokenRequest(request: RequestServerTokenAction) {
    let state: ServersState = yield select(s => s.servers as ServersState);
    
    if (
        state.servers[request.serverId] == null ||
        state.servers[request.serverId].token == null
    ) {
        yield put({
            type: 'REQUESTING_SERVER_TOKEN',
            serverId: request.serverId
        });
        let token: string = yield call(
            getServerToken(request.serverId, state.authToken)
        );
        yield put({
            type: 'RECEIVE_SERVER_TOKEN',
            serverId: request.serverId,
            token
        });
    }
}

function* handleConnectionRequest(request: RequestConnectionAction) {
    yield put({type: 'REQUESTING_CONNECTION', serverId: request.serverId});
    yield handleServerTokenRequest({
        type: 'REQUEST_SERVER_TOKEN',
        serverId: request.serverId
    });

    let response: {
        connection: signalR.HubConnection,
        error: any
    } = yield callServer(request.serverId, connect);
    
    if (response.error != null) {
        yield put({
            type: 'FAILED_CONNECTION',
            serverId: request.serverId,
            error: response.error
        });
    } else {
        yield put({
            type: 'RECEIVE_CONNECTION',
            serverId: request.serverId,
            connection: response.connection
        });
    }
}


export const serversSagas = {
    watchServerTokenRequests: function*() {
        yield takeLeading('REQUEST_CONNECTION', handleConnectionRequest)
        yield takeLeading('REQUEST_SERVER_TOKEN', handleServerTokenRequest)
    }
};

For a refresher course on Redux-Saga, take a look at our intro session where we discuss why and how you can integrate Redux-Saga into your React application.

On that note of refactoring

Consider the historical architecture of centralized Howler:

That looks a bit more complicated! Let’s break it down. Short of the DID integration needs for the UI, most of this is unchanged. So let’s focus strictly on the server layer:

You mentioned the backplane already, but what is Quilibrium? And did you bash your keyboard in the middle?

SignalR Backplanes

// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

namespace Microsoft.AspNetCore.SignalR;

/// <summary>
/// A lifetime manager abstraction for <see cref="Hub"/> instances.
/// </summary>
public abstract class HubLifetimeManager<THub> where THub : Hub
{
    // Called by the framework and not something we'd cancel, so it doesn't
    // take a cancellation token
    public abstract Task OnConnectedAsync(HubConnectionContext connection);

    // Called by the framework and not something we'd cancel, so it doesn't
    // take a cancellation token
    public abstract Task OnDisconnectedAsync(HubConnectionContext connection);

    public abstract Task SendAllAsync(
        string methodName,
        object?[] args,
        CancellationToken cancellationToken = default);

    public abstract Task SendAllExceptAsync(
        string methodName,
        object?[] args,
        IReadOnlyList<string> excludedConnectionIds,
        CancellationToken cancellationToken = default);

    public abstract Task SendConnectionAsync(
        string connectionId,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendConnectionsAsync(
        IReadOnlyList<string> connectionIds,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupAsync(
        string groupName,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupsAsync(
        IReadOnlyList<string> groupNames,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendGroupExceptAsync(
        string groupName,
         string methodName,
         object?[] args,
         IReadOnlyList<string> excludedConnectionIds,
         CancellationToken cancellationToken = default);

    public abstract Task SendUserAsync(
        string userId,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task SendUsersAsync(
        IReadOnlyList<string> userIds,
         string methodName,
         object?[] args,
         CancellationToken cancellationToken = default);

    public abstract Task AddToGroupAsync(
        string connectionId,
         string groupName,
         CancellationToken cancellationToken = default);

    public abstract Task RemoveFromGroupAsync(
        string connectionId,
         string groupName,
         CancellationToken cancellationToken = default);
}

Backplane configurability will need to be contingent on the active keyspaces to scan, one stream per keyspace
Message composite keys will be bucketed into 100 millisecond partition keys, using now()

To make our example a little easier to grok, we will omit the multi-Space consideration, and opt for only one stream, one keyspace. How does this play out in the schema?

CREATE TABLE "howler_messagebus"."message"(
	"message_bucket_id" BIGINT,
    "message_timestamp" TIMESTAMP,
	"message_id" ASCII,
    "payload" BLOB,
	PRIMARY KEY("message_bucket_id", "message_timestamp"))

So let’s get into the CQL version of the HubLifetimeManager constructor:

    /// <summary>
    /// Initializes a new instance of the
    /// <see cref="CqlKeyspaceHubLifetimeManager{THub}"/> class.
    /// </summary>
    /// <param name="logger">
    /// The injected instance of the logger.
    /// </param>
    /// <param name="options">
    /// The <see cref="CqlOptions"/> that configures the Cassandra Session.
    /// </param>
    /// <param name="hubProtocolResolver">
    /// The injected instance of <see cref="IHubProtocolResolver"/>, gets an
    /// <see cref="IHubProtocol"/> instance to write to connections.
    /// </param>
    /// <param name="globalHubOptions">
    /// The injected global <see cref="HubOptions"/>.
    /// </param>
    /// <param name="hubOptions">
    /// The injected <typeparamref name="THub"/> specific options.
    /// </param>
    public CqlKeyspaceHubLifetimeManager(
        ILogger<CqlKeyspaceHubLifetimeManager<THub>> logger,
        IOptions<CqlOptions> options,
        IHubProtocolResolver hubProtocolResolver,
        IOptions<HubOptions>? globalHubOptions,
        IOptions<HubOptions<THub>>? hubOptions)
    {
        this._logger = logger;
        this._options = options.Value;
        
        if (globalHubOptions != null && hubOptions != null)
        {
            this._handler = new CqlMessageHandler(
                new DefaultHubMessageSerializer(
                    hubProtocolResolver,
                    globalHubOptions.Value.SupportedProtocols,
                    hubOptions.Value.SupportedProtocols));
        }
        else
        {
            var supportedProtocols = hubProtocolResolver.AllProtocols
                .Select(p => p.Name).ToList();
            this._handler = new CqlMessageHandler(
                new DefaultHubMessageSerializer(
                    hubProtocolResolver,
                    supportedProtocols,
                    null));
        }

        this._session = this._options.CreateSession();
        this.StartSubscriberAsync(this._options.Interval, this._options.Tolerance);
    }

/// <inheritdoc />
public override Task AddToGroupAsync(
    string connectionId,
    string groupName,
    CancellationToken cancellationToken = default)
{
    if (connectionId == null)
    {
        throw new ArgumentNullException(nameof(connectionId));
    }

    if (groupName == null)
    {
        throw new ArgumentNullException(nameof(groupName));
    }

    var connection = this._connections[connectionId];
    if (connection != null)
    {
        // short circuit if connection is on this server
        return this.AddGroupAsyncCore(connection, groupName);
    }

    return this.SendGroupActionAsync(
        connectionId,
        groupName,
        "add");
}

private async Task SendGroupActionAsync(
    string connectionId,
    string groupName,
    string action)
{
    var message = this._handler.WriteGroupCommand(
        this._serverName,
        action,
        groupName,
        connectionId);
    await this.PublishAsync(message);
}

RemoveFromGroupAsync, and the User-level commands all look very similar to this. What about message sending?

/// <inheritdoc />
public override Task SendGroupsAsync(
    IReadOnlyList<string> groupNames,
    string methodName,
    object?[] args,
    CancellationToken cancellationToken = default)
{
    if (groupNames == null)
    {
        throw new ArgumentNullException(nameof(groupNames));
    }

    var publishTasks = new List<Task>(groupNames.Count);

    foreach (var groupName in groupNames)
    {
        if (!string.IsNullOrEmpty(groupName))
        {
            var payload = this._handler.WriteMethodInvocation(
                groupName,
                methodName,
                args);
            publishTasks.Add(this.PublishAsync(payload));
        }
    }

    return Task.WhenAll(publishTasks);
}

private async Task PublishAsync(IStatement payload)
{
    this._session.Execute(payload)
}

Couldn’t be easier. What about receiving? Remember that StartSubscriberAsync?

public async Task StartSubscriberAsync(TimeSpan interval, int tolerance, CancellationToken cancellationToken)
{
    var tasks = new List<Task>(groupNames.Count);

    while (true)
    {
        try
        {
            var payload = await this._handler.ReadIntervalAsync(interval, Math.Min(tolerance, skippedBeats));
            var results = await this._session.ExecuteAsync(payload);

            foreach (var result in results)
            {
                tasks.Add(this.ProcessMessageAsync(result));
            }

            await Task.Delay(interval, cancellationToken);
        }
        catch (CqlException e)
        {
            this._logger.LogError(e, "An exception was encountered when reading");
            this._skippedBeats += 1;
        }
    }
}

Fan-Out Authorization with an Authorization Gateway

2021-06-26T00:00:00+00:00

Hi Wolfpack! I hope you’re all staying cool during this absolutely mind-boggling summer weekend. This post serves as a recap for the past two sessions, both tracking the story arc of providing fan-out authorization by building our Authorization Gateway.

Fan-Out What?

Even for seasoned developers, fan-out authorization flows are extremely uncommon, mostly because the scopes of an authorized user’s token is usually sufficient to derive access, and if differing tokens between multiple services is needed, a few roundtrips with the authorization server is sufficient to check consents (or prompt for them) and issue relevant tokens. This hits a few limitations when you have specific scenarios that an auth server may not allow:

Dynamic, or arbitrarily large numbers of scopes (Cognito, for example, is limited to 50 scopes per client app, 100 per resource server, and dynamic generation is effectively impossible)
Dynamic audience targeting within same client app, oftentimes is 1:1 to the client id
As an addendum to the previous scenario: differing redirect uris equally requires client id separation

At this point, many would reach for an all-in-one solution like IdentityServer4 and implement authentication, authorization, and federated-server-specific authorization flows all under a series of client apps and scopes. IdentityServer4 is a great solution, but not without its own costs: notably, it does not work with any reasonable degree of performance when hosted as a Lambda (or App Function) due to its long cold start times, and addressing aggressive scale up/down scenarios with ECS and ASGs gets expensive, quickly. Cognito, on the other hand, keeps its costs associated to very simple metrics: Monthly Active Users (MAUs), and the first 50,000 users are free, and an intermediary lightweight web service running on Lambda is equally quite cheap and scalable. Really hard to argue with that.

All that aside, for most authorization scenarios, these limits and flow restrictions are not only reasonable, they are fundamental to ensuring security — allowing arbitrary redirect uris for a client is absolutely a vulnerability, unless there are mutually authenticated flows that establish those redirect uris, and the token is appropriately delegated, such as via audience, which brings us into fan-out authorization.

Howler has a unique problem to solve in allowing simple server federation without federated server owners having to set up their own auth server, and also in making the authentication flow seamless for the end users: Join a space on a server – any federated server using our auth services, and the client will negotiate authentication with the gateway service. Logically, the flow looks like this:

From A to AuthZ

This section assumes familiarity with the structure of JSON Web Tokens. An earlier session went over this, and when it is re-added to this site, this note will link to it. In the meantime, the tutorial at JWT.io is a great overview.

From the client-facing side of things, we need only implement a single endpoint. Consume an valid, authorized (via the Cognito auth service) user’s token with Gateway scope, replace the issuer with the gateway service, the audience with the id of the intended federated server, and the scope with the same identifier, adjusting issue/expiration time accordingly, and sign with the gateway service’s private key.

[HttpPost("auth")]
[ProducesResponseType(typeof(string), 200)]
[ProducesResponseType(404)]
public IActionResult Post(string serverId)
{
    var tokens = this.HttpContext.Request.Headers["Authorization"];

    if (tokens.Count != 1)
    {
        // Attempt at pushing multiple tokens, kick em out.
        return this.Forbid();
    }

    var token = tokens.First().Split(' ').Last();
    var jwt = new Microsoft.IdentityModel.JsonWebTokens
        .JsonWebToken(token);
    var validatedServerId = this._federatedDb.Servers
        .Where(s => s.ServerId == serverId)
        .Select(s => s.ServerId)
        .ToList()
        .FirstOrDefault();

    if (validatedServerId == null)
    {
        return this.NotFound();
    }

    // TODO: Whitelist/blacklist

    return this.Ok(
        new GatewayJWT(
            new GatewayJWTHeader(jwt.Kid),
            new GatewayJWTBody(
                jwt.Subject,
                jwt.GetClaim("device_key").Value,
                jwt.GetClaim("event_id").Value,
                serverId,
                long.Parse(jwt.GetClaim("auth_time").Value),
                "https://gateway.howler.chat",
                DateTimeOffset.UtcNow.AddHours(2).ToUnixTimeSeconds(),
                jwt.GetClaim("jti").Value,
                jwt.GetClaim("client_id").Value,
                jwt.GetClaim("username").Value),
            this._signingAlgorithm).ToString());
}

As you can see, it’s pretty straightforward – we abstract away the signing algorithm and structural interpretation of the JWT to relevant classes, check that the server is in our roster (and as the TODO notes, the user or server is not on our blacklist or bypass if the user is on the whitelist). Ultimately this gives us a perfectly-valid access token that a federated server should be configured to accept:

// ... in the Startup.cs ConfigureServices method:
services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(x =>
    {
        x.Authority = "https://gateway.howler.chat";
        x.Audience = "id-of-federated-server";
        x.RequireHttpsMetadata = true;
        x.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateAudience = true,
            ValidateIssuer = true,
            ValidateIssuerSigningKey = true,
        };
    });

Can it really be that easy? There’s a little bit of magic being glossed over here: OpenID Discovery.

Self-discovery of Auth Servers

ASP.NET authorization support lets you specify an authority, and the service will request the well-known configuration info, which is where some blatant standards-breaking is happening:

GET https://gateway.howler.chat/.well-known/openid-configuration

{
    "authorization_endpoint": "https://auth.howler.chat/oauth2/authorize",
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "issuer": "https://gateway.howler.chat",
    "jwks_uri": "https://gateway.howler.chat/.well-known/jwks.json",
    "response_types_supported": [
        "code",
        "token"
    ],
    "scopes_supported": [
        "openid",
        "email",
        "phone",
        "profile"
    ],
    "subject_types_supported": [
        "public"
    ],
    "token_endpoint": "https://auth.howler.chat/oauth2/token",
    "token_endpoint_auth_methods_supported": [
        "client_secret_basic",
        "client_secret_post"
    ],
    "userinfo_endpoint": "https://auth.howler.chat/oauth2/userInfo"
}

What’s so unusual about this response? For starters, all the actual OAuth2 endpoints are still the main auth server, but the two key differences are important for self-discovery: the issuer and jwks_uri. Both are used by the remaining part of the self-discovery process of ASP.NET:

GET https://gateway.howler.chat/.well-known/jwks.json

{
    "keys": [
        {
            "alg": "RS256",
            "e": "AQAB",
            "kid": "M9U+QO1DuXQHdNgt3BfZmVa04+q4PMF4raal0/Kc6sA=",
            "kty": "RSA",
            "n": "0Mf8RSwPcMfsH2Xvu7MAqTFSOLV....nqjIceBnkOQ6_5ec4DRovow",
            "use": "sig"
        }
    ]
}

These are not the actual values as this service is not yet deployed. Stay tuned for the next session!

This endpoint returns the public keys, that ASP.NET uses for validation of the signature component of the JWT.

Therefore to complete the loop, we must provide two actions to surface the .well-known paths:

/// <summary>
/// Retrieves openid-configuration info.
/// </summary>
/// <returns>
/// Returns openid-configuration info.
/// </returns>
[HttpGet("openid-configuration")]
[ProducesResponseType(typeof(OpenIDConfigurationInfo), 200)]
public OpenIDConfigurationInfo Config()
{
    return new OpenIDConfigurationInfo();
}

/// <summary>
/// Retrieves JWKS info.
/// </summary>
/// <returns>
/// Returns JWKS info.
/// </returns>
[HttpGet("jwks.json")]
[ProducesResponseType(typeof(JWKSInfo), 200)]
public JWKSInfo JWKS()
{
    return new JWKSInfo();
}

The two types OpenIDConfigurationInfo and JWKSInfo simply format the responses in the expected formats above. And like that: you now have a basic fan-out authorization gateway!

Advanced Authentication Flows with QR Codes

2021-06-12T00:00:00+00:00

Good evening, Code Wolfpack! Welcome to our advanced track, where we focus on features that require a combination of cross-cutting aspects to bring to fruition. Today’s case: QR Sign-in!

Magic with Cameras

You’ve likely seen this before in other apps, but if you haven’t, how it looks from the perspective of a user feels like magic: You, as a user, logged in on your mobile phone, scan a QR code displayed on the desktop app. The desktop app shows some visual indicator of communication success, you tap a button on your phone, and boom! Logged in on your desktop, no password, no 2FA required.

In short: magic. But every magic trick has slight of hand that can be explained and understood. Pulling the rabbit out of the hat can be explained in three acts: Presenting a QR Code, supporting mixed authentication providers, and creating a channel to allow the authenticated client to send what it needs to for the unauthenticated client to log in.

Presenting QR Codes

QR Codes are a matrix barcode format which stores data with error-correction keys. Thankfully, presenting QR Codes is a very well-solved problem, and there are great libraries to do this. In our case, supporting the use of QR Codes is as simple as

npm install react-qr-code

and then dropping in the component:

import QRCode from 'react-qr-code';

// ...

public render() {
    return <QRCode value="url-goes-here" level="H"/>;
}

Supporting Mixed Auth Providers

In our project, Howler, we support server federation, which is allowed to designate their own auth server instead of using the official Howler auth gateway. Because of this, we rule out any ability to use our built-in auth gateway (per separation of concerns) or the Cognito service’s auth challenge hooks, as we need to obtain auth tokens for the federated servers the user is connected to. The multi security domain model for authentication is not relevant to today’s topic, so for the sake of simplifying the example, it will be noted that auth tokens will be directly managed via localStorage.

Rather than complicating our workflows by requiring the user go through custom re-authorization flows, which can be relatively fragile, the approach we will use is simply copying the existing tokens for the user from the authorized client to the unauthorized client. But how do we get there? From the perspective of users, the benefit of Howler’s design is that federated servers do not need to rely on Howler infrastructure at all to operate, which means likewise the users who have a similar concern with Howler’s degree of access to auth credentials for those federated providers would object to simply trusting us popping open an SSL-wrapped socket and shuttling traffic through — unless we can prove that you don’t have to trust us. Enter the wonderful world of cryptographic key agreement protocols.

Creating the Channel

Let’s take the naive example first. Assume that you do trust we’re encrypting data at the very least, and the data is eventually expunged at some point, but obviously not for the duration of the exchange. It would likely look something like this: the desktop client creates the channel by providing an id, the mobile client sends the credentials to that id by scanning it as the QR code.

Here’s why that’s a mistake. An attacker can see the credentials too, by behaving like the unauthenticated client. An attacker can also substitute credentials under their own control as well, to honeypot valuable targets. This is an uncommon attack scenario for most users, but not remotely unheard of for activists and journalists.

So how do we solve the nosy neighbor reading credentials? Asymmetric cryptography? Just have the desktop client send a new public key and hang on to the private key to decrypt. The data gets encrypted by the sender, so only the desktop client can decrypt it. That should solve the nosy neighbor problem!

Except it doesn’t, it just changes the game. Compromised networks, either as a corporate or campus network with an SSL decrypting proxy or a malicious attacker controlled network offer the potential of intercepting the request, with the attacker substituting their own public key, allowing them to decrypt the auth tokens, then pass them back along using the desktop client’s public key, leaving the user none-the-wiser. And again, still no solution to the attacker sending credentials.

So how can we ensure that the mobile client receives the right public key? By making the ID itself a derivation of the public key — using a strong hashing algorithm is a great approach. Now the mobile client can confirm the public key is originating from the desktop client. But how do we know the mobile client is the one sending the right credentials? We can send a confirmation of the sender, like a user name or photo. This is about where most services stop their efforts. It looks something like this:

But here’s where they’re getting it wrong. This requires some intermediary trust in the provider of that data. Either the data itself is being implicitly trusted and displayed (the mobile client sent the user profile to show), or it is sourced from a third participant (a user info server), which doesn’t work well in our own model, but also can only be trusted if the auth tokens involved to source that data are exposed to that service, meaning this channel is no longer private to the two participants.

So we’re left with the final problem, eliminating any need for trust at all, by considering the channel to be completely hostile, and performing a type of key exchange where both sides generate elliptic curve keys, and perform a Diffie-Hellman exchange (ECDH). This exchange requires no revealing of private keys, and the mutually derived key never is exposed outside of the memory of the clients. The auth tokens can be safely passed through the channel when encrypted using the derived key as a symmetric key. The final consideration — a man in the middle exchanging keys with both sides — can be solved with a derivation of the symmetric key itself (i.e. a hash), and displaying either part or whole of the hash on both clients for the user to confirm. If they match, there is no MITM.

Want to see it all pieced together, take a look at the commit!

It’s very rudimentary, but it works, and we’ll style it, and get it rolled out before next Code Wolfpack. See you then!

Let’s Build Discord, Session 8 - Managing State with React-Redux and Sagas

2021-05-22T00:00:00+00:00

Good evening, Wolfpack! This session is building on top of the React session that precedes it, so if you need a refresher, check it out! Our goal today is to gain an understanding of React-Redux, and encapsulating complex state management flows with the effects system of Redux-Saga.

State? I thought we managed state in Components?

Component state should be considered something specific to the component itself, generally. As an example, consider an accordion component which manages its open/closed status with a state property. When managing the internal state of a component, keep it in the component. When managing a state that crosses multiple component boundaries, this is where Redux shines.

Redux

Redux is a very simple library that makes it easy to separate the logic that manages state from the things that initiate those changes. In pure Redux terms, the logic that compresses state changes to the latest state object is called a reducer, and the things that initiate those changes are called actions. The overall state management object which handles dispatched actions and reduces the state is called the store. When combined with React via react-redux, you get a way to manage state from anywhere in the app with some convenience methods that take away some of the burdens of tapping into a global state tree.

Actions

Actions are simply objects with a type key. They may carry additional data to parameterize the action, but the core requirement of an action is that the type key is present. Typical conventions utilize upper-case string constants, but this is not required. Action creators, thus, are simple functions that return an action:

export const actionCreators = {
    increment: () => ({ type: 'INCREMENT_COUNT' } as IncrementCountAction),
    decrement: () => ({ type: 'DECREMENT_COUNT' } as DecrementCountAction)
};

These are handy for two reasons:

They make construction of actions well-defined rather than expecting interactors with the store to dispatch arbitrary objects (which could break if the reducer’s expectations change).
Action creators are readily bound to the dispatch method with bindActionCreator (this is called when you connect the store to a component with React-Redux).

Reducers

Reducers are also simple: they consume the previous state, and return an applicative change to that state, as a new state object:

export const reducer: Reducer<CounterState> = 
    (state: CounterState | undefined, incomingAction: Action): CounterState => {
    if (state === undefined) {
        return { count: 0 };
    }

    const action = incomingAction as KnownAction;
    switch (action.type) {
        case 'INCREMENT_COUNT':
            return { count: state.count + 1 };
        case 'DECREMENT_COUNT':
            return { count: state.count - 1 };
        default:
            return state;
    }
};

You can see that this does not alter the previous state object directly, but instead returns a copy if changed. This is a crucial behavior that is important to maintain. Keeping the reducer idempotent (when calling the function, it does not change in behavior when called multiple times) helps prevent bugs, makes the reducer easier to comprehensively test, and is an assumption expected by enhancements in Redux like the devtools so that you can utilize its “time travel” feature.

React-Redux

Adding React into the mix is easy. With the store configured and created, simply provide it to the <Provider store={store}/> component from the react-redux package, wrapping this component around your entire React app:

ReactDOM.render(
    <Provider store={store}>
        <ConnectedRouter history={history}>
            <App />
        </ConnectedRouter>
    </Provider>,
    document.getElementById('root'));

In order to expose a given action creator (or set of action creators) to a component, you only need export your component with the connect wrapper function:

export default connect(
    (state: ApplicationState) => state.counter,
    CounterStore.actionCreators
)(Counter);

The first argument to connect is the mapStateToProps argument. It expects a function which when given the root state object, returns an object whose keys will become properties on the component. The second argument is mapDispatchToProps. It expects a variety of options, which inevitably are either dispatch calls, or become dispatch calls (like action creators which can be bound to dispatch with bindActionCreator). This method returns a higher-level component (or component enhancer) which wraps your component and provides the store and actions as props. When using TypeScript, note you will need to update the type signature of your component to allow these as part of the props type parameter:

type CounterProps =
    CounterStore.CounterState &
    typeof CounterStore.actionCreators &
    RouteComponentProps<{}>;

class Counter extends React.PureComponent<CounterProps> {
    // ...
}

Next Steps

So far, so good, right? You no longer have to pass down functions from a grandparent component to make it possible to call APIs and refresh state – it’s so nice! But what about promises? You have to thread through the dispatch method and call it inside your .then callback. Ok, not great, but it still works. But what if your API is variable in response rate? What if your connection is flaky? What if a user just clicks the everloving hell out of a button that dispatches an action to call your API? There’s many ways to handle this. Loads of React apps will employ some kind of state machine management that alters how action creators function (losing functional purity or getting really messy), or put this logic in the reducer, or some other kind of workaround. These work, but then it starts to become a common pattern as it’s not a unique problem in the app, and suddenly your meta state management becomes a sort of transaction scoping handler. Thankfully this is a well understood (and solved) problem, and also thankfully, it has been solved for Redux. Enter redux-saga.

Redux-Saga

It is a major frustration of mine that there are very few solid tutorials out there for Redux-Saga, because the library is very simple in design, but difficult to understand how to apply it. How do I know it’s difficult to understand how to apply it? Because it’s been out for quite a while, and one of the simplest problems emerging from plain Redux (or redux-thunk) usage has roughly six times the results on Google compared to Redux-Saga itself:

Oof. And while you could ascribe it to marketing, there’s definitely a lot of confusion when you look up tutorials, many of which echo the basic example from Redux-Saga’s website with no real nuance to why someone would want to use it, or even how to choose the various effects. So let’s talk about what makes it tick!

In the above scenario of a long-running API call and an impatient user, assume the naive implementation of an action creator with redux-thunk middleware (the recommendation from the official Redux docs):

export const actionCreators = {
    fetchWeatherForecast: () => dispatch => {
        dispatch({type: 'REQUESTING_WEATHER_FORECASTS'});
        fetch(`https://someservice/v1/api/weatherforecast`)
            .then(response => response.json())
            .then(forecast => dispatch({type: 'RECEIVE_WEATHER_FORECASTS', forecast}));
    }
};

You could optionally use async/await syntax to reduce callback hell, but the main issue remains present regardless. If you want to block repeat calls until the first call has completed, you have to incorporate state:

export const actionCreators = {
    fetchWeatherForecast: () => (dispatch, getState) => {
        const state = getState();
        if (!state.weatherForecast.isLoading) {
            dispatch({type: 'REQUESTING_WEATHER_FORECASTS'});
            fetch(`https://someservice/v1/api/weatherforecast`)
                .then(response => response.json())
                .then(forecast => dispatch({type: 'RECEIVE_WEATHER_FORECASTS', forecast}));
        }
    }
};

Of course, this now requires your reducer is setting isLoading appropriately based on start/completion of the action. This is the easy scenario, all is still right with the world, no extra libraries are needed to manage state. BUT! What if you cared about getting the response for the last time the action was dispatched, rather than the first? I’ll leave that as an exercise for the reader in implementing with redux-thunk alone, and in the meantime, show how Sagas can help.

How Sagas bring storytelling to actions

It starts with a handy recent addition (relatively speaking) to the Javascript language: generator functions. How do they work?

function* foo() {
    yield "this";
    yield "then this";
    yield "also this";
}

let bar = foo();
bar.next() // { value: "this", done: false }
bar.next() // { value: "then this", done: false }
bar.next() // { value: "also this", done: false }
bar.next() // { value: undefined, done: true }

Redux-Saga uses generator functions to wrap complex workflows with very simple calls that declare the sequence of events and expectations. To add Sagas to redux, just import createSagaMiddleware() and take the collection of sagas and pass them into the middleware’s run() method:

export default function configureStore(history: History, initialState?: ApplicationState) {
    const sagaMiddleware = createSagaMiddleware();
    const middleware = [
        sagaMiddleware,
        routerMiddleware(history)
    ];

    const rootReducer = combineReducers({
        ...reducers,
        router: connectRouter(history)
    });

    const enhancers = [];

    const store = createStore(
        rootReducer,
        initialState,
        compose(applyMiddleware(...middleware), ...enhancers)
    );

    sagaMiddleware.run(rootSaga);

    return store;
}

Your root saga will look like this (very much like the root reducer):

export const rootSaga = function*() {
    yield all([
        WeatherForecasts.weatherSagas
    ]);
};

And your individual section of the sagas:

export const weatherSagas = [
    function*() {
        yield takeLeading('REQUEST_WEATHER_FORECASTS', handleWeatherRequest)
    }
]

In that takeLeading method, there is magic happening. What this does is all the logic we described above, in tracking the completion state of handling the action 'REQUEST_WEATHER_FORECASTS', and essentially discards the subsequent dispatches until the the provided generator function handleWeatherRequest has completed. Now what does that method look like?

function* handleWeatherRequest(request: RequestWeatherForecastsAction) {
    let forecast: Body = yield call(fetch, `https://someservice/v1/api/weatherforecast`);
    let data: WeatherForecast = yield call(() => forecast.json());
    yield put({type: 'RECEIVE_WEATHER_FORECASTS', forecasts: data });
}

Ok, so this looks very different from before. What are call and put? They are part of the collection of methods that Redux-Saga provides to manage the relationships between dependent action chains, and even enable awesome features like cancellable actions with fork and cancel. This decomposition allows for much cleaner workflows. Back to that question that prompted Sagas in the first place: what about handling the last request rather than the first? Instead of takeLeading, use takeLatest. Need to pause a saga until another action has occurred? wait for it with take in your series of yields. Need to buffer actions to be processed one at a time? Saga can do that with actionChannel.

When thinking about building a UI for a chat client, this may start to ring a bell: we want users to have a responsive UI that lets them send messages as fast as they can type them, but each one needs to be processed in order. Using actionChannel buffers for individual rooms (or in Howler’s terms, Channels, but I want to avoid the ambiguity here), you can allow a user to (from their point of view), send many messages in one room, switch to another, send more messages, and if their connection is slow, or flaky, or Howler’s API itself has crumbled under their rapid-typing might, the ordering of messages delivered as actions are preserved relative to the room, giving a great user experience. You could do that with redux-thunk, but I don’t even want to begin thunking about how you’d do it well.

Let’s Build Discord, Session 7, Part 2 - React vs React Native

2021-05-21T00:00:00+00:00

Happy Friday! For those coming over from Part 1, hopefully you’ve had an opportunity to explore the React and React Native projects in the project repo and get things running locally. Once you’ve done so, it’s time to dive into the details of React and React Native.

For those with a very minimal background in Javascript, especially in the context of Node, it is common to feel lost initially. React apps, especially those generated via create-react-app have some very nice convenience scripts that are abstracting away some of the burdens of development. You can find more details about the react-scripts library in Session 0 materials (this is being re-built for the Code Wolfpack site, and this will be updated with a link when complete).

React vs React Native

React is somewhat overloaded as a term, as React itself through many iterations has moved much of the DOM-specific functionality into a separate package, react-dom. That being said, there is still a lot to React that finds itself tied into the DOM-related functionality, which is revealed in the @types/react supplementary library for TypeScript. As a consequence to this, you cannot simply interrogate package dependencies in a React library for the presence of react-dom to determine if the library can be used in React Native, and oftentimes your best bet when developing React Native applications is to approach the two libraries as separate entities with very similar semantics. More simply put: React for web, React Native for mobile.

If you want a curated directory of React Native libraries, check out https://reactnative.directory/

React

React takes flight the moment the render loop is started. You can see this in index.tsx:

ReactDOM.render(
    <Provider store={store}>
        <ConnectedRouter history={history}>
            <App />
        </ConnectedRouter>
    </Provider>,
    document.getElementById('root'));

JSX and TSX files allow us to substitute calls to React.createElement, otherwise this would look like:

ReactDOM.render(
    React.createElement(
        Provider,
        {store},
        React.createElement(
            ConnectedRouter,
            {history},
            React.createElement(App, null, null)
        )
    ),
    document.getElementById('root'));

It doesn’t take much of a leap to understand why the former is preferable. The PascalCased tags are Components (and in JSX, so too are the lower-cased DOM elements!). Components are classes which inherit from React.Component or one of its variant subtypes. Components exist to encapsulate logic in discrete, composable elements, by standardizing and exposing hooks to its own lifecycle, providing state management, and enforcing a pattern that drives data down into the render phase, and pipes actions up to where they are needed. This drastically simplifies data binding in web applications, as other libraries which allow two way binding have a bad tendency to be a source of bugs due to responding to unexpected state changes (e.g. “Somewhere down the chain of components the data was updated and now this component is sending events because I had a listener on it”).

Components are passed attributes like normal tags:

<ExampleComponent propA={2} propB={true} propC={nameOfFunction}/>

and in the ExampleComponent class, they can be accessed via props:

class ExampleComponent extends React.Component {
    render() {
        { propA, propB, nameOfFunction } = this.props;
        if (propB) {
            let result = nameOfFunction();
            return <div>{result}</div>;
        } else {
            return <div>{propA}</div>;
        }
    }
}

To manage state in a component, you use the state property:

class ExampleComponent extends React.Component {
    constructor(props) {
        super(props);

        this.state = {
            nameOfStateProperty: "banana"
        };
    }

    render() {
        let { propA, propB, nameOfFunction } = this.props;
        if (propB) {
            let result = nameOfFunction();
            return <div>{result} - {this.state.nameOfStateProperty}</div>;
        } else {
            return <div>
                <p>{propA}</p>
                <button onClick={(e) => this.setState({nameOfStateProperty: "apple"})}>
                    {this.state.nameOfStateProperty}
                </button>
            </div>;
        }
    }
}

React Native

Other than native-specific bindings, React Native is essentially the same thing as React, except instead of DOM components, you have to use the React Native components - these components are turned into native UI elements on the rendered application.

class Section extends Component {
  render() {
    let {children, title, isDarkMode} = this.props;
    return (
      <View style={styles.sectionContainer}>
        <Text
          style={[
            styles.sectionTitle,
            {
              color: isDarkMode ? Colors.white : Colors.black,
            },
          ]}>
          {title}
        </Text>
        <Text
          style={[
            styles.sectionDescription,
            {
              color: isDarkMode ? Colors.light : Colors.dark,
            },
          ]}>
          {children}
        </Text>
      </View>
    );
  }
}

Look familiar? Instead of <div/>s, we’re using <View/>. Instead of <p/> or <span/>, we’re using <Text/>. What happens if you try to use HTML elements instead?

They’re pretty clear about expectations. There are more advanced concepts like React Hooks, state management with Redux and advanced state management boilerplate like Sagas, but these are reserved for a more in-depth session. Happy experimenting!

Let’s Build Discord, Session 7, Part 1 - React/React Native Intro

2021-05-15T00:00:00+00:00

Good evening, Wolfpack! For today’s session we are going to be focusing on React and React Native. Our goal for this session will be to get our web app into a state where it can integrate with our auth server, and get you introduced to React and React Native.

Quick News

Intra-session update! During the time between last session, AWS finally fixed the issues with Cognito’s CloudFront distribution blocking the deploy of custom domains, and so I was able to move from the quite hideously verbose Cognito default auth URL and to our branded subdomain auth.howler.chat.

Ingredients List

For today’s session, you will need to have the following installed (if you participated in previous sessions you should already have these):

For Windows specifically:

For Mac:

Xcode

Be sure to also clone the latest source of Howler from https://github.com/HowlerChat/Howler using your terminal (on Windows, Git Bash):

git clone git@github.com:HowlerChat/Howler.git

Or if you’ve already cloned it before, just run git pull.

The other thing you’ll need to install is React Native tooling. In a terminal, simply run:

npm install -g react-native

One more thing for Windows folks: Getting all the right SDKs and tools installed is an absolute pain to do manually, so the React Native team has created a helpful setup script. Follow the instructions here: https://microsoft.github.io/react-native-windows/docs/rnw-dependencies

Prep-work, Cooking Show Style

The work involved in this session is already complete, so if you want to turn back time to what the repository looked like, you can check out a copy of the code from that time:

git checkout -b session-7-start 1ce0595565513df7481580f8ec23ca77aa54efd7

Pre-heating The Oven

Let’s make sure you can build and run both the web app server and the React Native apps.

React

Starting from the project root, we’re going to do a little spring cleaning and use the Howler.WebsocketTest project as our base for the React app, so let’s rename it:

mv Howler.WebsocketTest Howler.App
mv Howler.App/Howler.WebsocketTest.csproj Howler.App/Howler.App.csproj

Be sure to also rename all instances of Howler.WebsocketTest in the Howler.sln file and all the namespaces in the Howler.WebsocketTest/ folder.

Change directories to the Howler.App/ClientApp folder, install all node packages, then move back up a directory and start the web server:

cd Howler.App/ClientApp
npm install
cd ..
dotnet run

Visit the url indicated by the console (accept the certificate warning if present), and you should be greeted by a familiar sight of our old Websocket test page:

Great, the React app is now running. Next up, React Native

React Native

Move to the HowlerMobile folder, install all node packages, and start the Metro server (this is required in order to run the React Native apps):

cd ../HowlerMobile
npm install
npx react-native start

You should be presented with a tidy little ASCII graphic when this is done:

Depending on your target for React Native (Android, iOS, Universal Windows Platform), you will need to run the relevant command or tool. In this session, we demonstrate the Windows app, by opening HowlerMobile/windows/howlermobile.sln

Choose the relevant platform (probably will be x64), and hit Local Machine.

After some churning and burning, you should finally see your React Native app start:

Perfect! In the next post we will move on to the finer details of differences in Part 2: React vs React Native

Console.WriteLine(“Hello World”);

2021-05-14T00:00:00+00:00

Hi Wolfpack! After a brief hiatus, I have gotten a site put together to coordinate our Code Wolfpack sessions, which will let us grow bigger and better than ever! I’m looking forward to seeing you all soon!