Key Transparency Architecture

Introduction Before any information can be exchanged in an end-to-end encrypted system, two things must happen: First, participants in the system must provide the service operator with any public keys they wish to use to receive messages. Second, the service operator must somehow distribute these public keys amongst the participants that wish to communicate with each other. Typically, users upload their public keys to a simple directory where other users can download them as needed, or exchange public keys in-band with the communication being secured. If users simply trust the service operator to forward public keys correctly, the underlying encryption protocol can only protect them against passive eavesdropping. However, most messaging systems are designed such that all messages are exchanged through the service operator's servers, which makes it extremely easy for an operator to launch an active attack. That is, the service operator can generate a rogue public key for which it holds the corresponding private key, and associate that rogue public key with a user's account without the user's knowledge, allowing it to impersonate or eavesdrop on conversations with that user. Key Transparency (KT) solves this problem by requiring the service operator to store user public keys in a cryptographically protected append-only log. Any malicious entries added to such a log will generally be equally visible to both the affected user and the user's contacts. This allows users to detect whether they are being impersonated by viewing the public keys attached to their account. If the service operator attempts to conceal some entries of the log from some users but not others, this creates a "forked view" which is permanent and detectable. The term "public key" is used broadly in this document to refer to any public cryptographic material that a user associates with their account; depending on the application, this may be a raw public key, a certificate containing a public key, or some other application-defined structure. KT does not depend on the format of this data. The critical improvement of KT over related protocols like Certificate Transparency is that KT includes an efficient protocol to search the log for entries related to a specific participant. This means users don't need to download the entire log, which may be substantial, to find all entries that are relevant to them. It also means that KT can better preserve user privacy by only showing entries of the log to participants that genuinely need to see them.

Definitions

End-to-End Encrypted Communication Service:: A communication service that allows end-users to engage in text, voice, video, or other forms of communication over the internet, and uses public key cryptography to ensure that communications are only accessible to their intended recipients.
Service Operator:: The primary organization that provides the infrastructure for an end-to-end encrypted communication service and the software to participate in it.
End-User Device:: The device at the final point in a digital communication, which may either send or receive encrypted data in an end-to-end encrypted communication service.
User / Account:: A single end-user of an end-to-end encrypted communication service, which may interact with the service on multiple end-user devices (phone, laptop).
End-User Identity:: A unique and user-visible identity associated with an account (and therefore one or more end-user devices) in an end-to-end encrypted communication service. In the case where an end-user explicitly requests to communicate with (or is informed they are communicating with) an end-user uniquely identified by the name "Alice", the end-user identity is the string "Alice". A single account may be represented simultaneously by multiple end-user identities (email, phone number, username).
Transparency Log:: A specialized service capable of securely attesting to the information (such as public keys) associated with a given end-user identity. A transparency log is usually run either entirely or partially by the service operator, but could also be operated externally.
Transparency Log Configuration:: The fixed, public parameters of a Transparency Log such as the log's cipher suite and public key. Configuration is pre-distributed to users through a trustworthy channel, typically as part of the communication service's client software, and used for proof verification.
Label:: A lookup key in the key-value database that a transparency log represents, such as an end-user identity. See .
Label Owner:: A user that either initiates all changes to a label's value, or must be informed of all changes to it. See .
Prefix Tree:: A hash tree that commits to the set of all labels stored in a transparency log, and every version of each label's value, at a given point in time.
Log Tree:: A hash tree with one leaf for each change to the transparency log. Each leaf commits to a snapshot of the prefix tree, so the log tree as a whole commits to the history of all label modifications.
Tree Head:: The root hash and size of the log tree at a given point in time, signed by the transparency log (and possibly by a third party).

Protocol Overview From a networking perspective, KT provides a client-server architecture with a central transparency log, acting as a server, which holds the authoritative copy of all information and exposes endpoints that allow users to query or modify stored data. Users coordinate with each other through the server by uploading their own public keys and downloading the public keys of other users. Users are expected to maintain relatively little state, limited only to what is required to interact with the log and ensure that it is behaving honestly. From an application perspective, KT can be thought of as a versioned key-value database. Users insert key-value pairs into the database where, for example, the key is their username and the value is their public key. Users can update a key by inserting a new version with new data. They can also look up the most recent version of a key or any previous version. From this point forward, the term label will be used to refer to lookup keys in the key-value database that a transparency log represents to avoid confusion with cryptographic public or private keys. Users are considered to own a label if they are understood to either initiate all changes to the label's value, or if they must be informed of all changes to the label's value. The latter situation may occur if, for example, KT is deployed in a way where the service operator makes automated modifications to stored data. The owning user would then be informed, after the fact, of modifications to verify that they were legal. KT does not require the use of a specific transport protocol. This is intended to allow applications to layer KT on top of whatever transport protocol their application already uses. In particular, this allows applications to continue relying on their existing access control system. With some small exceptions, applications may enforce arbitrary access control rules on top of KT. This may include requiring a user to be logged in to make KT requests, only allowing a user to look up the labels of their contacts, or applying a rate limit. Most applications will likely need to, at minimum, prevent users from modifying labels they do not own. The exact mechanism for rejecting requests, and possibly explaining the reason for rejection, is left to the application. The exceptions are the requests that users need to make to verify the transparency log's behavior. In particular, a label's owner needs to always be able to monitor their label for as long as they own it, and in the Contact Monitoring deployment mode, users that have looked up a label need to be able to complete their monitoring of it even if their access to perform other queries is later revoked. Refusing these requests is indistinguishable from transparency log misbehavior (see ).

User Operations The operations that can be executed by a user are as follows:

Search: Looks up the value of a specific label in the most recent version of the log. Users may request either a specific version of the label or the most recent version available. If the label-version pair exists, the server returns the corresponding value and a proof that the search operation was executed correctly.
Update: Adds a new label-value pair to the log. The server returns a proof that the label-value pair was added correctly. Note that this means that new values are added to the log immediately and no provisional proof, such as a Signed Certificate Timestamp (SCT) as defined in , is provided.
Monitor: While Search and Update are run by the user as necessary, monitoring is done in the background on a recurring basis. It checks both that the log is continuing to behave honestly (all previously returned labels remain in the log) and that no changes have been made to labels owned by the user without the user's knowledge.

These operations are executed over an application-provided transport layer. The transport layer enforces access control by blocking queries which are not allowed:

Credentials While users are generally understood to interact directly with the transparency log, many end-to-end encrypted communication services require the ability to provide credentials to their users. Credentials convey a binding between an end-user identity and public keys or other information, and can be verified with minimal network access. Traditionally, this binding is provided by a certificate (such as an X.509 certificate) signed by a trusted authority. KT credentials are instead constructed from the transparency log's own responses. Credentials are especially useful in applications that support anonymous communication. These applications provide end-to-end encryption with a protocol like the Messaging Layer Security Protocol (with the encryption of handshake messages required) or Sealed Sender . When a user sends a message, these protocols have the sender provide their own credential in an encrypted portion of the message. Encrypting the sender's credential allows the sender to submit messages over an anonymous channel by specifying only the recipient's identity. The service operator can deliver the message to the intended recipient, who can decrypt it and validate the credential inside to be assured of the sender's identity. Note that the recipient does not need access to an anonymous channel to preserve the sender's anonymity. At a high level, KT credentials are created by serializing one or more Search request-response pairs. These Search operations correspond to the lookups the recipient would do to authenticate the relationship between the presented end-user identity and their public keys. Recipients verify the request-response pairs themselves, using the transparency log's configuration, without contacting the transparency log. Because the responses are authenticated by the transparency log (and any third-party manager or auditor, as described in ), a credential that has been modified will fail verification. To prevent stale credentials from being accepted, recipients only accept credentials that are based on a recent distinguished log entry (see ), where the application defines what is considered recent. Senders therefore need to periodically refresh their credentials. Any future monitoring that may be required should be provided to recipients proactively by the sender, as this is the most robust way to preserve the sender's anonymity. If required future monitoring isn't provided, either accidentally or because the system was intentionally designed that way, the recipient will need to perform the monitoring themself over an anonymous channel. shows an example message flow in an anonymous deployment. Users request their own label from the transparency log and provide the serialized response, functioning as a credential, in encrypted messages to other users. Required monitoring is provided proactively in the same way.

Detecting Forks It is sometimes possible for a transparency log to present forked views of data to different users. This means that, from an individual user's perspective, a log may appear to be operating correctly in the sense that all of a user's requests succeed and proofs verify correctly. However, the transparency log has presented a view to the user that's not globally consistent with what it has shown other users. As such, the log may be able to change a label's value without the label's owner becoming aware. The protocol is designed such that users always require subsequent queries to prove consistency with previous queries. As such, each user's view of the log always forms a single, consistent history. If a user is ever presented with a forked view, they hold on to this forked view forever and reject the output of any subsequent queries that are inconsistent with it. This provides ample opportunity for users to detect when a fork has been presented but isn't in itself sufficient for detection. To detect forks, users require either a trusted third party, anonymous communication with the transparency log, or peer-to-peer communication. Anonymous and peer-to-peer communication are collectively referred to as out-of-band (OOB) communication. With a trusted third party, such as a Third-Party Manager or Auditor as described in and , an outside organization monitors the operation of the transparency log. This third party verifies, among other things, that the transparency log is growing in an append-only manner. If verification is successful, the third party produces a signature on the most recent tree head. The transparency log provides this signature to users inline with their query responses as proof that they are not being shown a fork. This approach relies on an assumption that the third party is trusted not to collude with the transparency log to trick users into accepting malicious results. With anonymous communication, a single user accesses the transparency log over an anonymous channel and checks that the transparency log is presenting the same tree head over the anonymous channel as it does over an authenticated channel. The security of this approach relies on the idea that, if the transparency log doesn't know which user is making the request, it will show the user the wrong fork with high probability. Repeating this check over time makes it overwhelmingly likely that any fork presented to any user will be detected. To ensure this, requests over the anonymous channel must not contain any user-specific data or state, such as cookies, authentication tokens, or the tree size last observed by the user. They also must not be correlated in time with the user's authenticated requests, such as by being sent immediately before or after them. With peer-to-peer communication, two users gossip with each other to establish that they both have the same view of the transparency log. This gossip is able to happen over any supported out-of-band channel even if it is heavily bandwidth-constrained, such as scanning a QR code. However, this approach is only secure if gossip can be implemented such that gossiping users are reasonably expected to form a connected graph of all users. If not, then the transparency log can attempt to partition users into subsets that do not gossip and can present each subset of users with different forks. Applications that do not provide their users with a trusted third party, anonymous communication, or peer-to-peer communication are not secure, as the transparency log can present forked views to different users without detection. Regardless of the approach an application chooses, in the event that a fork is successfully detected, the user is able to produce non-repudiable proof of log misbehavior which can be published. illustrates fork detection with anonymous and peer-to-peer communication. Users receive tree heads while making authenticated requests to a transparency log. Users ensure consistency of tree heads by either comparing amongst themselves, or by contacting the transparency log over an anonymous channel. Requests that require authentication (Search(Bob) in this example) may be blocked over the anonymous channel. The tree heads that users compare out-of-band (DistinguishedHead in this example) correspond to distinguished log entries, which are described in .

Out-of-Band Fork Detection | | |<----------- Response{Head: 6c063bb, ...} | | | | | | | | | | | | | | (OOB check with peer) | (OOB check over anonymous channel) | | | | |<------- DistinguishedHead | DistinguishedHead ~~~~~~~~~~~~~~~~~~~~~> | | 6c063bb ----------------->| <~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6c063bb | | | | | | (Auth'ed reqs blocked over anon channel) | | | | | | Search(Bob) ~~~~~~~~~~~~~~~~~~~> X | | | | ]]>

Deployment Modes In the interest of satisfying the widest range of use-cases possible, three different modes for deploying a transparency log are supported. Each mode has slightly different requirements and efficiency considerations for both the service operator and the end-user. Third-Party Management and Third-Party Auditing are two deployment modes that require the service operator to delegate part of its transparency log operations to a third party. With both third-party modes, all requests from end-users are initially routed to the service operator, and the operator coordinates with the third party itself. End-users never contact the third party directly, although they will need a signature public key from the third party to verify its assertions. With Third-Party Management, the third party performs the majority of the work of actually storing and operating the service, and the service operator mainly enforces access control and signs new entries as they're added. With Third-Party Auditing, the service operator performs the majority of the work of storing and operating the service, only obtaining signatures from a third-party auditor at regular intervals asserting that the tree has been constructed correctly. To reduce the probability of collusion between the service operator and the third party, a service operator can have two or more independent third parties coordinate as one and produce threshold signatures. In this scenario, the threshold for a valid signature needs to be at least a majority of the third parties to prevent different subsets from authenticating forked views. Contact Monitoring, on the other hand, supports a single-party deployment with no third party. The cost of this is that, when a user looks up a version of a label that was inserted very recently, the user may need to retain some additional state and monitor the label until it is included in a distinguished log entry. Distinguished log entries are chosen at regular intervals, roughly one per Reasonable Monitoring Window (RMW), which is a duration configured by the transparency log that reflects how frequently label owners are expected to perform monitoring. Both are specified in more detail in . If a user looks up many label-version pairs that were inserted very recently, monitoring may become relatively expensive. Additionally, applications that rely on a transparency log deployed in Contact Monitoring mode need to regularly attempt to detect forks through anonymous communication with the transparency log or peer-to-peer communication, as described in . Applications that rely on a transparency log deployed in either of the third-party modes can allow users to enable a "Contact Monitoring mode". This mode, which affects only the individual client's behavior, would cause the client to behave as if its transparency log were deployed in Contact Monitoring mode. As such, it would start retaining state about previously looked-up labels and regularly engaging in out-of-band communication. Enabling this higher security mode allows users to double check that the third party is not colluding with the transparency log.

Contact Monitoring With the Contact Monitoring deployment mode, the monitoring burden is split between both the owner of a label and those that look up the label. Stated as simply as possible, the monitoring obligations of each user are:

On a regular basis, the label owner verifies that the most recent version of their label has not changed unexpectedly.
When a user that looked up a label sees that it was inserted very recently, they check back later to see that the label-version pair they observed was not removed before it could be detected by the label owner.

This ensures that if a malicious value for a label is added to the log, then either it is detected by the label owner, or if it is removed/obscured from the log before the label owner can detect it, then any users that observed it will detect its removal. As such, the transparency log is not trusted to behave honestly in this mode; rather, any deviation from the protocol or the addition of malicious values for a label is detected by users through monitoring. In the example shown in , Alice searches for Bob's label. One day later, Alice verifies that the label-version pair she observed remained in the transparency log. Another day later, Bob comes online and monitors his own label. Note that Alice does not need to wait on Bob to make his Monitor request before making hers.

Contact Monitoring | | |<------------- SearchResponse(...) | | | | | | (1 day later) | | | | | | Monitor(Bob) -------------------->| | |<------------ MonitorResponse(...) | | | | | | (1 day later) | | | | | | |<------------------- Monitor(Bob) | | | MonitorResponse(...) ----------->| | | | ]]> Importantly, Contact Monitoring impacts how the server is able to enforce access control on Monitor queries. While Search and Update queries can enforce access control on an arbitrary "point in time" basis, where a user is allowed to execute the query at one point in time but maybe not the next, users are sometimes required by the protocol to make Monitor queries based on the Search and Update queries they made in the past. These Monitor queries need to be permitted, regardless of whether or not the user is still permitted to execute such Search or Update queries.

Third-Party Management With the Third-Party Management deployment mode, a third party is responsible for the majority of the work of storing and operating the log. The service operator focuses mainly on enforcing access control and authenticating the creation of new versions of labels. All user queries are initially sent by users directly to the service operator, and the service operator proxies them to the third-party manager if they pass access control. New versions of a label, if allowed by the service operator, are signed by the service operator before being sent to the third-party manager to be inserted into the tree. As part of authenticating the creation of new versions of labels, the service operator must implement some mechanism to prevent the same version of a label from being submitted to the third-party manager multiple times with different associated values. Additionally, the service operator must implement some mechanism to detect when forks are presented by the third-party manager.

Third-Party Management | ------------------------------>| |<--------------------------- |<---------- SearchResponse(...) | | | | | Search(Bob) --------------->| ------------------------------>| |<--------------------------- |<---------- SearchResponse(...) | | | | | Update(Alice, ...) -------->| ------------------------------>| |<--------------------------- |<---------- UpdateResponse(...) | | | | | Search(Fred) ----------> X | | | Update(Bob, ...) ------> X | | | | | ]]> The security of the Third-Party Management deployment mode comes from an assumption that the service operator and the third-party manager do not collude to behave maliciously. If the third-party manager behaves honestly, then any improper modifications to a label's value that were requested by the service operator will be properly published such that the label owner will detect them when monitoring. If the service operator behaves honestly, the third-party manager will be unable to add any new unauthorized versions of a label such that a user will accept them, or remove any authorized version of a label without the label owner detecting it.

Third-Party Auditing With the Third-Party Auditing deployment mode, the service operator obtains signatures from a third-party auditor attesting to the fact that the tree has been constructed correctly. These signatures are provided to users as part of the responses for their queries. When running synchronously, the auditor can easily become a bottleneck for the transparency log. It's generally expected that third-party auditors run asynchronously, downloading and authenticating a log's contents in the background. As a result, signatures from the auditor may lag behind the view presented by the service operator. The maximum amount of time that the auditor may lag behind the service operator without its signature being rejected by users is set in the transparency log's configuration. As shown in , the auditor is updated on changes to the tree in the background, and a recent signature from the auditor is provided to users.

Third-Party Auditing | | | Update(Bob, ...) --------------------->| | | Update(Carol, ...) ------------------->| | | <===== Response{AuditorSig: 66bf, ...} | | | | | | | | | | AuditorUpdate ------------->| | |<----------- AuditorTreeHead | | | | ]]> Given the long-lived nature of transparency logs and the potentially short-lived nature of third-party auditing arrangements, KT allows third-party auditors to start auditing a log at any arbitrary point. This allows a new third-party auditor to start up without ingesting the transparency log's entire history. The point at which an auditor started auditing is provided to users in the transparency log's configuration. When verifying query responses, users verify that the auditor started auditing at or before the point necessary for the query to be secure.

Combining Logs There are many cases where it makes sense to operate multiple cooperating transparency log instances, for example:

A service operator may wish to gradually migrate to a transparency log that uses different cryptographic keys, a different cipher suite, or a different deployment mode.
A service operator may need to immediately migrate to a new transparency log to recover from log failure, such as the compromise of the log's private keys or the loss of its data.
A service operator may operate multiple logs to handle the total load of a high-traffic application.
A federated system may allow each participant in the federation to operate their own transparency log for their own users.

Client implementations should generally be prepared to interact with multiple independent transparency logs to support these use-cases, particularly recovery from log failure. When multiple transparency logs are used as part of one application, all users need to have a consistent policy for executing Search, Update, and Monitor queries against the logs in a way that maintains the high-level security model of KT:

If all transparency logs behave honestly, then users observe a globally consistent view of the data associated with each label.
If any transparency log behaves dishonestly, this will be detected in a timely manner by background monitoring or out-of-band communication.

Gradual Migration In the case of gradually migrating from an old transparency log to a new one, this policy may look like:

Search queries are executed against the old transparency log first, and then against the new transparency log only if the most recent version of a label in the old transparency log is a special application-defined 'tombstone' entry.
Update queries are only executed against the new transparency log, with the exception of adding a tombstone entry for the label to the old transparency log if one hasn't been added already.
Both transparency logs are monitored as they would be if they were run individually. Once the migration has completed and the old transparency log has stopped accepting modifications, the old transparency log needs to stay operational long enough for all users to complete their monitoring of it, keeping in mind that some users may be offline for a significant amount of time. If the old transparency log is shut down before a user is able to complete their monitoring of it, that user will be unable to detect some forms of misbehavior by the old transparency log.

Placing a tombstone entry for each label in the old transparency log gives users a clear indication as to which transparency log contains the most recent version of a label. Importantly, it prevents users from incorrectly accepting a stale version of a label.

Immediate Migration In some situations, the service operator may instead choose to stop adding new entries to a transparency log immediately and provide a new transparency log that is pre-populated with the most recent versions of all labels. In this case, the policy may look like:

Search queries must be executed against the new transparency log.
Update queries must be executed against the new transparency log.
The final tree head of the old transparency log is provided to users over a trustworthy channel. Users issue their final Monitor queries to complete monitoring up to this point. Label owners initiate monitoring state for the new transparency log by processing an Update for the migrated versions of their labels and verifying that the migration was done correctly. From then on, users will monitor only the new transparency log.

The final tree head of the prior transparency log needs to be distributed to users in a way that ensures that all users have a globally consistent view. This can be done by storing them in a well-known label of the new transparency log. Users need to process this well-known label as if they own it, so that they continue to monitor it for unexpected changes for the duration of the migration period. Alternatively, the final tree head may be distributed with the application's code distribution mechanism. Regardless, if a migration is the result of one or more private key compromises, the final tree head must not be authenticated by a compromised private key.

Partitioning Some applications partition labels across multiple transparency logs, where each transparency log is responsible for a disjoint subset of labels. In such applications, there needs to be a consistent policy for directing KT requests to the correct transparency log. The two primary cases where partitioning is necessary are high-traffic applications that require horizontal scaling, and federated applications. In a high-traffic application, many servers that are owned and operated by a single entity may need to cooperate to handle the total load of the application. In this case, requests would typically be routed to a transparency log based on a consistent hash of the label. Changing the set of transparency logs may change which transparency log is responsible for a given label; any such labels need to be moved between transparency logs as described in or . In a federated application, many servers that are owned and operated by different entities cooperate to provide a single end-to-end encrypted communication service. Each entity provides its own infrastructure (in particular, a transparency log) to serve the users that rely on it. Typically, the end-user identity directly specifies which entity requests should be directed to. For example, with an email end-user identity like alice@example.com, the controlling entity is example.com, indicating that requests related to this user should be directed to the servers of example.com. In a federated application, a controlling entity like example.com can act as an anonymizing proxy for its users when they query transparency logs run by other entities (in the manner of ), but should typically not attempt to 'mirror' or combine other transparency logs with its own. Doing so may prevent the other transparency logs from consistently enforcing their access control policies as intended, managing compliance with privacy laws, or modifying label values quickly and without interference from third-party caches. An entity that mirrors or combines another transparency log needs to ensure that these problems are either resolved, for example by coordinating with the other transparency log's operator, or acceptable for the application.

Pruning As part of the core infrastructure of an end-to-end encrypted communication service, transparency logs are required to operate seamlessly for several years. This presents a problem for general append-only logs, as even moderate usage can cause the log to grow to an unmanageable size in that time frame. This issue is further compounded by the fact that a substantial portion of the entries added to a log may be fake, having been added solely for the purpose of obscuring short-term update rates (discussed in ). Given this, transparency logs need to be able to manage their footprint by pruning data which is no longer required by the communication service. Broadly speaking, a transparency log's database will contain two types of data:

Serialized user data (the values corresponding to labels in the log), and
Cryptographic data, such as pre-computed portions of hash trees or commitment openings (values are stored in the log as cryptographic commitments).

The first type, serialized user data, can be pruned by removing data that has either expired or has become permanently inaccessible due to the service operator's access control policy. A transparency log may be configured with a maximum lifetime for log entries. Once a log entry is older than the maximum lifetime, it is considered expired and users no longer require proofs involving it. A version of a label expires when it is no longer possible to produce a valid search proof for the label-version pair, which happens when all of the necessary log entries have expired. This is specified in more detail in . Notably, the most recent version of a label is the only version that never expires through the maximum lifetime mechanism. However, service operators may define arbitrary access control policies that permanently block access to the most recent version (or any other version) of a label. The values corresponding to these label-version pairs may be deleted without consideration for the rest of the protocol. The second type of data, cryptographic data, can also be pruned, but only after considering which parts are no longer required by the protocol for producing proofs. For example, even though a particular version of a label may have been deleted, data corresponding to it may still need to exist in the latest version of the transparency log's prefix tree to produce valid search proofs for other versions of the label. The exact mechanism for determining which data is safe to delete will depend on the protocol and implementation. The distinction between user data and cryptographic data provides a valuable separation of concerns since KT does not provide a way for a service operator to convey its access control policy to a transparency log. That is, it allows the pruning of user data to be done entirely by application-defined code, while the pruning of cryptographic data can be done entirely by KT-specific code as a subsequent operation.

Security Considerations A user that correctly verifies a proof from the transparency log and does any required monitoring afterwards receives a guarantee that the transparency log operator executed the operation correctly, and in a way that's globally consistent with what it has shown all other users. That is, when a user searches for a label, they're guaranteed that the result they receive represents the same result that any other user searching for the same label at roughly the same time would have seen. When a user modifies a label, they're guaranteed that other users will see the modification within a bounded amount of time. Note that KT does not itself determine whether a new version of a label was authorized by the label's owner. Instead, it guarantees that any version shown to other users is also visible to the label's owner, who decides whether the change is authorized when monitoring their label. As a result, detecting unauthorized changes depends on the label owner performing regular monitoring. If the owner is offline for an extended period, such as because their account is dormant or abandoned, or if they lose their local state (see ), an unauthorized change may not be detected until the owner returns, or may not be detected at all. In particular, if the owner is offline for longer than the maximum lifetime of log entries, an unauthorized version that was inserted and then replaced while the owner was offline may expire before the owner is able to see it.

Transparency Log Misbehavior If the transparency log does not execute an operation correctly, then one of the following occurs:

The user detects the error immediately and rejects the proof.
The user permanently enters an invalid state. That is, the user accepts a result that is inconsistent with what the transparency log has shown other users, such as a forked view of the log or a version of a label that the transparency log later attempts to remove. Since users require all subsequent queries to prove consistency with previous ones, the transparency log is unable to reconcile the user's view with that of other users. This is intentional, as it ensures the misbehavior is eventually detected, either by background monitoring or by the mechanisms described in . Importantly, this means that users must stay online for a bounded amount of time after entering an invalid state for it to be successfully detected.
Instead of executing a query incorrectly, the transparency log can attempt to prevent the user from learning about more recent states of the log. This would allow the log to continue executing queries correctly, but on stale versions of data. To prevent this, applications configure an upper bound on how stale a query response can be without being rejected.

The security of a transparency log depends naturally on the security of the cryptographic primitives that it's configured to use, as well as the deployment mode that the transparency log relies on:

Third-Party Management and Third-Party Auditing require an assumption that the service operator and the third-party manager or auditor do not collude to trick users into accepting malicious results.
Contact Monitoring requires an assumption that all users that look up a label complete any necessary monitoring afterwards.

In short, assuming that the underlying cryptographic primitives used are secure, any deployment-specific assumptions hold (such as non-collusion), and that a label owner's devices don't go offline for longer than the maximum lifetime of log entries, then any attempt by the transparency log to present inconsistent views of a label to different users is always detected within a bounded amount of time. The parameters that determine the maximum amount of time before such misbehavior is detected are as follows:

The configured maximum amount of time by which a query response can be stale.
The configured Reasonable Monitoring Window (see ), weighed against how frequently users execute background monitoring in practice.
For logs that use the Contact Monitoring deployment mode: how frequently users engage in out-of-band communication.
For logs that use the Third-Party Auditing deployment mode: the configured maximum acceptable lag for an auditor.
For logs that use the Third-Party Management deployment mode: the amount of lag, or potential inefficacy, of the service operator's approach to detecting forks.

Ultimately, a transparency log can always refuse to answer requests that would reveal its own misbehavior. This includes refusing direct requests, requests made over an anonymous channel, and requests for the specific tree heads users need to efficiently gossip peer-to-peer. Similarly, a transparency log can refuse to apply updates to a user's own label, such as replacing a compromised public key. A user's persistent inability to perform monitoring, engage in out-of-band communication, or update their own label is indistinguishable from transparency log misbehavior and should be reported to the user as such.

State Loss The security of KT often depends on the ability of users to maintain robust local state. Users that lose their state in a Contact Monitoring or Third-Party Auditing deployment will have a correspondingly reduced ability to detect if they were shown a fork, or if the transparency log later obscured data that they consumed. State loss can occur, for example, when a user's device is lost, replaced, or reset, when the application is reinstalled or its data is cleared, or when the user begins using a new device without transferring state from an old one. In a Contact Monitoring deployment mode, state loss reduces a user's ability to detect misbehavior if it occurs after the user consumes a version of a label that was created either within the Reasonable Monitoring Window, or within a portion of the log that was insufficiently gossiped. In a Third-Party Auditing deployment mode, the same is true for a version of a label that was created within the auditor's maximum acceptable lag. Applications should consider the nature of possible state loss in their clients when configuring a transparency log and ensure that the relevant protocol parameters are set in a way that appropriately mitigates this risk. For example, in a Contact Monitoring deployment, the Reasonable Monitoring Window and the duration between out-of-band communication attempts should be much less than the typical time between state loss events. Similarly, in a Third-Party Auditing deployment, the maximum acceptable lag for an auditor should be much less than the typical time between state loss events. This minimizes the likelihood that a state loss event could be useful to a misbehaving transparency log. In applications where client state is typically ephemeral (like a web page), or where state loss could possibly be triggered adversarially (for example, by the service operator forcing a user to log out and re-register), a Third-Party Management deployment mode is recommended. Alternatively, applications could also consider implementing a policy of not consuming label-version pairs that were inserted too recently. Once a label-version pair is outside of the Reasonable Monitoring Window in a Contact Monitoring deployment, or beyond the maximum acceptable auditor lag in a Third-Party Auditing deployment, the risks associated with state loss are often already sufficiently mitigated.

Client Integrity The security of KT depends on several assumptions about the clients that users run:

Clients are able to execute code securely. That is, the device and operating system that a client runs on do not allow an attacker to tamper with the client's execution or local state.
Clients are executing a secure implementation of the protocol.
Clients have an authentic copy of the transparency log's configuration.
Clients have accurate local clocks.

The first assumption is common to all end-to-end encrypted systems, as a compromised device can generally access a user’s private keys and plaintext directly. To mitigate the risk of compromised devices, applications can rely on platform security features such as verified boot, application sandboxing, and hardware-backed key storage. Since KT requires clients to retain local state, applications should also consider protecting this state against modification and against being rolled back to an previous version. The second and third assumption are particularly important to address, since client software and the transparency log's configuration are usually distributed by the service operator, which is the same party that KT is intended to protect against. A service operator that is able to distribute modified client software or configuration to a user can defeat KT entirely for that user. For example, it could disable proof verification or monitoring, or replace the public key of the transparency log or a third party with one that it controls. To address this, applications can rely on a mechanism such as Binary Transparency. Binary Transparency systems require each released version of a piece of software to be published in a public, append-only log before clients will install it. This ensures that a modified version of the software cannot be distributed to specific users without also being visible to everyone else. Making client software open source and reproducibly built additionally allows outside parties to inspect each published version and verify that it implements the protocol correctly and contains the correct configuration. Finally, a client with an inaccurate clock may accept a stale view of the log that would otherwise be rejected. Applications can ensure that clients have an accurate clock with a protocol like Roughtime , which provides authenticated time from multiple independent servers and allows clients to detect servers that provide incorrect time.

Privacy Considerations

Access Control For applications deploying KT, service operators expect to be able to control when sensitive information is revealed. In particular, a service operator may need to restrict who is able to learn that a user is a member of their service, or information about that user's account. KT only allows a user to learn whether or not a label exists in the transparency log if the user obtains a valid search proof for that label. Similarly, KT only allows a user to learn about the value of a label if the user obtains a valid search proof for that exact version of the label. When an application's access control policy previously permitted a user to look up or change a label's value but no longer does, KT prevents the user from learning whether or not the label's value has changed since the user's access was revoked. This is true even in Contact Monitoring mode, where users are still permitted to perform monitoring after their access to perform other queries has been revoked. Applications determine the privacy of data in KT by relying on these properties when they enforce access control policies on the queries issued by users, as discussed in . For example, an application's access control policy may only allow users to search for the labels of their contacts, or of users they are actively starting a conversation with. This prevents users that are not contacts or actively communicating from learning about each other's existence. If two users were previously contacts but no longer are, the application can prevent the users from searching for each other's labels and learning whether there have been any subsequent account updates.

Population-Level Metrics Service operators also expect to be able to control sensitive population-level metrics about their users. These metrics include the size of their user base, the frequency with which new users join, and the frequency with which existing users update their labels. KT allows a service operator to obscure the size of its user base by batch inserting a large number of fake label-version pairs when a transparency log is first initialized. Similarly, KT also allows a service operator to obscure the rate at which "real" changes are made to the transparency log by padding "real" changes with the insertion of other fake label-version pairs, such that it creates the outside appearance of a constant baseline rate of insertions.

Leakage to Third Parties In the event that a third-party manager or auditor is used, there's additional information leaked to the third party that's not visible to outsiders. In the case of a third-party auditor, the auditor is able to learn the total number of distinct changes to the log. It is also able to learn the order and approximate timing with which each change was made. However, auditors are not able to learn the plaintext of any labels or values. This is because labels are masked with a Verifiable Random Function (VRF) , and values are only provided to auditors as commitments. They are also not able to distinguish whether a change represents a label being created for the first time or being updated, or whether a change represents a "real" change from an end-user or a "fake" padding change. In the case of a third-party manager, the manager generally learns everything that the service operator would know. This includes the total set of plaintext labels and values and their modification history. It also includes traffic patterns, such as how often a specific label is looked up.

Right to Erasure Consumer privacy laws often provide a right to erasure. This means that when a consumer requests that a service operator delete their personal information, the service operator is legally obligated to do so. This may seem to be incompatible with the description of KT in as an "append-only log". Once an entry is added to a transparency log, it indeed cannot be removed. The important caveat here is that user data is not directly stored in the append-only log. Instead, it primarily contains privacy-preserving VRF outputs and cryptographic commitments. The value corresponding to a label is typically some serialized user account data, like a public key or internal identifier. KT uses cryptographic commitments to ensure that users interacting with a transparency log are unable to learn anything about a label's value until the transparency log explicitly provides the commitment's opening. A transparency log responding to an erasure request can delete the commitment opening and the associated data. This can be done immediately and permanently prevents recovery of the associated value. Labels themselves are typically serialized end-user identifiers, like a username or email address. All labels are processed through a VRF, which uses a private key to deterministically map each label to a fixed-length pseudorandom value. The set of all labels stored in a transparency log is committed to by a prefix tree, and each version of the prefix tree is committed to by a log tree. While all the VRF outputs corresponding to labels affected by an erasure request can be deleted from the most recent version of the prefix tree immediately, previous versions of the prefix tree will still contain the VRF outputs and will still be needed by the protocol for a bounded amount of time. This bound is determined by the maximum lifetime of log entries (see ). As such, the VRF outputs can only be fully purged from the transparency log once all log entries that contain them have passed their maximum lifetime.

Implementation Guidance Fundamentally, KT can be thought of as guaranteeing that all the users of a service agree on the contents of a key-value database (noting that this document refers to these keys as "labels"). It takes special care to turn the guarantee that all users agree on a set of labels and values into a guarantee that the mapping between end-users and their public keys is authentic. Critically, to authenticate an end-user identity, it must be both unique and user-visible. However, what exactly constitutes a unique and user-visible identifier varies greatly from application to application. Consider, for example, a communication service where users are uniquely identified by a fixed username, but KT has been deployed using each user's internal ID as their label. While the ID might be unique, it is not necessarily user-visible if, when a user attempts to look up a contact by username, the service operator translates the username into a user ID under the hood. If this mapping (from username to ID) is unauthenticated, the service operator can manipulate it to eavesdrop on conversations by returning the ID for an account that it controls. From a security perspective, this would be equivalent to not using KT at all. However, that's not to say that the use of internal IDs in KT is never appropriate. Many applications don't have a concept of a fixed explicit identifier, like a username, and instead rely on their UI (underpinned internally by a user's ID) to indicate to users whether a conversation is with a new person or someone they've previously contacted. The fact that the UI behaves this way makes the user's ID a user-visible identifier even if a user may not be able to actually see it written out. An example of an application with these identifier semantics, although not one that provides end-to-end encryption, would be Slack. A primary end-user identity is one that is unique, user-visible, and unable to change. (Or equivalently, if it changes, it appears in the application UI as a new conversation with a new user.) An example of this type of identity would be an email address on an email service. A primary end-user identity should always be a label in KT, with the end-user's public keys and other account information as the associated value. A secondary end-user identity is one that is unique and user-visible, and that can change without being interpreted as a different account because it is associated with a primary end-user identity. Secondary end-user identities are used solely for initial user discovery, during which they're converted into a primary end-user identity that the application uses to identify the end-user from then on. For example, an application may allow users to change their username without disrupting existing conversations because each username is associated with an internal user ID. The username is a secondary end-user identity, and the internal user ID is the primary end-user identity. The internal user ID is user-visible by virtue of the fact that existing conversations with the user continue uninterrupted when their username changes. A secondary end-user identity should be a label in KT with the primary end-user identity as its associated value, so that the user discovery process is authenticated. While likely helpful to most common applications, the distinction between handling primary and secondary end-user identities is not a guaranteed rule. Applications must be careful to ensure they fully capture the semantics of identity in their application with the labels and values they store in KT. Relatedly, while KT authenticates the mapping from a label to its value, it does not help users distinguish between labels that appear similar. For example, two usernames that differ only in visually similar characters are distinct labels to KT, but may not be distinct to a user. Applications should take care to prevent confusable identifiers from being registered.

IANA Considerations This document has no IANA actions.