JMAP Object History

Introduction JMAP ( — JSON Meta Application Protocol) provides a standard set of methods for each data type: Foo/get, Foo/changes, Foo/set, Foo/query, and Foo/queryChanges. These methods operate on the current state of objects. The Foo/changes method tells a client which objects have changed, but does not provide the previous values. In many applications it is useful to access historical versions of objects. For example, a user may want to see previous versions of a contact record () or recover a deleted email (). This extension adds history support to the standard Foo/get method. When the urn:ietf:params:jmap:object-history capability is included in the request's using array, Foo/get gains additional parameters that allow the client to request previous versions of objects, including objects that have been destroyed, Foo/query can search the destroyed objects, and Foo/set can restore them.

Conventions Used in This Document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Addition to the Capabilities Object The capabilities object is returned as part of the JMAP Session object; see , Section 2. This document defines an additional capability URI.

urn:ietf:params:jmap:object-history The capability urn:ietf:params:jmap:object-history being present in the "accountCapabilities" property of an account represents support for retrieving historical object versions in that account. Servers that include the capability in one or more "accountCapabilities" properties MUST also include the property in the "capabilities" property. The value of this property in the JMAP session "capabilities" property MUST be an empty object. The value of this property in an account's "accountCapabilities" property is an object that MUST contain the following information on server capabilities for that account: maxHistoryDuration: "UnsignedInt|null" The number of seconds after replacement for which the server intends to retain old versions, or null if the server does not have a time-based retention policy. This is a statement of server policy, not a guarantee: retention is best effort, and the server MAY discard versions sooner (for example, due to storage limits). A version whose replaced timestamp is more than this many seconds ago has probably been discarded. Clients that need to be certain an object remains available should not destroy it. supportedTypes: "String[]" The names of the data types, from the "JMAP Data Types" registry or private types, for which the server supports the history extensions in this account. A server MAY support history for only a subset of the types it supports, and MAY retain history for only some objects of a supported type (for example, only for some folders). Clients MUST ignore type names they do not recognise.

Extensions to Foo/get When this capability is included in the request's using array, the standard Foo/get method (, Section 5.1) accepts the following additional request arguments: includeReplaced: "Boolean" (default: false) If true, the server returns previous versions of the requested objects in addition to the current version. Each previous version appears as a separate entry in the list array, with the same id as the current version. Every entry includes an objectHistory property (see below). Entries with the same id MUST be ordered by objectHistory.version (ascending). Entries for different objects MAY be interleaved. includeDestroyed: "Boolean" (default: false) If true, the server also returns objects that have been destroyed. If includeReplaced is false, only the final version before destruction is returned. If includeReplaced is true, all retained historical versions of destroyed objects are returned. For destroyed objects, every entry has a non-null objectHistory.replaced value, since every version has been superseded (the final version was replaced by destruction). An object counts as destroyed however it was removed: by Foo/set, through another protocol (for example an IMAP EXPUNGE or a POP3 DELE ), or by the server itself, for example when it empties a Trash mailbox automatically or destroys a Mailbox with onDestroyRemoveEmails (, Section 2.5). Likewise, changes made through other protocols can create versions just as changes made through JMAP do. historyAfter: "UTCDate|null" (default: null) If supplied, only versions with an objectHistory.replaced value strictly after this date are returned, plus the current (live) version which has a null replaced value. If null, no time-based filter is applied. historyBefore: "UTCDate|null" (default: null) If supplied, only versions with an objectHistory.replaced value strictly before this date are returned. The current (live) version, which has a null replaced value, is not returned. If null, no time-based filter is applied. A client can use this to page backwards through history: it passes the oldest replaced value from the previous response as historyBefore in the next request. Since historyBefore applies to every id in the call, a client paging through the history of several ids makes a separate Foo/get call for each id. Clients SHOULD pass the replaced value exactly as received, rather than parsing and re-serialising it, since re-serialising at a lower precision could cause versions to be skipped. historySkip: "UnsignedInt" (default: 0) The number of entries to skip for each id, counting from the most recent (the highest version number), before any entries are returned. The current version, if it would otherwise be returned, counts as one entry. For example, a client that wants only versions older than the 20 most recent passes historySkip: 20; any id with 20 or fewer matching entries is returned in notFound. A client can also use this to page backwards through history, by adding the number of entries returned for an id to historySkip in the next request. This is less robust than historyBefore: if a new version is recorded between requests, the window shifts and one entry is returned again. historyLimit: "UnsignedInt|null" (default: null) If supplied, the maximum number of entries to return in the list array for each id, including the current version if any. For each id, the server MUST return the most recent entries (those with the highest version numbers) remaining after historySkip is applied, not the oldest. If null, the server returns all remaining retained versions, subject to its own limits. The server applies includeReplaced, includeDestroyed, historyAfter, and historyBefore to select the matching versions of each id, then skips historySkip entries, then returns at most historyLimit of the remaining entries. If includeReplaced and includeDestroyed are both false, the server ignores historyAfter, historyBefore, historySkip, and historyLimit. If the ids argument is null, the server returns all objects of the type as per , Section 5.1. If includeDestroyed is true, this includes every destroyed object for which the server still retains any version. The "maxObjectsInGet" limit and the "requestTooLarge" error (, Section 5.1) apply to the number of distinct ids, not to the number of entries in the list array. Each id can have up to historyLimit entries. If, after applying includeReplaced, includeDestroyed, historyAfter, historyBefore, and historySkip, no versions of a requested id remain to be returned, that id is included in the notFound array.

The objectHistory property When includeReplaced or includeDestroyed is true, each object in the list array includes an additional property: objectHistory: "ObjectHistoryInfo" An ObjectHistoryInfo object with the following properties: version: "UnsignedInt" A per-object monotonically increasing number that provides ordering among the versions of a single object within a response. Version numbers MAY have gaps (e.g., a server may use internal sequence numbers such as modseq values). The current (live) version of an object always has the highest version number. Servers SHOULD return consistent version numbers across requests, but clients MUST NOT rely on this — version numbers exist primarily to provide ordering within a single response. replaced: "UTCDate|null" The date and time at which this version was superseded, either by a subsequent update or by destruction. If null, this is the current live version of the object. For destroyed objects, every version has a non-null replaced value — the final version's replaced is the time of destruction. The replaced values of different versions of the same object MUST be distinct, so that historyBefore and historyAfter partition the history without ambiguity; a server that records two versions with the same timestamp can collapse them into a single version. The value MAY include fractional seconds (, Section 1.4), but servers SHOULD NOT use more than millisecond precision, as many client date libraries cannot represent more. hasOlder: "Boolean" If true, the server retains at least one older version of this object than this one, whether or not that version was returned in this response. A client can check hasOlder on the oldest entry returned for an id to learn whether more history is available, and retrieve it using historyBefore or historySkip. The server MAY return false even if older versions exist (e.g., if it cannot efficiently determine whether older versions remain). If the server returns true, the client MUST NOT assume the older versions will still be available in a subsequent request, as the server may purge history at any time. expires: "UTCDate|null" The time after which the server expects to discard this version, or null if it has no planned time. The current (live) version of an object always has a null expires. Like maxHistoryDuration, this is a best-effort statement and not a guarantee: the server MAY discard the version sooner, for example when storage runs short, and MAY return a different value in a later response. A client can use this to tell the user how long a destroyed object can still be recovered, but MUST NOT rely on the version remaining available until then. When includeReplaced and includeDestroyed are both false (the default), the objectHistory property is not included in the response, and the behaviour is identical to standard Foo/get. The client MAY include "objectHistory" in the properties argument. It is returned if and only if includeReplaced or includeDestroyed is true, regardless of whether it is listed in properties. Which changes produce a version is server dependent: a version is a snapshot of the object that the server recorded at some point. The server is not required to create a new version for every individual change to an object. For example, rapid successive updates may be collapsed into a single version, and a server may choose not to record a version for changes that only affect server-computed properties. The server MAY also prune old versions in any order, so there can be gaps in the version history available to the client. Clients MUST NOT assume that the version history is complete or contiguous. The values of server-computed properties (for example, the counts on a Mailbox) in a historical version are also server dependent. The server SHOULD return the value the property had at the time of that version if it has that information. If a historical version contains a reference to a blob (for example, the blobId of an Email), that blob MUST be downloadable for as long as the server returns that version. A version that is not current does not count as a reference to the blob, for garbage collection (, Section 6) or for any other purpose, such as the "blobHasReference" error and Blob/lookup method of . A server that discards a blob referenced by a version MUST NOT return that version. If includeReplaced or includeDestroyed is true and the data type is not listed in the supportedTypes capability property, the server MUST reject the call with an "invalidArguments" error. If the data type is supported but the server does not retain history for a particular object, it MUST still return the current version with an objectHistory property when requested. In this case, the server returns only the current version, with a replaced of null.

Examples

Contact history Requesting the history of a contact, including all versions:

The response includes three versions: the original creation, a name update, and the current version with an additional email address:

Recovering destroyed emails A client can use Email/changes to discover which emails have been destroyed, then use a backreference to fetch their final state. This is useful for displaying a "recently deleted" view:

The Email/get response returns the final state of each destroyed email. The objectHistory.replaced value is the time of destruction:

Since includeReplaced was not set, only the final version of each destroyed email is returned. The non-null replaced value indicates when each email was destroyed. The server in this example keeps history for 30 days.

Extensions to Foo/query The Foo/changes approach in only works for a client that holds a state string from before the objects were destroyed, and only while the server can still calculate changes from that state. Foo/get with a null ids argument and includeDestroyed returns every retained object of the type at once, with no filtering, sorting or paging. When this capability is included in the request's using array, the standard Foo/query method (, Section 5.5) accepts the following additional request arguments: onlyDestroyed: "Boolean" (default: false) If true, the query runs over the destroyed objects of this type for which the server still retains at least one version, instead of over current objects. An object that has been restored () is a current object again, and is not included. destroyedAfter: "UTCDate|null" (default: null) If supplied, only objects destroyed strictly after this date are included. destroyedBefore: "UTCDate|null" (default: null) If supplied, only objects destroyed strictly before this date are included. The time an object was destroyed is the replaced value of its final version. The server MUST reject the call with an "invalidArguments" error if onlyDestroyed is true and the type is not listed in supportedTypes, or if destroyedAfter or destroyedBefore is supplied and onlyDestroyed is false. When onlyDestroyed is true: The filter and sort arguments apply to the final version of each destroyed object. The server need not support every filter condition or comparator for destroyed objects that it supports for current objects. It rejects one it does not support with "unsupportedFilter" or "unsupportedSort", as usual. If sort is null or empty, the results are ordered by time of destruction, most recent first. Arguments that apply only to current objects, such as collapseThreads for Email, are ignored. Foo/queryChanges works as for any other query. An object leaves the results when it is restored or discarded, and enters them when it is destroyed. A server that cannot track these changes MAY return a "cannotCalculateChanges" error. The ids returned can then be passed to Foo/get with includeDestroyed set, usually with a backreference.

Example A client builds a "recently deleted" view of the 50 emails destroyed most recently since 1 March:

The Email/query response lists the destroyed emails, most recently destroyed first:

The Email/get response is the same as in .

Restoring Objects To restore an earlier version of an existing object, the client uses a normal Foo/set update with the values from that version. To restore a destroyed object, the client uses the restore argument of Foo/set.

Extensions to Foo/set When this capability is included in the request's using array, the standard Foo/set method (, Section 5.3) accepts the following additional request argument: restore: "Id[PatchObject|null]|null" (default: null) A map of the ids of destroyed objects to restore. Each value is null, or a PatchObject to apply to the restored object. An empty PatchObject means the same as null. The response has the following additional properties: restored: "Id[Foo|null]|null" A map of the ids of objects that were successfully restored. The value for each is an object containing any properties of the restored object whose values differ from those in the final version with the patch applied, or null if there are none. This is null if no objects were restored. notRestored: "Id[SetError]|null" A map of the ids of objects that could not be restored to a SetError object explaining why. This is null if every restore succeeded or none was requested. The server MUST reject the call with an "invalidArguments" error if restore is non-null and the type is not listed in supportedTypes. The server processes each restore before any create, update, or destroy in the same call, so these can refer to a restored object. The restored object takes the id of the destroyed object, and the values of the final version of that object (), with the PatchObject applied. The patch follows the same rules as a PatchObject in an update in Foo/set. This includes server-set properties, such as the threadId and size of an Email. These keep their values from the final version, unless the server cannot keep them, for example if the Thread no longer exists; the server then sets new values and returns them in restored. The history of the restored object continues from that of the destroyed object, so the destroyed versions become previous versions of the restored object. To restore a version other than the final one, the client includes the values of that version in the PatchObject. The server rejects a restore with a SetError as follows: notFound: The id is not the id of a destroyed object of this type for which the server retains any history. This includes the id of a current object. invalidPatch: The PatchObject is not valid. invalidProperties: The restored object would not be valid, as for an update in Foo/set. forbidden: The server does not allow this object to be restored, for example because of its content (). overQuota: The restored object would exceed a quota. The server MAY also return any other SetError that Foo/set returns for an update or create of this type. The server MUST NOT reuse the id of a destroyed object for which it retains any history, other than by restoring that object. Foo/changes reports a restored id in updated if the object existed at sinceState, and in created otherwise. It never reports a restored id in destroyed.

Type-Specific Notes A restored Email is the original message: it has the same blobId, and so the same content, as the final version. A client can restore an Email with Email/set without fetching its content first. A SieveScript is deactivated before it is destroyed (, Section 2.4), so a restored SieveScript is not active. The client can activate it with onSuccessActivateScript, using its id. The final version may refer to an object that no longer exists, such as a Mailbox in an Email's mailboxIds. The server then rejects the restore as it would an update that refers to a nonexistent object. The client can restore that object first, in the same call or an earlier one, since Mailbox, AddressBook, and Calendar can have history too. Or it can choose another one in the PatchObject, for example the Mailbox with the "inbox" role. The server can also reject a restore that conflicts with a current object, using the error that Foo/set would normally return; for example, a SieveScript whose name is now used by another script is rejected with "alreadyExists" (, Section 2.4).

Example Having found email42 with Email/query (see ), the client restores it. The Mailbox it was in has since been destroyed, so the client puts it in the Inbox (mb1):

The server restores the Email. It has kept its threadId, so no properties are returned:

Design Rationale This section is not normative. History is returned through the existing Foo/get method rather than a new method. The entries in the list array are ordinary objects of the data type, so a client can feed them to the same parser it uses for current objects, and the existing properties argument works unchanged. It also allows a client to fetch the final state of many destroyed objects in a single call, using a backreference to the destroyed list from Foo/changes, as shown in the examples above. The history metadata is embedded in each object as the objectHistory property, rather than wrapping each object in a separate container, for the same reason: the entries remain ordinary objects. This follows the existing pattern of JMAP-specific properties such as id being added to objects whose format is defined elsewhere; the objectHistory name is reserved in the relevant property registries (see ). Each entry is a full copy of the object rather than a patch against the previous version. Patches would save space, but the client would need every intermediate version to reconstruct any one of them, which conflicts with the server's freedom to prune versions, and the final version of a destroyed object would need to be returned in full anyway. Every superseded version carries a replaced timestamp. The server always knows, at least approximately, when it replaced a snapshot it was keeping, and the timestamp is what a client needs to present history to a user and to filter it with historyAfter. Destroyed objects are restored with a new restore argument to Foo/set rather than by creating an object with the old id. A create in Foo/set cannot set the id or server-set properties, so the restored object would differ from the original; an Email created with Email/set is a new message, not the original one. Restoring the final version, with a patch, brings back the object exactly, and the client need not send the whole object back to the server. Destroyed objects are found with Foo/query rather than a new method, for the same reasons: the ids it returns go straight into Foo/get with includeDestroyed. The query runs over destroyed objects only, rather than over current and destroyed objects together, because "recently deleted" is normally a separate view. It also lets a server keep destroyed objects in a separate index.

IANA Considerations

JMAP Capability Registration for urn:ietf:params:jmap:object-history IANA is requested to register the "Object History" Capability as follows: Capability Name: urn:ietf:params:jmap:object-history Intended use: common Change Controller: IETF Specification document: this document Security and privacy considerations: this document, Security Considerations

Update to the JMAP Data Types Registry This document adds a "Supports History" column to the "JMAP Data Types" registry . The value is "Yes" or "No", indicating whether the data type supports the history extensions defined in this document. The initial values for existing registrations are: Type Name Supports History Core No PushSubscription No Mailbox Yes Thread No Email Yes EmailDelivery No SearchSnippet No Identity Yes EmailSubmission No VacationResponse Yes MDN No Quota No SieveScript Yes Principal No ShareNotification No AddressBook Yes ContactCard Yes Calendar Yes CalendarEvent Yes CalendarEventNotification No ParticipantIdentity Yes FileNode Yes FileNode is registered by . The default value for this column is "No". New registrations that do not explicitly include a value are assumed not to support history. This column records whether history is meaningful for a data type. Whether a particular server supports history for a type is advertised in the supportedTypes capability property.

JSContact Property Registration IANA is requested to register the following property in the "JSContact Properties" registry defined in , Section 3, so that the name cannot be used for a property of a JSContact Card. Property Name: objectHistory Property Type: not applicable Property Context: Card Intended Usage: reserved Since Version: 1.0 Change Controller: IETF Reference: this document

JSCalendar Property Registration IANA is requested to register the following property in the "JSCalendar Properties" registry defined in , Section 8.2, so that the name cannot be used for a property of a JSCalendar object. Property Name: objectHistory Property Type: not applicable Property Context: Event, Task, Group Intended Usage: reserved Change Controller: IETF Reference: this document Specifications that define a JMAP data type whose object format is also governed by a separate property registry SHOULD reserve the objectHistory property name in that registry.

Security Considerations All security considerations from apply to this document.

Storage and Resource Consumption Retaining historical versions of objects can consume significant server storage. Servers SHOULD impose reasonable limits on history retention, either by duration (advertised via maxHistoryDuration) or by total storage. Servers MAY silently discard history entries when storage limits are reached. Counting retained history towards a user's quota () is NOT RECOMMENDED. If it counts, a user who destroys objects to free space sees no change until the history expires. A restored object counts towards quota as any newly created object does.

Access Control Historical versions of an object MUST respect the same access controls as the current version. A user who does not have permission to read an object MUST NOT be able to read its history. Access rights can change over time, for example when an object is shared with a new user, or moved into a shared container. The server MUST NOT return a historical version to a user who could not have read that version at the time it was current. This requires the server to track changes to access rights. A server that does not keep a full history of access rights can take a conservative approach: record the point at which the access rights affecting an object (or its containing object, such as a Mailbox or AddressBook) last changed, for example as a modseq, and not return any version that was replaced before that point. Similarly, for destroyed objects, the server MUST NOT reveal the existence or content of a destroyed object to a user who could not read it at the time it was destroyed.

Information Disclosure Object history may reveal information that the user or an administrator intended to remove. For example, a contact record that was updated to remove a phone number will still have that phone number visible in its history. Servers SHOULD provide administrators with the ability to purge history for specific objects when required for privacy or compliance reasons.

Privacy Considerations When a server supports this extension, updating or destroying an object does not immediately remove the previous content from the server; it remains retrievable by anyone with read access to the object until the server discards it under its retention policy. Users may not expect this. This is not new in kind (servers commonly retain deleted data in backups, and many mail systems offer a "recently deleted" view), but this extension makes such data retrievable through the protocol. Servers SHOULD make their retention policy visible to users, for example through the maxHistoryDuration capability property and the server's user interface. Clients that present a "destroy" operation to users SHOULD NOT describe it as irreversible or permanent when the server advertises this capability for that data type. Whether, and how, a user can request that history be purged before the end of the retention period is a matter of server policy and is outside the scope of this document.

Harmful Content An administrator may remove content from an account because it is harmful, for example a phishing message or malware. The server MAY decline to retain history for such an object, and MAY reject an attempt to restore it with a "forbidden" SetError, so that the user cannot bring the content back.

Changes EDITOR: please remove this section before publication. The source of this document exists on github at: https://github.com/brong/draft-gondwana-jmap-object-history/ draft-ietf-jmap-object-history-03 Added John Coffey as co-author; folded in material from draft-coffey-jmap-undelete. Added Foo/query arguments onlyDestroyed, destroyedAfter, and destroyedBefore, for finding destroyed objects without an old state string. Replaced restoring by creating with the old id (and Email/import with the old id) with a new Foo/set restore argument, which restores the final version with an optional patch, keeping server-set properties. Notes on Email, SieveScript, missing containers, and conflicts. Added a per-version expires to objectHistory. Destruction through other protocols and by the server counts as destruction. Security: quota, harmful content. Paging several ids with historyBefore takes one Foo/get call per id. The history arguments are ignored unless includeReplaced or includeDestroyed is true. Foo/changes reports a restored id as updated or created, never as destroyed. Versions that are not current do not count as blob references. Added FileNode to the "Supports History" table. Thanks to Mark Atwood for an implementation review. draft-ietf-jmap-object-history-02 Replaced the top-level hasMoreHistory response property with a per-entry objectHistory.hasOlder property, so the client can tell which ids have more history. Added historySkip, to skip the N most recent versions of each id; ids with N or fewer versions are returned in notFound. draft-ietf-jmap-object-history-01 Removed "Updates: 8620"; the JMAP Data Types registry is not defined by RFC 8620. The registry is now referenced by name rather than by its defining document. Corrected the registry table to match the current registry. historyLimit is now per id. Defined behaviour for ids: null with includeDestroyed, and clarified that "maxObjectsInGet" counts ids, not entries. Ids with no matching versions are returned in notFound. Which changes create a version, and the values of server-computed properties, are server dependent. Blobs referenced by returned versions must be downloadable. Expanded access control considerations for changing access rights. Added supportedTypes to the account capability; requesting history for an unsupported type is an error. Reserved the objectHistory property name in the JSContact and JSCalendar property registries. Added acknowledgements. Added a Design Rationale section (Foo/get rather than a new method, embedded metadata, full objects rather than patches, timestamps). Added a Restoring Objects section: a client may ask to recreate a destroyed object with its old id, and the server may accept or reject; ids of destroyed objects are otherwise not reused. maxHistoryDuration is now a best-effort statement of server policy. Clarified that "objectHistory" may appear in properties. Added Privacy Considerations. Added historyBefore for paging backwards through history; versions of the same object have distinct replaced values, with at most millisecond precision. Removed the audit use case from the introduction. draft-ietf-jmap-object-history-00 Uploaded with working-group adopted name. draft-gondwana-jmap-object-history-00 initial proposal

Acknowledgements The authors would like to thank Mark Atwood, Ben Bucksch, Mauro De Gennaro, Arnt Gulbrandsen, Neil Jenkins, and Robert Stepanek for their feedback on this document.