Internet Engineering Task Force G. Banks, Ed. Internet-Draft R. Mueller Updates: 3501 (if approved) B. Gondwana Intended status: Informational OSA Expires: October 31, 2015 April 29, 2015 Internet Message Access Protocol Version 4 - Extension for Conversations draft-banks-imap-conversations-00 Abstract The XCONVERSATIONS extension to the Internet Message Access Protocol allows messages to be linked into conversations, automatically efficiently and persistently, and entirely in the server. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on October 31, 2015. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Banks, et al. Expires October 31, 2015 [Page 1] Internet-Draft IMAP4 - Conversations April 2015 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Conversations . . . . . . . . . . . . . . . . . . . . . . 3 2.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3. CIDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.4. Conversation Properties . . . . . . . . . . . . . . . . . 5 2.5. Inserting a New Message . . . . . . . . . . . . . . . . . 6 2.6. Merging Conversations . . . . . . . . . . . . . . . . . . 6 3. IMAP Protocol Changes . . . . . . . . . . . . . . . . . . . . 8 3.1. General Considerations . . . . . . . . . . . . . . . . . . 8 3.2. Interaction with Message Delivery . . . . . . . . . . . . 8 3.3. Interaction with APPEND . . . . . . . . . . . . . . . . . 8 3.4. Interaction with COPY . . . . . . . . . . . . . . . . . . 8 3.5. Interaction with MOVE . . . . . . . . . . . . . . . . . . 8 3.6. Interaction with EXPUNGE . . . . . . . . . . . . . . . . . 9 3.7. Interaction with RENAME . . . . . . . . . . . . . . . . . 9 3.8. Interaction with STORE . . . . . . . . . . . . . . . . . . 9 3.9. New Data Items for FETCH . . . . . . . . . . . . . . . . . 9 3.10. New Criteria in SEARCH . . . . . . . . . . . . . . . . . . 9 3.11. New Keys in SORT . . . . . . . . . . . . . . . . . . . . . 9 3.12. New Data Items in STATUS . . . . . . . . . . . . . . . . . 9 3.13. New Command XCONVMETA . . . . . . . . . . . . . . . . . . 10 3.14. New Command XCONVFETCH . . . . . . . . . . . . . . . . . . 10 3.15. New Command XCONVSORT . . . . . . . . . . . . . . . . . . 10 3.16. New Command XCONVUPDATES . . . . . . . . . . . . . . . . . 11 4. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 11 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 14 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 7. Security Considerations . . . . . . . . . . . . . . . . . . . 14 8. Normative References . . . . . . . . . . . . . . . . . . . . . 14 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15 Banks, et al. Expires October 31, 2015 [Page 2] Internet-Draft IMAP4 - Conversations April 2015 1. Introduction A server advertising the XCONVERSATIONS capability tracks conversation (i.e. message threading) information automatically and persistently, and provides ways for a client to present to users both complete new views and efficient updates to existing views of conversations. 1.1. Requirements Language In examples, "C:" and "S:" indicate lines sent by the client and server, respectively. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. Data Model 2.1. Conversations Conversations are groups of messages automatically connected together by the server. In this sense they are like threads as defined by the THREAD command [RFC5256] or implemented by various modern clients, but with the following important differences. o Conversations are a flat unordered group without the tree structure of threads. o Conversations are cross-folder, so that e.g. related messages in a user's Inbox, Sent and Drafts folders can all be connected into one conversation. o Conversations are constructed entirely by the server, without client involvement, and in particular without the client needing to download or cache the headers of every single message. o Conversations are persistent and updated by the server at message delivery time, rather than when the client connects, so all the information is present without delay when the client connects. Conversations are intended for advanced clients to display directly to the user as the basic units of folder structure, i.e. a client MAY present a folder as a sequence of conversations. Conversations are not intended to replace the fundamental IMAP concept of folders entirely, but to augment them. Banks, et al. Expires October 31, 2015 [Page 3] Internet-Draft IMAP4 - Conversations April 2015 2.2. Scope Conversations are unique within some scope, known as the conversation scope. An IMAP folder is associated with at most one conversation scope. All messages in a given folder MUST be in the same conversation scope. That scope MUST NOT change while a folder's UIDVALIDITY does not change; if a folder's conversation scope changes the UIDVALIDITY MUST also change. A folder MAY have no scope at all, in which case all the messages in the folder MUST NOT belong to any conversation. A server MAY decide automatically to treat any particular folder this way, e.g. a Spam or Trash folder. A server SHOULD assign all of a user's folders to the same conversation scope, which is only available to that user. 2.3. CIDs Conversations are identified by Conversation IDs (CIDs). A CID uniquely identifies a conversation in a given conversation scope. A CID is lexically an or NIL, which represents no conversation at all. A server MUST NOT assign NIL as the ID of a conversation. CIDs are created by the server automatically, and are opaque to the client. The details of their format are not defined here. The client MUST NOT interpret CIDs in any way, except to compare them case-sensitively for uniqueness. In all other respects, CIDs are opaque tokens whose only purpose is to be passsed back to the server. In particular, clients MUST not make any assumptions about ordinality of CIDs. Each message has a property, formally a CID, which identifies the conversation to which the message is associated (or NIL if the message is not associated with any conversation). This property is automatically assigned by the server and is read-only to the client. No protocol mechanism is provided to change the CID. However, note that the CID property is mutable. A server MAY change the CID of a message under certain rare circumstances (e.g. when Merging Conversations (Section 2.6)), but doing so MUST increase the message's MODSEQ. Banks, et al. Expires October 31, 2015 [Page 4] Internet-Draft IMAP4 - Conversations April 2015 2.4. Conversation Properties In addition to being groups of messages, conversations are also first class objects with some properties of their own. These properties are maintained automatically by the server, and are read-only to the client. The client can query conversation properties using the XCONVMETA (Section 3.13) command. With the exception of MODSEQ, these properties are summaries of properties of the messages contained in the conversation, provided for the convenience of conversations-aware clients. A server MAY implement them dynamically rather than pre-calculating and storing them, although pre-calculating is recommended for best performance as visible to the user. Conversation properties include at least: MODSEQ Reflects the highest MODSEQ value of any message which has ever been associated with the conversation, including messages since deleted. This is analagous with the HIGHESTMODSEQ property of folders. Note that this means that the server MUST maintain the MODSEQ property persistently. EXISTS A count of the number of messages in the conversation. UNSEEN A count of the number of messages in the conversation which do not have the \Seen flag set. COUNT A set of (FLAG COUNT) pairs which count the number of messages in the conversation which have various flags (user or system flags) set. Which flags are so counted is not defined here. The server MAY choose any set of flags to count, but it SHOULD include \Draft and \Flagged flags. SENDERS A list of addresses, reported as broken-out tuples (NAME ROUTE MAILBOX DOMAIN), from which messages in the conversation have been sent. FOLDEREXISTS A list of folders containing messages which are in the conversation, with counts of the number of messages from this conversation in each folder. Reported as (MBOXNAME EXISTS) tuples. Banks, et al. Expires October 31, 2015 [Page 5] Internet-Draft IMAP4 - Conversations April 2015 2.5. Inserting a New Message When a message arrives in the server via any means, the server MUST associate the message with a conversation. This process includes the following steps. o The server parses the message to discover the values of certain header fields. o The server uses those values to look up zero or more existing conversations in the conversations scope of the message. The server SHOULD use the message ids in the Message-Id, In-Reply-To and References header fields and search for conversations with messages which list those ids in their Message-Id, In-Reply-To or References header fields. The server MAY also apply other criteria designed to make conversations more friendly to humans, such as matching the Subject header field, or limiting the length of conversations. o Depending on how many conversations were found, the server MUST take one of these actions to choose a single conversation. * If zero existing conversations were found, the server MUST create a new conversation and use that. * If one existing conversation was found, the server MUST use that conversation. * If two or more existing conversations were found, the server merges the conversations Section 2.6 to create a single resulting conversation and uses that. The server MUST set the message's CID attribute to the CID of the chosen conversation, and MUST update the conversation's SENDERS, FOLDERS, and counter information to account for the message. o At this point, the server MAY split the final conversation to enforce some server-specific limit on conversation size. 2.6. Merging Conversations The server MUST, under some rare circumstances, be able to merge two or more conversations together One simple example is when particular combinations of messages arrive out of order. Imagine three messages like this: Banks, et al. Expires October 31, 2015 [Page 6] Internet-Draft IMAP4 - Conversations April 2015 Message-Id: Subject: Hello World Message-Id: In-Reply-To: Subject: Re: Hello World Message-Id: In-Reply-To: Subject: Re: Re: Hello World Now imagine that the messages arrive in the order , , . When message is delivered, the server associates it with a new conversation A. When message is delivered, there is no detectable connection to message , so the server associates with a new conversation C. When message finally arrives, the message ids indicate that it should be associated to both conversations B and C. As a message can only be associated with at most one conversation, the conversations will need to be merged. When the server decides that a newly inserted message belongs in two or more conversations, it MUST merge those two conversations together into one larger conversation. However the server MAY split those conversations again to enforce some server specific size limit on conversations. Merging a conversations involves at least the following steps. o The server chooses a final conversation. The server SHOULD choose one of the existing conversations rather than create an entirely new conversation. o The server MUST associate the messages in all the conversations to be merged, with the final conversation. In the case that the message is already in the final conversation, this MAY be a no-op. In the case that the message is in another conversation, the server MUST update the message's CID property and increase the message's MODSEQ value. o The server MUST update the final conversation's MODSEQ value to be the highest of all the MODSEQ values of the messages in the conversation. o The server MUST delete the non-final conversations, so that they are not visible again from any IMAP command. The server MAY re- use the CID for an entirely new conversation at some time in the future. Banks, et al. Expires October 31, 2015 [Page 7] Internet-Draft IMAP4 - Conversations April 2015 After the conversations have been merged, every message in the original conversations MUST be associated with the final conversation. The SENDERS, FOLDERS and counter information in the final conversations must be a union of the respective information in the original conversations. At this point, the server MAY split the final conversation to enforce some server-specific limit on conversation size. 3. IMAP Protocol Changes 3.1. General Considerations 3.2. Interaction with Message Delivery When a message arrives in the server via any delivery mechanism (i.e. not via the IMAP protocol), the server MUST associate the message with a conversation as described in Section 2.5. 3.3. Interaction with APPEND When a message is added to the server using the APPEND command, the server MUST associate the message with a conversation as described in Section 2.5. 3.4. Interaction with COPY When a message is added to a folder using the COPY command, the server MUST associate the message with a conversation as described in Section 2.5. Note that if a message is copied to a folder in the same conversation scope as the original folder, the server SHOULD associate the new message with the same conversation as the original, except that the server MAY split conversations which exceed some server-specific size limit 3.5. Interaction with MOVE A server MAY implement both this extension and the MOVE extension [I-D.krecicki-imap-move]. When a message is moved from a folder to another folder in the same conversation scope, the server MUST preserve the association between the message and it's conversation. When a message is moved from a folder to another folder in a Banks, et al. Expires October 31, 2015 [Page 8] Internet-Draft IMAP4 - Conversations April 2015 different conversation scope, the server MUST drop any association between the message and a conversation and associate the message with a conversation in the new scope as described in Section 2.5. 3.6. Interaction with EXPUNGE 3.7. Interaction with RENAME 3.8. Interaction with STORE 3.9. New Data Items for FETCH This extension adds the following data items to the FETCH command and the FETCH untagged response. CID The Conversation ID of the message. FOLDER The name of the folder containing the message. This is not very useful in the FETCH command, as all the messages are in the same folder. However it is useful for the new XCONVFETCH (Section 3.14) command which uses the same syntax as FETCH. UIDVALIDITY The UIDVALIDITY value of the folder containing the message. Again, this is only useful for the new XCONVFETCH (Section 3.14) command. 3.10. New Criteria in SEARCH 3.11. New Keys in SORT 3.12. New Data Items in STATUS This extension adds the following data items to the STATUS command and the STATUS untagged response. XCONVEXISTS The number of conversations which have messages in this folder. XCONVUNSEEN The number of conversations with messages in this folder, which have at least one message without the \Seen flag set (note that such messages might be in a different folder). Banks, et al. Expires October 31, 2015 [Page 9] Internet-Draft IMAP4 - Conversations April 2015 XCONVMODSEQ The highest MODSEQ value of any conversation which is associated with a message in this folder. Note that this is in general greater than or equal to the folder's HIGHESTMODSEQ value. Note that these status data items all report summaries of properties of the messages associated with conversations which are associated with a message in this folder, provided for the convenience of conversations-aware clients. A server MAY implement them dynamically rather than pre-calculating and storing them, although pre- calculating is recommended for best performance as visible to the user. 3.13. New Command XCONVMETA Arguments: cid list conversation data items (one or more) Data: untagged responses: XCONVMETA Result: OK - request completed NO - request error: no such conversation BAD - command unknown or arguments invalid This extension adds the XCONVMETA command, which can be used by clients to query the state of a conversation. See Section 2.4 for a description of the data items which can be queried. 3.14. New Command XCONVFETCH Arguments: cid list unchanged-since modseq fetch data items (one or more) Data: untagged responses: FETCH Result: OK - request completed NO - request error: no such conversation BAD - command unknown or arguments invalid This extension adds the XCONVFETCH command, which can be used by clients to query the messages in a conversation. 3.15. New Command XCONVSORT Banks, et al. Expires October 31, 2015 [Page 10] Internet-Draft IMAP4 - Conversations April 2015 Arguments: sort criteria (zero or more) window arguments search criteria (one or more) Data: untagged responses: TODO Result: OK - request completed NO - request error: no such conversation BAD - command unknown or arguments invalid This extension adds the XCONVSORT command, which can be used by clients to list the conversations in a folder. 3.16. New Command XCONVUPDATES Arguments: sort criteria (zero or more) window arguments search criteria (one or more) Data: untagged responses: ADDED,CHANGED,REMOVED TODO Result: OK - request completed NO - request error: no such conversation BAD - command unknown or arguments invalid This extension adds the XCONVUPDATES command, which can be used by clients to discover changes in a folder since some arbitrary state in the past. 4. Formal Syntax The following syntax specification uses the Augmented Backus-Naur Form (ABNF) notation as specified in [RFC4234]. Non-terminals referenced but not defined below are as defined by [RFC3501], or [RFC4466]. Except as noted otherwise, all alphabetic characters are case- insensitive. The use of upper or lowercase characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion. capability =/ "XCONVERSATIONS" command-auth =/ convfetch / convmeta command-select =/ convsort / convupdates Banks, et al. Expires October 31, 2015 [Page 11] Internet-Draft IMAP4 - Conversations April 2015 convsort = "XCONVSORT" SP sort-criteria [SP convsort-windowargs] SP search-criteria ;; sort-criteria defined in RFC5256 ;; search-criteria defined in RFC5256, includes initial charset convsort-windowargs = "(" convsort-windowarg *(SP convsort-windowarg) ")" convsort-windowarg = "CONVERSATIONS" / "POSITION" SP "(" nz-number SP number ")" / "ANCHOR" SP "(" uniqueid SP number SP number ")" ;; position limit ;; anchor offset limit convupdates = "XCONVUPDATES" SP sort-criteria SP convupdates-windowargs SP search-criteria ;; sort-criteria defined in RFC5256 ;; search-criteria defined in RFC5256, includes initial charset convupdates-windowargs = "(" convupdates-windowarg *(SP convupdates-windowarg) ")" convupdates-windowarg = "CONVERSATIONS" / "CHANGEDSINCE" SP "(" mod-sequence-value SP ;; highestmodseq uniqueid ")" / ;; uidnext "UPTO" SP "(" uniqueid ")" ;; mod-sequence-value is defined in RFC4551 ;; number and nz-number are defined in RFC3501 ;; Note: CHANGEDSINCE is mandatory for xconvupdates convmeta = "XCONVMETA" SP cid-list SP conv-atts ;; TODO: which conversation scope are we getting ;; with XCONVMETA and XCONVFETCH? cid-list = cid / "(" cid *(SP cid) ")" conv-atts = conv-att / "(" conv-att *(SP conv-att) ")" conv-att = "EXISTS" / "UNSEEN" / "COUNT" SP flag-list / "SENDERS" / "FOLDEREXISTS" SP "(" mailbox *(SP mailbox) ")" ;; TODO: this appends not to but to the middle Banks, et al. Expires October 31, 2015 [Page 12] Internet-Draft IMAP4 - Conversations April 2015 ;; part of it, MEH ;; response-data = "*" SP (resp-cond-state / resp-cond-bye / ;; mailbox-data / message-data / capability-data) CRLF response-data =/ conv-data conv-data = "XCONVMETA" SP cid (conv-data-item / "(" conv-data-item *(SP conv-data-item) ")") conv-data-item = "MODSEQ" SP mod-sequence-value / "EXISTS" SP nz-number / "UNSEEN" SP number / "COUNT" SP conv-count-list / "SENDERS" SP "(" 1*address ")" / "FOLDEREXISTS" SP conv-folder-list conv-count-list = conv-count-item / "(" conv-count-item *(SP conv-count-item) ")" conv-folder-list = conv-folder-item / "(" conv-folder-item *(SP conv-folder-item) ")" conv-count-item = flag SP number conv-folder-item = mailbox SP number ;; TODO describe this better ;; TODO: specify what happens if the client requests a count ;; for a flag ;; we do not count ;; TODO: almost all these rules are incorrect about whether ;; zero-length lists of things are acceptable fetch-att =/ "CID" / "FOLDER" / "UIDVALIDITY" msg-att-dynamic =/ "CID" SP cid / "UIDVALIDITY" SP nz-number msg-att-static =/ "FOLDER" SP mailbox status-att =/ "XCONVEXISTS" / "XCONVUNSEEN" / "XCONVMODSEQ" convfetch = "XCONVFETCH" SP cid-list SP mod-sequence-value SP (fetch-att / "(" fetch-att *(SP fetch-att) ")") ;; TODO: xconvfetch always returns the FOLDER and UIDVALIDITY items Banks, et al. Expires October 31, 2015 [Page 13] Internet-Draft IMAP4 - Conversations April 2015 5. Acknowledgements This template was derived from an initial version written by Pekka Savola and contributed by him to the xml2rfc project. 6. IANA Considerations This memo includes no request to IANA. All drafts are required to have an IANA considerations section (see Guidelines for Writing an IANA Considerations Section in RFCs [RFC5226] for a guide). If the draft does not require IANA to do anything, the section contains an explicit statement that this is the case (as above). If there are no requirements for IANA, the section will be removed during conversion into an RFC by the RFC Editor. 7. Security Considerations All drafts are required to have a security considerations section. See RFC 3552 [RFC3552] for a guide. 8. Normative References [I-D.krecicki-imap-move] Krecicki, W., "Internet Message Access Protocol (IMAP) - MOVE extension", draft-krecicki-imap-move-01 (work in progress), June 2010. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3501] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003. [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, July 2003. [RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. [RFC4466] Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4 ABNF", RFC 4466, April 2006. [RFC4551] Melnikov, A. and S. Hole, "IMAP Extension for Conditional Banks, et al. Expires October 31, 2015 [Page 14] Internet-Draft IMAP4 - Conversations April 2015 STORE Operation or Quick Flag Changes Resynchronization", RFC 4551, June 2006. [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008. [RFC5256] Crispin, M. and K. Murchison, "Internet Message Access Protocol - SORT and THREAD Extensions", RFC 5256, June 2008. [RFC5322] Resnick, P., Ed., "Internet Message Format", RFC 5322, October 2008. Authors' Addresses Greg Banks (editor) Opera Software Australia Pty. Ltd. Level 1, 91 William St Melbourne, Victoria 3000 Australia Email: gnb@opera.com Robert Mueller Opera Software Australia Pty. Ltd. Bron Gondwana Opera Software Australia Pty. Ltd. Banks, et al. Expires October 31, 2015 [Page 15]