802.11 Comment Resolution – a Tutorial

802.11 Comment Resolution – a Tutorial Authors: Date: 2016-01-19 Adrian Stephens, Intel Corporation

This tutorial is intended to help 802.11 members to approve adequate comment resolutions to working group and sponsor letter ballots The audience is anybody in 802.11 involved in writing or voting on comment resolutions Red text is reserved for examples of poor practice Introduction Adrian Stephens, Intel Corporation

We haven’t always don’t an adequate job in the past, which has cost us time/hot-air/embarrassment/quality We have specific rules that we need to follow New people are coming in all the time, which dilutes “corporate memory” of what to do and what to avoid We should, as much as we can, avoid “learning by making mistakes” because the cost of making mistakes in this process can be significant Why do we need a tutorial? Adrian Stephens, Intel Corporation

To increase the level of support by balloters for a draft To improve the quality of a draft standard To engage with voters and gain from their viewpoint/experience To complete a ballot with a draft of high quality that meets the approval of most (>75%, typically 95%) of its voters in a timely fashion. Note the tension – frequently cannot satisfy all the commenters and complete in a timely fashion The purpose of comment resolution Adrian Stephens, Intel Corporation

We do not gain sufficiently from the possible benefits of engaging with voters We produce a poor quality draft document We get inadequate involvement from WG members who might give up trying to the process We risk delay by not dealing with real issues in a responsive or timely fashion We risk delay in the EC through appeals to process We risk delay in the IEEE-SA standards board What can happen if we do this badly? Adrian Stephens, Intel Corporation

A comment needs to identify where the issue is in the draft It needs to identify what the issue is It needs to identify a proposed change in sufficient detail that the CRC can readily identify changes that they would reasonably expect to satisfy the commenter. The wording from the IEEE-SA Standards Board Operations Manual (2010 p24) is: “This vote must be accompanied by one or more specific objections with proposed resolution in sufficient detail so that the specific wording of the changes that will cause the Do Not Approve voter to change his or her vote to Approve can readily be determined .” Needs to be in scope (next slide) Valid Comments Adrian Stephens, Intel Corporation

A comment is in scope if: • It is in the initial ballot • It is in a recirculation, and relates to material in the balloted draft changed since the previous draft, or relates to material affected by that change • Is in a recirculation, and relates to text that is the target of an unsatisfied “must be satisfied” comment from a “no” voter. Valid comment scope Adrian Stephens, Intel Corporation

Examples of invalid comments - 1 Adrian Stephens, Intel Corporation

Examples of invalid comments – 4 Adrian Stephens, Intel Corporation

Address all comments – be responsive to the technical issue wherever this is possible • Answer the comment made, not the comment you wish they’d made. Answer all the points in the comment. • The proposed change should relate to the comment directly, not be an excuse to fix something else. • Minimize procedural resolutions, except near the “end” • Address the technical issues in preference to using a “scope” or “invalid comment” resolution. • Use neutral language – address the issue, not the commenter Recommendations for comment resolution Adrian Stephens, Intel Corporation

Use the correct resolution • Accepted means “we agree with the comment and the proposed change”. It cannot be used if the proposed change is not sufficiently detailed that the editor and TG members know what the intended change is, e.g. if the commenter offers alternative solutions. • Revised means “we agree with the comment, but we approved a change different from the commenter’s proposed change” • Rejected means “we disagree with the comment, and made no changes”. Don’t use “Rejected” if the comment is addressed in another comment. Instead, repeat part of the resolution of that comment as appropriate. Don’t use “Rejected - Fixed in Draft x.y” as a resolution because it doesn’t say what you changed to resolve the comment. Technical aspects of comment resolution - 1 Adrian Stephens, Intel Corporation

Avoid over-use of references to submissions • This forces the commenter to do work that you could have done. • Copy and paste textual resolutions from submissions. • Reserve references for material that cannot be represented in plain text, e.g. graphics and heavy use of insert/delete. • e.g. – bad usage: “Rejected – see 11-12/1234r0”. • Use URLS • Any references to external documents should be URLs, not 802.11 document numbers. This is particularly important in sponsor ballot, because the ballot group member don’t necessarily know where to look for 802.11 documents. Technical aspects of comment resolution - 2 Adrian Stephens, Intel Corporation

Avoid references to other comment resolutions • When we present unsatisfied comments to the EC (or RevCom read them), they might see only a subset of comments, e.g. the “Must be Satisfied” ones. When RevCom look at a comment resolution, they do not necessarily see the same comment ID as the TG works with. • These both mean that references to comment IDs in resolutions are useless. • Instead of referencing a comment resolution – copy it. • For example: • Revised. At page 1234 line 56, change ‘nurgle’ to ‘flange’. • Note to editor (this is the same as comment resolution for CID 321). Technical aspects of comment resolution - 3 Adrian Stephens, Intel Corporation

Invalid Resolution:“Rejected. We will consider this comment in a later ballot, but we ran out of time to discuss it now.” • Avoid promises of future action. Any such promise is an immediate red flag to the EC / RevCom. Two exceptions to this rule: • Editorial comments can be deferred to the publication editor • For comments out of scope: “Such comments need not be addressed in the current standards balloting process and may be considered for a future revision of the standard” (IEEE-SA SB OM) • A member of the TG who is in the sponsor ballot pool might offer to submit certain comments from the end of WG ballot into sponsor ballot; but if he does so, it is an individual act, not an official one. Technical aspects of comment resolution – 4 – promises of future action Adrian Stephens, Intel Corporation

Resolution should include: • “Accepted”, “Revised” or “Rejected” classification. • A response to each issue raised in the comment. • Clearly distinguish between any response directed to address a question raised in the comment and instructions to the editor. • e.g.: • “Revised. • At p 1234 line 56 change “shall” to “should”. • In reply to the commenter, this allows legacy STAs to continue this behaviour, but allows new STAs to choose what to do. Technical aspects of comment resolution – 5 structure of resolution Adrian Stephens, Intel Corporation

Comments should be resolved without regard to how the commenter classifies them, except: • Usually the editor gets to propose resolutions to most of the editorial comments • In WG ballot, editorial comments that are not part of a no vote might be resolved “This comment has been passed to the TG technical editor for consideration during preparation of a subsequent draft” • Near the end of the sponsor ballot, editorial comments might be resolved with “This comment will be passed to the IEEE-SA editor for consideration during publication editing.” • The last two are permissible exceptions to the “don’t promise future action” rule. Technical / General / Editorial comments Adrian Stephens, Intel Corporation

Examples of Unresponsive Comment Resolutions Adrian Stephens, Intel Corporation

Resolution: Reject. The TG considered the comment and did not agree with the proposed resolution. Analysis: Unresponsive. The response is generic and does not include the subject matter of the comment. Responsive / unresponsive - 1 Adrian Stephens, Intel Corporation

Resolution: The TG wasn’t sure whether to accept the comment or not and couldn’t agree on a change Analysis: Unresponsive. The response is wishy-washy and leaves the commenter with inadequate information on the TG’s position. Responsive / unresponsive - 2 Adrian Stephens, Intel Corporation

Resolution: Reject. Coexistence with mode X devices is not a requirement of the PAR. The TG considered the comment and did not agree with the proposed resolution on the basis that it added significant complexity with insufficient benefit. Analysis: Responsive. The response includes the subject matter of the comment, and provides concise reasons why the comment was rejected. It is not necessary for the resolution to provide extra detail on precisely how much complexity would be tolerable and how the TG members traded off complexity versus benefit. Responsive / unresponsive - 3 Adrian Stephens, Intel Corporation

Types of Procedural Rejection Adrian Stephens, Intel Corporation

Procedural Rejections – 1 (insufficient detail) Adrian Stephens, Intel Corporation

Procedural Rejections – 2 (asking a question) Adrian Stephens, Intel Corporation

Rejected. The comment fails to identify changes in sufficient detail so that the specific wording of the changes that will satisfy the commenter can be determined. Note, the CRC can (but is not required to) admit or accept the validity of any reported “big issue”. Note that the CRC doesn’t need a valid comment to make any changes it wishes, and if it agrees that there is a big issue and some ballot cycles later determines how to resolve that issue, it is at liberty to make those changes. Procedural Rejections – 3 (big issue) Adrian Stephens, Intel Corporation

Rejected. The comment is out of scope: i.e., it is not on changed text, text affected by changed text or text that is the target of an existing valid unsatisfied comment. Procedural Rejections – 4 (out of scope) Adrian Stephens, Intel Corporation

Context: The TG cannot reach a consensus on any other resolution. • The CRC has considered one or more other resolutions that attempt to satisfy the comment, but no such resolution has received 75% approval. • It should be recorded in the minutes what other resolution(s) have been considered, together with the results of any straw polls or motions on those alternative solutions so that the commenter can understand that this solution was used after due diligence to satisfy the comment. • Rejected. The CRC could not reach consensus the changes necessary to address the comment this comment. Procedural Rejections – 5 (can’t agree on a change) Adrian Stephens, Intel Corporation

Context: A commenter repeats (or is substantively similar to) either their own or another commenter’s valid unsatisfied comment from a previous ballot. The resolution should include a statement that the CRC has previously considered the comment (or a substantively similar comment), along with identification (by reference or copy) of the original comment and its disposition detail and status. Example: Rejected. The CRC has previously considered this comment in the Initial Sponsor Ballot. The comment considered was “The flange needs to be nurgled” and the resolution of that comment was “Rejected. The comment resolution committee believe the cost of nurgling the flange outweighs any benefit from doing so.” Procedural Rejections – 6 (pile on) Adrian Stephens, Intel Corporation

Approval cycle: • WG Ballot • EC approval to proceed to sponsor ballot • Sponsor Ballot • EC approval to proceed to RevCom • IEEE-SA RevCom and Standards Board approvals • Permissive vs restrictive • Solicit input from “no” voters • Recirculate unchanged • EC approval • RevCom / IEEE-SASB approval Typical Ballot Lifecycle Adrian Stephens, Intel Corporation

In the first ballot, all comments are in scope (although not all comments are necessary valid). Early in the process, the TG will probably attempt to satisfy comments regardless of whether they are valid As the draft nears the end of a sequence of recirculation ballots, the TG wants to proceed to the next stage in the process. At this point they are likely to become restrictive, rejecting comments on the basis of validity or other procedural grounds. Ballot Lifecycle - Permissive vs restrictive Adrian Stephens, Intel Corporation

Comment resolution provides an opportunity for 2-way communication between the commenter and the TG. There is no reason to limit this to communicating via the official tools. The TG are encouraged to engage directly with commenters to understand better what they are saying in a comment, and to seek feedback on its possible resolution. The TG chair should seek feedback from unsatisfied comments from previous rounds so that the actual unsatisfied comments are identified for presentation to the EC. Ballot Lifecycle - Solicit input from “no” voters Adrian Stephens, Intel Corporation

At the end of a recirculation ballot, unless the ballot received no comments, the group will determine that there are reasons to validly reject all the comments received. • In this case an unchanged draft is sent to the next (final) recirculation along with the reasons for these rejections • Comments might be received from this final recirculation, but they are either a reiteration of an earlier comment (i.e., not new) or they are out of scope. • Thus the process is guaranteed to terminate, should the TG so wish • i.e., no commenter can force the process to continue indefinitely Ballot Lifecycle - Recirculate unchanged Adrian Stephens, Intel Corporation

The approval of the EC is required to progress between WG ballot and Sponsor ballot, and to go to RevCom Approval is granted (either unconditional or conditional) at plenary sessions of 802 (March, July, Nov), and can also be sought in EC telecons, or by EC ballot The TG produces and approves a report on the status of the ballot and unsatisfied voters/comments to solicit this approval. In the case of conditional approval, an email on meeting the terms of the conditional approval is sent by the WG chair to the EC when the terms are met. Ballot Lifecycle - EC approval Adrian Stephens, Intel Corporation

RevCom reviews material it retrieves from the MyBallot system in order to determine whether the process for sponsor ballot has been properly followed. They have visibility of all comment resolutions. RevCom members might enter into Q&A with the TG chair about issues they have with comment resolutions. RevCom makes a recommendation to the standards board (IEEE-SASB), which is responsible for the actual approval. Ballot Lifecycle - RevCom / IEEE-SASB approval Adrian Stephens, Intel Corporation

During WG comment resolution, the TG resolves comments • Motions can be made only during 802.11 sessions, except • Once conditional approval has been obtained from the EC, a TG telecon can approve resolutions by motion (802.11 OM) • During Sponsor ballot comment resolution, the WG chair delegates responsibility to a comment resolution committee (CRC) to resolve comments • Any 802.11 voter can join this CRC, which is practically indistinguishable from a TG, except that it can vote during telecons. Difference between WG and Sponsor Ballot Adrian Stephens, Intel Corporation

Post timely spread sheets. All 802.11 members should have essentially equal access to the comments and their proposed resolutions. • Ensure all comments have a valid resolution • Ensure resolutions are actioned in the draft. This is mainly the responsibility of the editor. • Nothing discombobulates commenters more than the group approving a change and then not making that change. • Chase unresponsive “no” voters • Prepare reports to the EC • (Tools, support and training are available from WG officers) Requirements on TG officers Adrian Stephens, Intel Corporation

If this tutorial informed you, but has no effect on your future behaviour, your attendance was a waste of time. • What are you going to do to improve the quality of your comment resolutions? • Call to action: Be critical of your own and others comment resolutions. • Spending time getting this right now should save you time in the future. The end? Adrian Stephens, Intel Corporation

802.11 Comment Resolution – a Tutorial