3GPP satisfaction survey, 2012 Analysis

3GPP satisfaction survey, 2012Analysis John M Meredith Director, ETSI Mobile Competence Centre 3GPP Specifications Manager

In late 2012 MCC conducted a survey of the satisfaction of delegates and users of the 3GPP web site and of the services offered by MCC. There were: 236 (79)responses from delegates (of which about 10% were chairmen or vice-chairmen) representing a reasonable spread amongst the TSGs and WGs indicated in the chart* … and 173 (89) non-delegate responses Introduction Throughout this presentation, where similar questions were asked in the 2010 survey, the corresponding scores are shown in gray italic font. “Instant” MCC reactions offered in blue font. * Respondents could choose more than one TSG/WG.

Question 9 (delegates): (2010 survey) Which TSGs or WGs do you regularly attend? Average score: (2010 survey) delegates: 3.4 (3.2) non-delegates: 3.3 (3.2)

Responders were asked to identify themselves as either delegates (including chairmen and vice-chairmen) or as 3GPP non-participants. Question 8 (delegates): What is your status? Most questions were rated1 (very poor) to 5 (very good).The scores are shown graphically, and the mean values given.

Overall score out of 5: Average of averages: 3.5 (3.5) No question scored less than the median of possible scores, 2.5. Executive summary (1) What is good? Up to date web site content Web site overall, including ease of navigation MCC service • What is bad? • Finding information on the web site (not involving the search engines) • The advanced search engine • TDoc allocation methods / tools

Executive summary (2) What are recurrent comments ? The search engine: too difficult to restrict the search to find, for example, a single Tdoc Too many clicks on the web site needed to reach the desired page. TDoc number allocation method varies from group to group Manual allocation suffers from bottlenecks at busy times Automatic allocation (ADN application) not really fit for purpose Need local server based application during meetings, to relieve secretary of this burden Need improved method to identify (group of) Specs sought

Detailed results …

Question 1: How easy is it to navigate our web site? Average score: (2010 survey) delegates: 3.4 (3.2) non-delegates: 3.3 (3.2)

Logical structure, easy to navigate. • Not easy to find (U)SIM test cases. • Too many clicks needed to reach documents. • Button icons on banner are not in fact buttons! • Links labelled “click here” not descriptive enough. • Good in general, but with some inconsistencies. • Search engine useless. • Nice layout, easy to find what I was looking for. • Inconsistent spec series per responsible WG. • FTP platform is a nightmare. • I like the style and design. • Poor design and layout, poor colour scheme. • Work plan pages not easy to navigate, very wide pages. Question 1 – non-delegate remarks: How easy is it to navigate our web site?

Question 2: How easy is it to find the information you are looking for on our web site? Average score: delegates: 3.1 non-delegates: 3.0 Remark: Includes use of search engines?

Search results not always very useful. • Easy to find specs via numbering plan. • Difficult to identify a spec if you don’t know the number. • Search using Google better than 3GPP site search. • Need summaries with real time aspects. • Feature list per Release is vague and doesn’t help determine the improvements actually achieved from Release to Release. • Needs more explanation of terminology (eg Stages 1, 2, 3). • Keywords. • Need to be able to search by date, and within databases (specs, TDocs).The advanced search engine provides some of this; too difficult to use? • Where to start? Help! • Layer 1 specs use Word’s Equation Editor, which makes searches difficult. Question 2 – non-delegate remarks: How easy is it to find the information you are looking for on our web site?

Question 3: How visually appealing is our web site? Average score: (2010 survey) delegates: 3.2 (3.7) non-delegates: 3.2 (3.5) Remark: Unchanged since last survey. New look needed?

Banner image of doctor using her tablet may confuse end users; not intuitive. • Very nice, appealing. • Unattractive, geeky. • Needs some Flash video. • Simple and explanatory. • Not interested in visual appearance. • Need more variety of images on page banners. Question 3 – non-delegate remarks: How visually appealing is our web site?

Question 4: How up-to-date is the content on our web site? Average score: (2010 survey) delegates: 3.6 non-delegates: 4.6

Seems to be up-to-date. • Yes, I am only interested in accessing published standards. • As a technical writer, I am mildly impressed by the quality of the MS Word templates. Question 4 – non-delegate remarks: How up-to-date is the content on our web site?

Question 5: How easy is it to access and to use the specification database? Average score: (2010 survey) delegates: 3.4 (3.2) non-delegates: 3.3

Finding particular versions of specs is easy, but not to find the rationale behind choice of technologies. • Finding a particular spec is difficult, especially if you don’t know its number. • Need graphical hierarchy of specs, with possibility for block downloads by technology area. • No centralized WG document list per meeting, hence difficult to trace the WG discussions.These do exist, of course, but maybe not easy enough to find? • Specs to have an index referring to contributing TDocs and meeting minutes. • Web site not structured to allow a view of the whole picture. Question 5 – non-delegate remarks: How easy is it to access and to use the specification database?

54 Question 6 (delegates): How satisfied are you with the information about membership? Average score: (2010 survey) delegates: 3.3 (3.0)

Question 6 (non-delegates): If you visited the web site seeking membership information, did you find what you needed?

Question 7: How likely are you to recommend our web site to others? Average score: delegates: 3.7 non-delegates: 3.9

Would like to be able to use site on a smartphone. • Would like direct links to WGs (eg for incoming LSs). • Handling of mailing lists is very confusing. Why two different pages for managing subscriptions? • Can’t get to the ETSI meeting calendar home page directly from the main menu. • Don’t hide TDocs inside zip files, prevents Google search.(But doesn’t prevent 3GPP search!) • News items are too European. Consider other regions. • Not clear who are the stakeholders in 3GPP.This is revealed on the Partners page; too cryptic? • Too many clicks needed to reach your target page. • Need instructions how to use the FTP site. • Links on work items per WG pages are to full work plan page, which takes too long to load; would be better to a page for that one work item. Question 7 – delegate remarks - 1: How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

Need electronic handling of TDocs: checking out/in; numbering, allocating to agenda items, co-signing, revising, etc. Present system is outdated. • Not only are technical terms difficult to deduce from abbreviations, but also administrative terms like TSG, WG, CR. • Should have a delegate feedback survey for every meeting.This is already planned, in response to the PCG ahg on IT Improvement recommendations. • Workplan still difficult to navigate. • Some key pages difficult to find. • Difficult to find 3GPP procedural documents. • Very poor design, links hard to find. • No phone numbers for MCC personnel.Of course, these are given; too hard to find? Question 7 – delegate remarks - 2: How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

Would recommend to engineers working in this field. • I use it every day, it is critical to my work. • Fast download. • Good site for the latest technology updates for LTE. • Beginners find the site difficult to navigate, keep it simple. • I want to know the member organizations of 3GPP. Question 7 – non-delegate remarks: How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

Question 8 (non-delegates) - 1: • Can find what but not why. • Poor search engine.Poor, or poorly understood, thus poorly explained? • Search tool to go directly to a particular Spec page, or Tdoc. • Web site needs massive index to find relevant spec easier.It has one, and it drives the advanced search engine! • Powerful, self-learning, search tool like Google. • Spec numbering scheme not a very good guide to Spec contents. • Long specs should have an executive summary. • No FAQ page.Actually, failed to find FAQ page; because not easy to find? • Too structured, not sufficiently results-oriented. Can you point out any missing features or any weaknesses on the 3GPP web site?

Provide specs in PDF. • Provide specs in HTML. • Provide implementation examples. • More technology tutorials, by RAT, by Release; with message flows. • No link between specs and contributing TDocs.Perhaps they don’t understand the principle of CRs … ? • Mapping of specs to network architecture. • Need guide to spec series numbering.There is, of course, exactly this. Too difficult to find? • Make white papers available in PDF. • Smartphone rankings.Seek elsewhere! Question 8 (non-delegates) - 2: Can you point out any missing features or any weaknesses on the 3GPP web site?

Need overview of architecture, protocols, procedures, which points to corresponding specs; maybe graphical interface*. • Graphical interface to (all) CRs. • Grouping of specs by function (eg Charging).More rigorous use of keywords? • Grouping of CRs by function. • Need link on home page to technology areas.There is one, from main menu! • Need link to GSMA site, collaboration is important. • Selective download of TDocs by subject area, agenda item, duration of discussion. • CRs should indicate which RAT they apply to. • Pointers to external press articles relating to 3GPP technologies. • Explanation of why some specs sometimes not available. • Add TS and TR number to the Release Description documents. • Need a high level overview page, describing what 3GPP is.Is the “About” page deficient? • Need different areas of web site targeted at different audiences. • Better grouping of work items into categories such as “most challenging”. * Commercial offering proposed! Question 8 (non-delegates) - 3: Can you point out any missing features or any weaknesses on the 3GPP web site?

Question 10 (delegates): How satisfied are you with the following on-line services …

Question 10a: E-mail exploder list management? Average score: (2010 survey) delegates: 3.7 (4.1) Remark: The e-mail exploder facilities are unchanged. Delegates expectations have evidently increased.

Question 10b: Specification Group pages? Average score: (2010 survey) delegates: 3.5 (3.8) Remark: These pages are almost unchanged in content type and format. Delegates expectations have increased.

Question 10c: Meeting information? Average score: (2010 survey) delegates: 3.6 (3.9) Remark: These pages are unchanged in content type and format. Delegates expectations have increased.

Can’t manage exploder list subscriptions due to technical issue in my company.Possibly due to asymmetric addresses (different rx & tx). • Too many clicks to reach meeting invitations. • Information for next meeting seems to be in several different places, difficult for newcomers to find. Propose one page for next meeting, and one for most recent meeting, each containing all relevant info. • Meetings page is confusing. • Meeting invitations and attendee lists difficult to find. • Too many clicks to reach frequently accessed pages. • TDoc handling needs improving. • Junk mail on exploder lists. • Pages for groups out of date. • Submitting TDocs by email is painful. • Lists should be used more for discussions. • Meeting registration: ICS file difficult to find. Question 10 – delegate remarks: How satisfied are you with … on-line services?

Question 11 (delegates): How satisfied are you with the support you receive from your MCC support officer for TSGs and WGs? Average score: (2010 survey) delegates: 4.0 (3.2)

Question 12 (delegates): How would you rate the IT infrastructure during meetings …?

Question 12a: Access and robustness of the local server? Average score: (2010 survey) delegates: 3.6 (3.2) Remark: A pleasing, if modest, improvement.

Question 12b: Speed and reliability of the public internet access? Average score: (2010 survey) delegates: 3.2 (3.2)

WiFi bandwidth has improved considerably in recent years but hotel IT infrastructure not so good. • Sometimes excellent, sometimes poor. (Especially internet speed.) • In general, performs well. Fit for purpose. • Poor in China. (Internet and local server in Qingdao) • Takes me half a day to get IP address at beginning of (RAN WG) meeting. • Current solution does not allow me to run VPN.This is generally a problem with companies’ VPN settings, not with MCC infrastructure. • Internet bandwidth needs to be considerably increased. • Very variable, depending on location and host. • Hotel (bedroom?) internet access frequently poor. • Occasional interference in meeting rooms from delegates’ ad hoc networks. • Need automatic TDoc number allocation tool on meeting server. • Better since the introduction of 5GHz band. • Occasional problems early in the week. • IT support could have been more responsive (at a recent meeting). Question 12 – delegate remarks - 1: How would you rate the IT infrastructure during meetings …?

Internet access often virtually unusable in meetings. Unacceptable due to need for email discussions. • WiFi infrastructure in meetings is very variable. • Collocated meetings can be poor (eg Jacksonville megameeting, November 2010). • Difficult to find the right meeting folder for CRs and meeting reports. Suggest a URL such as http://3GPP70.On the meeting server? Really?? • Difficult to get IP address at “rush-hour”. • Sometimes 3GPP wifi network suffers from interference from hotel network. MCC should know how to fix this. • Overall current experience is excellent. • Frequent reboots. Question 12 – delegate remarks - 2: How would you rate the IT infrastructure during meetings …?

Question 13 (delegates): How satisfied are you with the accuracy and speed of implementation of approved change requests? Average score: (2010 survey) delegates: 3.8 (4.0)

Had to resubmit 2 CRs in the last 12 months.Because not implemented first time? • Some CR packs presented to one meeting were empty. • Formatting issues with one spec, crashed MS Word. • Occasionally CRs not implemented, or misimplemented.Current reported error rate is 2 un/misimplemented CRs per thousand. • Problem when WG meeting follows soon (3 weeks) after TSG meeting. • I don’t understand the question. Question 13 – delegate remarks: How satisfied are you with the accuracy and speed of implementation of approved change requests?

Question 14 (delegates): How would you rate the following tools …

Question 14a: 3GPP search tools? Average score: (2010 survey) delegates: 3.1 (2.7) non-delegates: (2.8) Remark: In view of plain text remarks, surprised to find this score higher than last time.

Question 14b: Membership web page? Average score: (2010 survey) delegates: 3.3 (3.0) non-delegates: (3.4)

Question 14c: Administrative information? Average score: (2010 survey) delegates: 3.4 (3.0) non-delegates: (3.4)

Question 14d: On-line registration? Average score: (2010 survey) delegates: 3.8 (4.4) Remark: New registration tool introduced shortly before start of survey.

Question 14e: Document number allocation? Average score: (2010 survey) delegates: 3.5 (3.8) Remark: Modest decline in satisfaction corresponds with verbal comments.

ADN performance poor under overload conditions. • ADN should allocate CR numbers as well as TDoc numbers. • TDoc number allocation differs from group to group; needs harmonizing across all groups. • Manual TDoc number allocation, poor response during pre-meeting rush. • Manual TDoc numbering is acceptable. • ADN now asks for too much information. • ADN is far too slow if you need several documents. • ADN sucks! • “Hiring company” and “Represented company” as entered at registration time are not accurately reproduced in the participants list. The order of clicks is important at registration time. • Improve the TDoc upload process, need a progress bar for large documents. • Membership page could do with wildcard capability when searching for company name. • Revisions of TDocs should use the same number, with “rev” suffix. • During meetings, new TDoc numbers communicated by word of mouth, often misheard, so multiple appearances of same number. • TDoc numbering scheme is good, but finding them is very difficult. Question 14 – delegate remarks: How would you rate the […] tools?

Question 15 (delegates): Use of the 3GPP databases …

Question 15a: Do you use the CR database?

Question 15b: Do you use the specifications status database?

Question 15c: (33%) Are you satisfied with the information provided in the 3GPP workplan? (67%)

3GPP satisfaction survey, 2012 Analysis