Development of a Survey Question on Sexual Identity

2. Purpose of presentation Provide an overview of the project. Report on work undertaken. Demonstrate advantages of examining the response process from multiple perspectives. Presenting work carried out by colleague, Amanda Wilmot, and her team of Data Collection methodologists on the development of a question on sexual identity. This presentation will: Provide an overview of the project and the reasons behind its inception; And describe the methodology employed and findings, as the project progressed. In doing so, the presentation will also hope to demonstrate the benefits of combining qualitative and quantitative approaches to give insight into the main issue of providing high quality data. Presenting work carried out by colleague, Amanda Wilmot, and her team of Data Collection methodologists on the development of a question on sexual identity. This presentation will: Provide an overview of the project and the reasons behind its inception; And describe the methodology employed and findings, as the project progressed. In doing so, the presentation will also hope to demonstrate the benefits of combining qualitative and quantitative approaches to give insight into the main issue of providing high quality data.

3. Project aims Project initiated in response to: A change in equality legislation. 2011 Census consultation. Project aimed to: Examine feasibility of providing benchmark data. Examine feasibility of providing measure of disadvantage. Provide advice on best practice with regard to data collection. The project was initiated Firstly, in response to equality legislation in the UK, relating to discrimination and disadvantage on the basis of sexual orientation; and Secondly, in response to the 2011 Census consultation process, which discussed whether or not a question on this subject could be included in the next Census. The project aimed to examine the feasibility of providing benchmark data, as well as providing a measure of disadvantage, either from Social surveys or from the Census. And to provide advice on best practice with regard to the data collection. Both Census and Survey managers were most concerned about the possible impact such questioning could have on response. The project was initiated Firstly, in response to equality legislation in the UK, relating to discrimination and disadvantage on the basis of sexual orientation; and Secondly, in response to the 2011 Census consultation process, which discussed whether or not a question on this subject could be included in the next Census. The project aimed to examine the feasibility of providing benchmark data, as well as providing a measure of disadvantage, either from Social surveys or from the Census. And to provide advice on best practice with regard to the data collection. Both Census and Survey managers were most concerned about the possible impact such questioning could have on response.

4. Establishing the measurement concept Source: Prevalence of the Three Dimensions of Sexual Orientation from Laumann et al.�s (1994) U.S. Probability Sample Firstly we needed to establish what concept we were intending to measure. The literature suggests that sexual orientation is multi-dimensional, including the dimensions of Attraction, Behaviour and Identity. The data here shows the greatest prevalence is in attraction, followed by behaviour, and then identity. It should be noted that in the legal framework uses the term �sexual orientation�. The legal definition is intended to be as wide as possible. The current UK government estimate for the prevalence of the Lesbian, Gay and Bisexual GB population is between 5 and 7 per cent. Firstly we needed to establish what concept we were intending to measure. The literature suggests that sexual orientation is multi-dimensional, including the dimensions of Attraction, Behaviour and Identity. The data here shows the greatest prevalence is in attraction, followed by behaviour, and then identity. It should be noted that in the legal framework uses the term �sexual orientation�. The legal definition is intended to be as wide as possible. The current UK government estimate for the prevalence of the Lesbian, Gay and Bisexual GB population is between 5 and 7 per cent.

5. Measurement concept: Sexual identity More stable over time. Less intrusive. De facto: Sexual identity within a household environment. However, our intended measure is sexual identity. This is because identity is the most stable dimension over time, and from a practical perspective, to ask people to reveal aspects of their sexual attraction or behaviour in the home environment is probably too intrusive for most ONS multi-purpose surveys, and indeed the Census. Since the project was concerned with developing a question that could be administered in face-to-face interviews in the home of the respondent, or on a Census form, one could argue that, what is actually being measured is sexual identity within a household environment. However, our intended measure is sexual identity. This is because identity is the most stable dimension over time, and from a practical perspective, to ask people to reveal aspects of their sexual attraction or behaviour in the home environment is probably too intrusive for most ONS multi-purpose surveys, and indeed the Census. Since the project was concerned with developing a question that could be administered in face-to-face interviews in the home of the respondent, or on a Census form, one could argue that, what is actually being measured is sexual identity within a household environment.

6. Methodology UK and International survey review Quantitative research Four Omnibus survey trials. General Lifestyle Survey pilot. Analysis of proxy data. Qualitative research Telephone interviews with those who preferred not to answer. Feedback from interviewers / Field observations. Focus groups with members of the public. Cognitive/in-depth interviews with members of the public. The research was divided into different work packages, each designed to address various elements of the response process. Firstly, a review was carried out of data already available, both in the UK and Internationally, to learn from others experiences and set our findings in context. This was followed by both qualitative and quantitative research The Qualitative element provided insight into whether respondents were answering the question in the way researchers intended. And we could examine the accuracy of the reporting as well as how respondents felt about it. The Quantitative element helped to establish the effect on response, attrition, and question administration, as well as the effect of any changes to question wording on the estimates produced. This research is particularly important in addressing the concerns of survey managers. The research was divided into different work packages, each designed to address various elements of the response process. Firstly, a review was carried out of data already available, both in the UK and Internationally, to learn from others experiences and set our findings in context. This was followed by both qualitative and quantitative research The Qualitative element provided insight into whether respondents were answering the question in the way researchers intended. And we could examine the accuracy of the reporting as well as how respondents felt about it. The Quantitative element helped to establish the effect on response, attrition, and question administration, as well as the effect of any changes to question wording on the estimates produced. This research is particularly important in addressing the concerns of survey managers.

7. Quantitative trials : Omnibus trials 1 and 2 Computer Assisted Self-Interviewing (CASI) Trial 1: July/August 2006 �Which of the following best describes your sexual identity? Heterosexual 92.0% Gay or Lesbian 1.3% Bisexual 1.2% Other (please specify) 0.9% Prefer not to say 4.6% Base = 2124 (Estimates not suitable for publication) Trial 2: November/December 2006 �Do you consider yourself to be�� Heterosexual or Straight 96.8% Gay or Lesbian 0.8% Bisexual 0.6% Other (please specify) 0.3% Prefer not to say 1.5% Base = 1910 (Estimates not suitable for publication) A question was trialled first on our Omnibus Survey which interviews one person in the household. In our first Omnibus trial, the question was asked at the very end of the interview so as not to affect response. Respondents were asked to self-complete using the laptop (CASI) and interviewers were told not to ask the question, even where respondents could not use the laptop. No obvious effect on response was observed, so a second test took place taking a slightly different approach. The question was also bought forward in the interview. No respondents dropped out of the interview immediately after being asked the question. Estimates obtained from the trials are shown in the slide. But these are incomplete because we told interviewers they could skip the question when they could not administered it in CASI. About 15% of cases were skipped, mainly because of respondent difficulties with the laptop and privacy issues. Interviewers were more likely to skip the question with older respondents, and those from lower socio-economic backgrounds. Male interviewers were more likely to skip if the respondent was female, particularly over the age of 60. This issue we needed to address in our interviewer training. Very few answered �other�, and nearly all went on to describe themselves as �normal�, or indicated that they did not understand the term �heterosexual�. A problem, however, was that around 5% preferred not to answer in our first test, which was more than the combined total of those reporting LGB or other. It was important to reduce the proportion of this missing data, but the data alone did not provide reasons why. So we conducted a qualitative telephone follow-up. There were certainly comprehension issues, particularly with the term �heterosexual�. Which is why they had chosen the PNTS option. But the proportion fell from 4.6% to 1.5% in the second test with inclusion of the term �straight� was added aiding comprehension. A question was trialled first on our Omnibus Survey which interviews one person in the household. In our first Omnibus trial, the question was asked at the very end of the interview so as not to affect response. Respondents were asked to self-complete using the laptop (CASI) and interviewers were told not to ask the question, even where respondents could not use the laptop. No obvious effect on response was observed, so a second test took place taking a slightly different approach. The question was also bought forward in the interview. No respondents dropped out of the interview immediately after being asked the question. Estimates obtained from the trials are shown in the slide. But these are incomplete because we told interviewers they could skip the question when they could not administered it in CASI. About 15% of cases were skipped, mainly because of respondent difficulties with the laptop and privacy issues. Interviewers were more likely to skip the question with older respondents, and those from lower socio-economic backgrounds. Male interviewers were more likely to skip if the respondent was female, particularly over the age of 60. This issue we needed to address in our interviewer training. Very few answered �other�, and nearly all went on to describe themselves as �normal�, or indicated that they did not understand the term �heterosexual�. A problem, however, was that around 5% preferred not to answer in our first test, which was more than the combined total of those reporting LGB or other. It was important to reduce the proportion of this missing data, but the data alone did not provide reasons why. So we conducted a qualitative telephone follow-up. There were certainly comprehension issues, particularly with the term �heterosexual�. Which is why they had chosen the PNTS option. But the proportion fell from 4.6% to 1.5% in the second test with inclusion of the term �straight� was added aiding comprehension.

8. Reasons for selecting �prefer not to say� Omnibus trial 2: November/December 2006 The second test included quantitative questioning as to why respondents preferred not to answer. Respondent characteristics associated with PNTS were: being older; being female (particularly if the interviewer was male); having no qualifications, belonging to a lower socio-economic group, living in London (which we think is probably associated with ethnicity), and coming from a deprived neighbourhood. The second test included quantitative questioning as to why respondents preferred not to answer. Respondent characteristics associated with PNTS were: being older; being female (particularly if the interviewer was male); having no qualifications, belonging to a lower socio-economic group, living in London (which we think is probably associated with ethnicity), and coming from a deprived neighbourhood.

9. Quantitative trials: Omnibus trial 3 July/August 2007 - Combination of Computer Assisted Self-Interviewing (CASI) and Computer Assisted Personal Interviewing (CAPI) A third trial was run last summer. The intention here to remove the skip option for interviewers and administer the question to everyone. Also to allow interviewers to ask the question if privacy was assured. The question was still omitted in around 2% of cases. These were a combination of respondent refusals due to discomfort with the topic, as well as cases where interviewers felt the question could not be administered because of the age of the respondent. The order of the response options was also tested; no measurable effects were observed. A third trial was run last summer. The intention here to remove the skip option for interviewers and administer the question to everyone. Also to allow interviewers to ask the question if privacy was assured. The question was still omitted in around 2% of cases. These were a combination of respondent refusals due to discomfort with the topic, as well as cases where interviewers felt the question could not be administered because of the age of the respondent. The order of the response options was also tested; no measurable effects were observed.

10. Quantitative trials: Omnibus trial 4  Nov/Dec/Jan 2007/8Computer Assisted Personal Interviewing (CAPI: concealed showcards) Which of the options on this card best describes how you think of yourself? Please just read out the letter next to the description. D. Heterosexual / Straight T. Gay/Lesbian V. Bisexual M. Other (No PNTS) (Spontaneous DK/ Refusal) Looking at this card, which of these do you consider yourself to be? Please just read out the letter next to the description. K. Heterosexual / Straight N. Gay/Lesbian Z. Bisexual Q. Other (No PNTS) (Spontaneous DK/ Refusal) However, so far we had only been able to test the question administration to one adult in the household, but the IHS interviews all household members, sometimes together. Therefore a fourth Omnibus trial was run, with three aims: to test in a pseudo concurrent CAPI environment, interviewing all adult household members. This was done using a selection of concealed show cards, a different card being given to each household member; interviewers were asked to key in the letter associated with the option the respondent read out. to test the wording of the question stem, using a split sample Question wording was changed with the word �homosexual� dropped. to test the question without a specific �prefer not to say� option, although the usual spontaneous DK and refusal codes were available to interviewers. However, so far we had only been able to test the question administration to one adult in the household, but the IHS interviews all household members, sometimes together. Therefore a fourth Omnibus trial was run, with three aims: to test in a pseudo concurrent CAPI environment, interviewing all adult household members. This was done using a selection of concealed show cards, a different card being given to each household member; interviewers were asked to key in the letter associated with the option the respondent read out. to test the wording of the question stem, using a split sample Question wording was changed with the word �homosexual� dropped. to test the question without a specific �prefer not to say� option, although the usual spontaneous DK and refusal codes were available to interviewers.

11. Quantitative trials: Omnibus trial 4  Nov/Dec/Jan 2007/8Computer Assisted Personal Interviewing (CAPI: concealed showcards) We found that: The concealed show card approach was successful from the interviewer�s perspective. There were no significant differences in the estimates produced from the two question stems; and The effect on the data of removing the PNTS option was to increase the proportion reporting heterosexual. Significantly, the item non-response rate dropped to around 1%. We found that: The concealed show card approach was successful from the interviewer�s perspective. There were no significant differences in the estimates produced from the two question stems; and The effect on the data of removing the PNTS option was to increase the proportion reporting heterosexual. Significantly, the item non-response rate dropped to around 1%.

12. Qualitative Research: Focus groups Six single-sex groups: heterosexual; gay/lesbian; bisexual. (52 participants) Primary sample criteria: age; sex; sexual identity; household type. Secondary sample criteria: educational attainment; ethnic group/religion. Age: 19-72. Location: London. Focus groups with members of the public informed the subject matter and methodology for later stages, exploring conceptualisation, language, acceptability and administration. Six groups were run in London with: heterosexual men; heterosexual women; gay men, gay/lesbian women; bisexual men and bisexual women. The research team aimed to include people across a range of age groups and household compositions as well as educational attainment and ethnic group.Focus groups with members of the public informed the subject matter and methodology for later stages, exploring conceptualisation, language, acceptability and administration. Six groups were run in London with: heterosexual men; heterosexual women; gay men, gay/lesbian women; bisexual men and bisexual women. The research team aimed to include people across a range of age groups and household compositions as well as educational attainment and ethnic group.

13. Qualitative research: Focus groups General acceptance of question. Confidentiality a concern. Not always understanding purpose. Location: alongside similar questions. Objections to proxy responses. Some difficulties with conceptualising sexual identity. A detailed report is available on the ONS web-site. Overall there was no strong reluctance to answer the question from any of the groups, although assured anonymity and trust in data security were key to response. Whether sexual minority groups would answer accurately varied depending on whether they were �out� in their home environment. This was strongly related to age and whether they had found a community, which helped them express their identity and gave them confidence. There was a general lack of understanding as to why the questioning was necessary even among LGB groups. Participants felt more comfortable if the question was placed alongside similar questions about national identity, ethnic group and religion. Setting the context with other identity questions helped to explain purpose. Proxy responses were an issue and all of the groups objected to others in the household providing a response on their behalf or vice versa. They said it was an individuals right to provide this information for themselves and since it was obvious that the quality of the data would be affected by asking for proxy information, they said it would affect how they thought about the quality of the organisation collecting the data. From the analysis it was apparent that some people had difficulty with the concept of sexual identity, rather than other aspects of sexual orientation such as attraction or behaviour. For heterosexuals it was not something that they had necessarily ever thought about before, so did not always have a ready made answer to the question. Because of this we referred to heterosexuals as �latent identifiers�. Sexual minority groups were much more likely to have considered the concept and found it easier to answer the question. However, some bisexual participants were reluctant to categories themselves. We have referred to gay and lesbian participants as �conscious identifiers�. Bisexuals participants can be spilt into �reluctant� and �political� identifiers, and women sometimes as �fluid� identifiers, since their identity would change depending on time and context. Although bisexual participants included those who felt little sense of sexual identity and had reservations about being categorised, they said they would choose �bisexual� if none of the other options were appropriate, in order to be counted. A detailed report is available on the ONS web-site. Overall there was no strong reluctance to answer the question from any of the groups, although assured anonymity and trust in data security were key to response. Whether sexual minority groups would answer accurately varied depending on whether they were �out� in their home environment. This was strongly related to age and whether they had found a community, which helped them express their identity and gave them confidence. There was a general lack of understanding as to why the questioning was necessary even among LGB groups. Participants felt more comfortable if the question was placed alongside similar questions about national identity, ethnic group and religion. Setting the context with other identity questions helped to explain purpose. Proxy responses were an issue and all of the groups objected to others in the household providing a response on their behalf or vice versa. They said it was an individuals right to provide this information for themselves and since it was obvious that the quality of the data would be affected by asking for proxy information, they said it would affect how they thought about the quality of the organisation collecting the data. From the analysis it was apparent that some people had difficulty with the concept of sexual identity, rather than other aspects of sexual orientation such as attraction or behaviour. For heterosexuals it was not something that they had necessarily ever thought about before, so did not always have a ready made answer to the question. Because of this we referred to heterosexuals as �latent identifiers�. Sexual minority groups were much more likely to have considered the concept and found it easier to answer the question. However, some bisexual participants were reluctant to categories themselves. We have referred to gay and lesbian participants as �conscious identifiers�. Bisexuals participants can be spilt into �reluctant� and �political� identifiers, and women sometimes as �fluid� identifiers, since their identity would change depending on time and context. Although bisexual participants included those who felt little sense of sexual identity and had reservations about being categorised, they said they would choose �bisexual� if none of the other options were appropriate, in order to be counted.

14. Data Collection Instrument Integrated Household Survey (IHS) the preferred data collection instrument. 278,000 individual adult interviews per year. National estimates available in year 1. Sub-national estimates within 3-5 years. Based on the work so far we came to the conclusion that the Census was probably not the appropriate data collection vehicle at this time, and believed that better quality estimates could be provided through inclusion on the Integrated Household Survey being developed in the UK, more accurately, quicker, and more regularly. Due to the size of the IHS, which will achieve around 278,000 adult interviews, and assuming a 2% LGB prevalence rate, we would be able to produce national figures by sex, age, and multiple equality characteristics in one year, with sub-national data available for the majority of local authorities within 3-5 years, to an acceptable level of precision. So the development work focussed on a social survey context. Based on the work so far we came to the conclusion that the Census was probably not the appropriate data collection vehicle at this time, and believed that better quality estimates could be provided through inclusion on the Integrated Household Survey being developed in the UK, more accurately, quicker, and more regularly. Due to the size of the IHS, which will achieve around 278,000 adult interviews, and assuming a 2% LGB prevalence rate, we would be able to produce national figures by sex, age, and multiple equality characteristics in one year, with sub-national data available for the majority of local authorities within 3-5 years, to an acceptable level of precision. So the development work focussed on a social survey context.

15. Quantitative trials: General Lifestyle Survey Longitudinal Household survey (CAPI) April 2008 � ongoing Face-to-Face (Concealed showcards) �Which of the options on this card best describes how you think of yourself? Please just read out the number next to the description� 27. Heterosexual / Straight 21. Gay / Lesbian 24. Bisexual 29. Other (Spontaneous DK/ Refusal) Telephone �I will now read out a list of terms people sometimes use to describe how they think of themselves.� Heterosexual or Straight Gay or Lesbian Bisexual Other (Spontaneous DK/ Refusal) �As I read the list again, please say �yes� when you hear the option that best describes how you think of yourself.� Our prototype questions are currently being piloted on the General lifestyle Survey. This is the first time that we�ve been able to test the question in a real concurrent interview, both face-to-face and over the telephone. In the F-t-F interview, if more than one person was being interviewed, each person would get a different card with the same response options but different numbers associated with those options. Only half the sample are receiving the question in order to carry out a final test of the effect on response in a concurrent interview, and the agree to recall question will be used to inform any potential effect on attrition. We don�t have data from this pilot yet. Our prototype questions are currently being piloted on the General lifestyle Survey. This is the first time that we�ve been able to test the question in a real concurrent interview, both face-to-face and over the telephone. In the F-t-F interview, if more than one person was being interviewed, each person would get a different card with the same response options but different numbers associated with those options. Only half the sample are receiving the question in order to carry out a final test of the effect on response in a concurrent interview, and the agree to recall question will be used to inform any potential effect on attrition. We don�t have data from this pilot yet.

16. Qualitative research: Cognitive/In-depth interviews Around 30 cognitive/in-depth interviews. Sampled purposively to achieve breadth and diversity. Sampling strategy compliments that used for focus groups. Sub-groups include: transgender people, youngest and oldest age-groups, lower education attainment, rural communities, particular ethnic minority and faith groups. We are currently conducting one-to-one cognitive/in-depth interviews to further explore the response process. Small sub-groups that could not form part of the focus group work are covered at this stage. For example, the transgender community (even though this relates to gender and not sexual identity we still need to be sure that the question can be answered), those in the very youngest and oldest age-groups, those with lower educational attainment, living in more rural communities and in other parts of the country and particular ethnic minority, and religious groups. Therefore, the results from the focus group work need to be considered in conjunction with the one-to-one interviews, since the sampling strategies compliment each other. Initial findings support those from the previous tests. Although the question is a potential issue with some faith groups where homosexuality is not accepted. In particular, those from the Muslim community. Whilst they are not against the questioning per se, it is the administration of the question within a concurrent household environment that is the concern. That is how they would be perceived by other household members when answering the question. And in contrast to the focus groups, one 16 year old preferred the term �homosexual� to the term �gay� which she thought of as a derogatory term. Among young people in the UK, the word �gay� has become synonymous with �not very good� or �uncool�. And we are debating the use of the �other� category. This is sometimes used by bisexual and transgender people who are anti-categorisation, but more often by heterosexuals due to comprehension issues. Those who use this category will in effect be excluded from the analysis, although numbers are very small. With this is mind, we also think that further thought be given to an �asexual� category. We are currently conducting one-to-one cognitive/in-depth interviews to further explore the response process. Small sub-groups that could not form part of the focus group work are covered at this stage. For example, the transgender community (even though this relates to gender and not sexual identity we still need to be sure that the question can be answered), those in the very youngest and oldest age-groups, those with lower educational attainment, living in more rural communities and in other parts of the country and particular ethnic minority, and religious groups. Therefore, the results from the focus group work need to be considered in conjunction with the one-to-one interviews, since the sampling strategies compliment each other. Initial findings support those from the previous tests. Although the question is a potential issue with some faith groups where homosexuality is not accepted. In particular, those from the Muslim community. Whilst they are not against the questioning per se, it is the administration of the question within a concurrent household environment that is the concern. That is how they would be perceived by other household members when answering the question. And in contrast to the focus groups, one 16 year old preferred the term �homosexual� to the term �gay� which she thought of as a derogatory term. Among young people in the UK, the word �gay� has become synonymous with �not very good� or �uncool�. And we are debating the use of the �other� category. This is sometimes used by bisexual and transgender people who are anti-categorisation, but more often by heterosexuals due to comprehension issues. Those who use this category will in effect be excluded from the analysis, although numbers are very small. With this is mind, we also think that further thought be given to an �asexual� category.

17. Conclusion Provide an acceptable measure of self-perceived sexual identity for examining relative disadvantage at national and sub-national level. A single precise count of LGB population unlikely. Combination of qualitative and quantitative research techniques has been effective. So, who would have thought that one question could have produced quite so much work. From the research we acknowledge that we are unlikely to achieve a single precise count of the LGB population, either from a survey as large as the IHS, or from the Census, where the proxy issue would be more of a concern and privacy issues would be even more apparent due to its administration. Yet, we have shown that the question can be understood and is acceptable to most people if they feel safe proving the information. And that the estimates achieved are relatively stable and can be used for examining disadvantage. And, I believe that by taking a more holistic approach to the development work we have provided the basis on which data can be interpreted and used effectively by analysts. (But it is the difficulty of translating the complex umbrella concept of sexual orientation, comprising different dimensions, yet having difference salience and importance to different groups, and who use language differently in subtle ways, into a concept that can be measured in a social survey environment, that has led to this amount of work.) So, who would have thought that one question could have produced quite so much work. From the research we acknowledge that we are unlikely to achieve a single precise count of the LGB population, either from a survey as large as the IHS, or from the Census, where the proxy issue would be more of a concern and privacy issues would be even more apparent due to its administration. Yet, we have shown that the question can be understood and is acceptable to most people if they feel safe proving the information. And that the estimates achieved are relatively stable and can be used for examining disadvantage. And, I believe that by taking a more holistic approach to the development work we have provided the basis on which data can be interpreted and used effectively by analysts. (But it is the difficulty of translating the complex umbrella concept of sexual orientation, comprising different dimensions, yet having difference salience and importance to different groups, and who use language differently in subtle ways, into a concept that can be measured in a social survey environment, that has led to this amount of work.)

18. Finally� Acknowledgements Data Collection Methodology Branch Amanda Wilmot, Peter Betts and Tamara Taylor. Social Analysis and Reporting Division: Madhavi Bajekal and Joe Traynor. Contact information Sexual identity project website: www.ons.gov.uk/about-statistics/measuring-equality/sexual-identity-project Sexual identity project email: sexual.identity@ons.gov.uk