HINTS Instrument Development and Validation Procedures
Construct and Item Development
NCI gives careful consideration to the content and quality of HINTS instruments. Each HINTS instrument includes both a set of core items / constructs that are a part of each data collection cycle and can be trended over time, as well as items reflecting rotating or new constructs of interest. Before each HINTS cycle, the HINTS Management Team identifies a “champion” with content expertise for each construct. The champion takes responsibility for identifying the survey items to be fielded for that construct. Many constructs include a combination of core and emergent content. Each champion establishes a working group of NCI staff and/or outside experts to review the literature relevant to their construct and propose a pool of items to field. Champions note whether the proposed items have been fielded previously on HINTS, have been fielded on another survey, or were newly developed. The HINTS Management Team decides how many items can be tested and ultimately fielded for each construct based on their assessment of NCI's priorities and target survey length. Champions are encouraged to use existing items where possible—from a previous version of HINTS or from another federal survey or validated instrument source. Newly proposed items undergo expert review and cognitive testing.
Expert Review
Each champion submits their proposed items to Westat, the contracting organization assisting NCI with questionnaire development, question testing, and survey fielding. Westat conducts an expert review of the proposed items, recommending the order in which they should be fielded and identifying design-related issues that may make them difficult to answer or incompatible with other HINTS content. For example, a proposed item that was previously fielded in an interviewer-administered survey will need to be adapted to the HINTS self-administered format. Westat works carefully with each champion on their proposed items to make them suitable for HINTS.
Cognitive Testing
Following the expert review, Westat develops a draft instrument for cognitive testing. Cognitive testing focuses on the survey content that is new to HINTS. Items which have been included on prior HINTS cycles are not typically tested because they underwent testing prior to the cycle in which they were introduced. However, pre-existing items are sometimes fielded during testing as context for the new items under evaluation. For example, many existing items serve as filter questions that determine whether a respondent should answer subsequent items based on that topic.
Cognitive testing involves using in-depth, semi-structured interviews with respondents that match the HINTS population to gather insights into the cognitive sources of potential misinterpretation of survey items.1 This qualitative methodology generates information about how respondents comprehend questions, retrieve relevant information to answer, form summary judgments, and select the appropriate response options. By applying cognitive testing methods to draft HINTS questionnaires, NCI and Westat are able to assess how well respondents understand certain words, phrases, and concepts used in the survey items as well as assess any difficulties they have answering the questions, helping to ensure the items are valid and useful.
Respondent recruitment for participation in HINTS cognitive testing has evolved over the years. In early testing, respondents were recruited through advertisements in community periodicals and through lists of respondents who had participated in Westat testing in the past. With the increasing use of the internet and the elimination of local papers, Westat started to recruit online through Craig’s List. Efforts are made to recruit a demographically diverse pool of respondents based on age, sex, race, and education. In some cycles, respondents with specific characteristics relevant to survey constructs (for example, caregivers or cancer survivors) are deliberately recruited to test items that would not be relevant to the general population.
Interviews are conducted by Westat interviewers trained in cognitive testing methodology and survey design. The interviewer observes the respondent as they answer the survey and asks probing questions to identify points of confusion or misunderstandings. HINTS cognitive interviews typically involve a combination of concurrent and retrospective probing to gather information from participants. Participants are also asked to read aloud and encouraged to share any thoughts they have about the questions as they answer them. Respondents receive a monetary incentive for their participation in testing. Each test typically lasts 90 minutes.
Cognitive testing has been conducted on each HINTS instrument since 2001. The format of the testing has changed over time corresponding with changes to the HINTS mode of administration. For HINTS 1-2 (fielded only over the telephone), testing was conducted in-person with a cognitive interviewer administering the survey questions and follow-up probing verbally, without the respondent seeing a written instrument. Starting with HINTS 3, testing was conducted using a self-administered version of the survey to reflect the change from telephone to paper-and-pencil survey mode. Starting with HINTS 6, when mixed mode (web and paper) was established as the main data collection methodology, HINTS incorporated online video conferencing for cognitive testing of the web instrument.
HINTS cognitive testing has usually been done in two rounds, allowing an opportunity to revise and re-test questions that underperformed in the first round. After the first round, Westat provides NCI with a written report summarizing the performance of each tested item and recommendations for improving (or eliminating) them. Construct champions then review the recommendations and make decisions on how to revise their content. The second round of testing identifies whether the revised content has improved. Following the second round of testing, Westat provides a final report that reflects all the issues identified, issues resolved, and final recommendations for fielding. Champions decide whether or not to accept the report recommendations, with the HINTS Management Team ultimately deciding which items will be fielded.
Final Instrument Development
Following cognitive testing, Westat works with Champions on the final decisions for their items. This typically involves selecting the items that performed best in cognitive testing and deciding where they should be placed in the final instrument. Additional minor revisions may be made to ensure that the individual constructs all blend together into a logical and comprehensive survey. The HINTS Management Team makes all final decisions and ensures that the instrument can be completed by respondents in 30 minutes.
The table below provides additional information about each round of HINTS cognitive testing.
|
| Approval Date |
Number |
|
HINTS 1
|
Jan 16, 2002 |
MPA M-1531 |
Spring 2002 |
Round 1: 9
Round 2: 9 |
In person |
Recruitment information unavailable |
Incentive amount unavailable |
|
HINTS 2
|
Apr 8, 2004 |
7845 |
Spring 2005 |
Round 1: 9
Round 2: 12 |
In person |
Recruitment information unavailable |
Incentive amount unavailable |
|
HINTS 3
|
Apr 26, 2006 |
8137.04 |
Dec 12, 2006 – Feb 9, 2007 |
Round 1: 9
Round 2: 9
Round 3: 9 |
In person |
Recruitment information unavailable |
Incentive amount unavailable |
|
HINTS 4, Cycle 1
|
Dec 23, 2010 |
8861.01.04 |
Mar 9 – 25, 2011 |
English: 33
Spanish: 17 |
In person |
Craig’s List, ads in local papers, existing lists |
$50 cash |
|
HINTS 4, Cycle 2
|
Dec 27, 2011 |
8861.01.04 |
Mar – Apr 2012 |
English: 22
Round 1: 13
Round 2: 9
Spanish: 15
|
In person |
Craig’s List, ads in local papers, existing lists |
$50 cash |
|
HINTS 4, Cycle 3
|
Nov 26, 2012 |
8861.01.04 |
Jan 2013 – Mar 20, 2013 |
English: 20
Round 1: 10
Round 2: 10
Spanish: 10
|
In person |
Craig’s List, ads in local papers, existing lists |
$50 cash |
|
HINTS 4, Cycle 4
|
Nov 15, 2013 |
8861.01.04 |
Nov – Dec 2013 |
Round 1: 15
Round 2: 15 |
In person |
Craig’s List, ads in local papers, existing lists |
$50 cash |
|
HINTS-FDA
|
Mar 4, 2013 |
8861.01.04 |
Dec 2013 – Jan 2014 |
English: 24
Round 1: 12
Round 2: 12
Spanish: 10
|
In person |
Craig’s List, ads in local papers, existing lists |
$50 cash |
|
HINTS 5, Cycle 1
|
Mar 3, 2016 |
6048.14 |
Apr 18-29 and
Jun 24-7, 2016 |
Round 1: 15
Round 2: 15 |
In person |
Craig’s List, existing lists |
$50 cash |
|
HINTS 5, Cycle 2
|
Apr 27, 2017 |
6048.14 |
May 2-11 and
Jul 13-26, 2017 |
Round 1: 15
Round 2: 15 |
In person |
Craig’s List, existing lists |
$50 cash |
|
HINTS 5, Cycle 3
|
Apr 26, 2018 |
6048.14 |
May 1-11 and
Jul 16-27, 2018 |
Round 1: 15
Round 2: 15 |
In person |
Craig’s List, existing lists |
$50 cash |
HINTS 5, Cycle 4
|
Apr 16, 2019 |
6048.14 |
Apr 22-29 and
Aug 2-23, 2019 |
Round 1: 15
Round 2: 15 |
In person |
Craig’s List, existing lists |
$50 cash |
|
HINTS 6
|
May 10, 2021 |
6632.03.51 |
May 17-28 and
Aug 2-13, 2021 |
Round 1: 15
Round 2: 15 |
Online |
Craig’s List, existing lists |
$50 Amazon gift card |
|
HINTS 7
|
May 2, 2023 |
6632.05.73 |
May 9-24 and
Jul 14-26, 2023 |
Round 1: 15
Round 2: 15 |
Online and In person |
Craig’s List, existing lists, local connections |
$75 cash or $75 Amazon gift card |