HINTS Instrument Development and Validation Procedures
Construct and Item Development
NCI gives careful consideration to the content and quality of HINTS instruments. Each HINTS instrument includes both a set of core items (constructs that are a part of each data collection cycle and can be used for trend analyses), as well as items reflecting new constructs of interest. Before each HINTS cycle, the HINTS Management Team identifies a “champion” with content expertise for each construct. The champion takes responsibility for identifying the survey items to be fielded for that construct. Each champion establishes a working group of NCI staff and/or outside experts to review the literature relevant to their construct and propose a pool of items to field. Champions note whether the proposed items have been fielded previously on HINTS, have been fielded on another survey, or were newly developed. The HINTS Management Team decides how many items can be tested and ultimately fielded for each construct based on their assessment of NCI priorities and target survey length. Champions are encouraged to use existing items where possible—from a previous version of HINTS or from another federal survey or validated instrument source. Newly proposed items undergo expert review and cognitive testing.
Expert Review
NCI staff and survey methodologists from the supporting contractor (currently Westat) conduct an expert review of the proposed items, recommending the order in which they should be fielded and identifying design-related issues that may make them difficult to answer or incompatible with other HINTS content. For example, a proposed item that was previously fielded in an interviewer-administered survey would need to be adapted to the HINTS self-administered survey format. The contractor works carefully with each champion on their proposed items to make them suitable for HINTS.
Cognitive Testing
Following the expert review, the contractor develops a draft instrument for cognitive testing. Cognitive testing focuses on the survey content that is new to HINTS. Items that have been included on prior HINTS cycles are not typically tested because they underwent testing prior to the cycle in which they were introduced. However, pre-existing items are sometimes fielded during testing to provide context for the new items under evaluation. For example, many existing items serve as filter questions that determine whether a respondent should answer subsequent items related to that topic.
Cognitive testing involves using in-depth, semi-structured interviews with respondents to gather insights into the cognitive sources of potential misinterpretation of survey items.1 This qualitative methodology generates information about how respondents comprehend questions, retrieve relevant information to answer the questions, form summary judgments, and select response options. By applying cognitive testing methods to draft HINTS questionnaires, NCI and the contractor can assess how well respondents understand certain words, phrases, and concepts used in the survey items and identify any difficulties they have answering the questions. This helps to ensure the items on HINTS are valid and useful.
Efforts are made to recruit a demographically diverse pool of respondents for cognitive testing based on age, sex, race, and education. In some cycles, respondents with specific characteristics (for example, caregivers or cancer survivors) are deliberately recruited when items that would be particularly relevant to that group are being tested.
Interviews are conducted by contractors trained in cognitive testing methodology and survey design. The interviewer observes the respondent as they answer the survey and asks follow-up questions (cognitive probes) to identify points of confusion or misunderstanding. HINTS cognitive interviews typically involve a combination of concurrent and retrospective probing to gather information from participants. Participants are also asked to read aloud and encouraged to share any thoughts they have about the questions as they answer them. Respondents receive a monetary incentive for their participation in testing. Each test typically lasts 90 minutes.
Cognitive testing has been conducted for each HINTS instrument. The format of the testing has changed over time, corresponding with changes to the HINTS mode of administration. For HINTS 1 and 2 (which were fielded over the telephone), testing was conducted in-person with a cognitive interviewer administering the survey questions and follow-up probing verbally, without the respondent seeing a written instrument. Starting with HINTS 3, testing was conducted using a self-administered version of the survey to reflect the change from telephone to paper-and-pencil survey mode. Starting with HINTS 6, when mixed mode (web and paper) was established as the main data collection methodology, HINTS incorporated online video conferencing to conduct cognitive testing.
HINTS cognitive testing is usually done in two rounds, allowing an opportunity to revise and re-test questions that underperform initially. After the first round, the contractor provides NCI with a written report summarizing the performance of each tested item and recommendations for improving (or eliminating) the items. Construct champions then review the recommendations and make decisions on how to revise their items. The second round of testing assesses whether the revised items perform better. Following the second round of testing, the contractor provides a final report that reflects all the issues identified, issues resolved, and final recommendations for fielding. Champions decide whether or not to accept the report’s recommendations, with the HINTS Management Team ultimately deciding which items are fielded.
Final Instrument Development
Following cognitive testing, the contractor works with champions on the final decisions for their items. This typically involves selecting the items that performed best in cognitive testing and deciding where they should be placed in the final instrument. Additional minor revisions may be made to ensure that the individual constructs all work together in a logical and comprehensive survey. The HINTS Management Team makes all final decisions and ensures that the instrument can be completed by respondents in approximately 30 minutes.
Please contact the HINTS team if you need additional information about cognitive testing.