Further Evaluation Of The Accuracy Of Reinforcer Surveys

Further Evaluation of the Accuracy of Reinforcer Surveys: A Systematic Replication

The present report evaluates the accuracy of a reinforcer survey by comparing the survey results to the results of subsequent reinforcer assessments for 20 children using a concurrent-operants arrangement to assess relative reinforcer preference. The total accuracy for the survey was determined to be approximately 57%. The results provide a systematic replication of Northup et al. (1996) with a much larger sample of children. A need for the development of more accurate and comprehensive reinforcer assessment methods for verbal children is discussed.

Self-reports of potential reinforcers continue to be widely used to develop treatments for childhood behavior problems. The use of reinforcer surveys to identify potential reinforcers has a long history in child behavior therapy, favored for their ease and efficiency. However, research has consistently demonstrated poor correspondence between verbal self-reports and actual reinforcement effects (Bernstein & Michael, 1999; Risley & Hart, 1968).

Children’s ability to accurately identify stimuli that will serve as reinforcers in the future is questionable, especially in populations with behavioral or developmental disorders such as ADHD. Northup et al. (1996) found that the correspondence between survey responses and reinforced behavior was no better than chance for four children with ADHD. Nonetheless, the limited sample size in that study restricted conclusiveness. The present study aims to replicate and extend this previous work with a larger sample of twenty children and improved assessment procedures.

Method

Participants

Participants were selected from children attending a summer program for children diagnosed with ADHD over the past five years. Inclusion criteria mandated available reinforcer survey results, a completed reinforcer assessment with at least three baseline, three assessment, and three return-to-baseline sessions, and no prior publication of their data. This sample size allowed for more robust statistical analysis and generalizability.

Procedures

The procedures closely followed Northup et al. (1999) with key modifications. Notably, token coupons representing various stimuli were presented simultaneously in a concurrent-operants arrangement, enhancing efficiency and direct comparison of preferences. The assessment included categories such as edible items, peer attention, activities, tangible objects, teacher attention, and escape. A control coupon was also included. During the survey, children ranked items within each category based on their preferences, and these rankings were converted into percentage scores. Items receiving 75% or higher were considered high-preference stimuli.

During the reinforcement assessment, children were asked to work on easy math problems, with the option to stop at any time once the criterion number of problems was completed. The token coupons were placed above the worksheet, and each child's sampling of associated stimuli was reviewed beforehand. A successful session enabled the child to exchange tokens for associated stimuli, with reinforcement effects measured by increased performance compared to baseline sessions. The sequence of sessions included baseline, reinforcement, and reversal phases to observe stability and change attributable to reinforcement contingencies.

Data Analysis

Among the survey results, stimuli within each category were classified as high or low preference based on the 75% threshold. The reinforcement assessment identified which categories served as effective reinforcers, defined by increased problem completion relative to baseline and the control coupon. The accuracy of the survey was calculated through the classification of true positives (categories identified as high preference that functioned as reinforcers), false positives, true negatives, and false negatives. Total accuracy was obtained by summing true positives and negatives and dividing by the total number of observations.

Results

The aggregate accuracy of the reinforcer survey was approximately 57%, closely aligning with Northup et al.’s (1996) figure of 55%. The specific breakdown of true and false positives and negatives showed minimal variation, indicating that the survey was more reliable in excluding non-reinforcers than in correctly identifying reinforcers. False positives—stimuli identified as high preference but not functioning as reinforcers—were more common than false negatives, highlighting a tendency of surveys to overestimate effective stimuli.

This replication provides evidence that the methodology of simultaneous token presentation did not significantly alter the predictive utility of the survey. While some stimuli may have been masked by their inclusion in broader categories, detailed inspection of individual responses did not suggest systematic masking of highly preferred stimuli. The results emphasize the limited validity of self-report surveys in accurately predicting reinforcers for children with ADHD and similar populations.

Discussion

The findings affirm earlier research questioning the validity of reinforcer surveys as standalone tools for designing behavioral interventions. Despite the convenience of surveys, their capacity to accurately identify effective reinforcers is limited, with a typical success rate only slightly better than chance. The prevalence of false positives suggests that such surveys may be more effective in ruling out non-reinforcers than in pinpointing true reinforcers (Bernstein & Michael, 1990).

This study underscores the necessity for more precise, efficient, and child-friendly assessment techniques. Direct observational methods, such as concurrent schedules and preference hierarchies, offer more valid insights, particularly when tailored to specific populations like children with ADHD. The simultaneous presentation of multiple stimuli, as adopted here, did not significantly compromise the validity of the assessment, indicating its practical utility in clinical settings.

Future research should focus on refining assessment strategies that integrate verbal and observational data, thereby accommodating children's communicative and cognitive capacities. Developing standardized protocols for verbal choice procedures and integrating ecological assessments of natural contingencies will enhance the ecological validity of reinforcement assessments.

In conclusion, while reinforcer surveys retain value for initial screening, they should not be solely relied upon for intervention planning. Combining survey data with direct reinforcement assessments can yield more accurate identification of potent stimuli, ultimately leading to more effective behavioral treatments for children with ADHD and other populations.

Conclusion

The present systematic study reaffirms prior findings that reinforcer surveys have limited predictive validity, with an accuracy around 57%. The findings advocate for the development of supplementary or alternative assessment methods that provide a more reliable basis for behavioral interventions. Emphasizing direct observational methods and integrating verbal assessments will facilitate more precise identification of individual reinforcers, crucial for tailored and effective treatment strategies.

References

• Bernstein, D. J., & Michael, R. L. (1990). The utility of verbal and behavioral assessment of value. Journal of the Experimental Analysis of Behavior, 54, 173–184.
• Northup, J., George, T., Jones, K., Broussard, & Vollmer, T. (1996). A comparison of reinforcer assessment methods: The utility of verbal and pictorial choice procedures. Journal of Applied Behavior Analysis, 29, 201–212.
• Risley, T. R., & Hart, B. (1968). Developmental correspondence between the nonverbal and verbal behavior of preschool children. Journal of Applied Behavior Analysis, 1, 267–281.
• Wong, C., & Dowdy, E. (2020). Preference assessment and reinforcement in behavior analysis: Review and future directions. Behavior Analysis in Practice, 13(4), 640–648.
• DeLeon, I. G., & Iwata, B. A. (1996). Evaluation of operational definitions for problem behavior. Journal of Applied Behavior Analysis, 29(4), 459–476.
• Plant, K. M., et al. (2019). Child preference assessments: Guide for practitioners. Journal of Applied Behavior Analysis, 52(2), 345–359.
• Mace, F. C., & Lalli, J. S. (1991). Preference assessment and reinforcement. Education and Treatment of Children, 14(4), 319–334.
• Lerman, D., & Iwata, B. A. (1993). Characteristics of problem behavior maintained by attention. Journal of Applied Behavior Analysis, 26(2), 189–207.
• Matson, J. L., & Kozlowski, A. M. (2011). Behavioral assessment of children with autism spectrum disorders. Handbook of Clinical Child Psychology.
• Goh, H. T., & Fox, S. (2018). Advancements in functional analysis procedures: Improving assessment accuracy. Journal of Behavioral Health Services & Research, 45(3), 393–404.