Background: The ability to draw accurate inferences from research depends heavily on the quality and representativeness of research samples. Research samples in the social sciences, including communication, are frequently criticized for being small and unrepresentative, yet there is substantial variation in sample characteristics. Objective: The objective of this project was to undertake a systematic examination of the characteristics of human samples used in communication research in major communication journals, in order to respond to the criticisms that such samples are small, underpowered, and lacking in external validity. Method: To ascertain the status of human samples in communication research, this project examined every empirical study published between 2010 and 2019 in four top communication journals—Communication Monographs, Communication Research, Human Communication Research, and Journal of Communication—that reported data from human subjects. The data set included 1,264 individual studies and a total sample size of 932,060 participants. Results and Conclusion: Sample sizes ranged from 10 to 57,847 participants, with an average of 740.12 participants, and were larger for non-experiments than experiments, quantitative than qualitative studies, and secondary than primary data analyses. Ninety-four countries were represented in the samples, although more than 70% of samples were recruited exclusively from the United States. Compared to U. S. demographics, such studies oversampled younger participants, female participants, and white participants.
| Published in | Communication and Linguistics Studies (Volume 9, Issue 2) |
| DOI | 10.11648/j.cls.20230902.12 |
| Page(s) | 27-41 |
| Creative Commons |
This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited. |
| Copyright |
Copyright © The Author(s), 2023. Published by Science Publishing Group |
Samples, Representativeness, Statistical Power
| [1] | Afifi, W. A., & Cornejo, M. (2020). #CommSoWEIRD: The question of sample representativeness in interpersonal communication research. In M. L. Doerfel & J. L. Gibbs (Eds.), Organizing inclusion: Moving diversity from demographics to communication processes (pp. 238–259). Routledge. https://doi.org/10.4324/9780429450495 |
| [2] | Altman, D. G., & Bland, J. M. (2005). Standard deviations and standard errors. British Medical Journal, 331 (7521), 903. https://doi.org/10.1136/bmj.331.7521.903 |
| [3] | Andrew, S. (2020, June 30). The US has 4% of the world’s population but 25% of its coronavirus cases. CNN Health. https://www.cnn.com/2020/06/30/health/us-coronavirus-toll-in-numbers-june-trnd/index.html |
| [4] | Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63 (7), 602–614. https://doi.org/10.1037/0033-066X.63.7.602 |
| [5] | Bates, B. (2021). Making communication scholarship less WEIRD. Southern Communication Journal, 86 (1), 1–4. https://doi.org/10.1080/1041794X.2020.1861078 |
| [6] | Carminati, L. (2018). Generalizability in qualitative research: A tale of two traditions. Qualitative Health Research, 28 (13), 2094–2101. https://doi.org/10.1177/1049732318788379 |
| [7] | Centers for Disease Control and Prevention. (2022). National Health and Nutrition Examination Survey. National Center for Health Statistics. https://www.cdc.gov/nchs/nhanes/index.htm |
| [8] | Chakravartty, P., Kuo, R., Grubbs, V., & McIlwain, C. (2018). #CommunicationSoWhite. Journal of Communication, 68, 254–266. https://doi.org/10.1093/joc/jqy003 |
| [9] | Coleman, R. (2018). Designing experiments for the social sciences. Sage. |
| [10] | Coles, R. L. (2002). Black single fathers: Choosing to parent full-time. Journal of Contemporary Ethnography, 31 (4), 411–439. https://doi.org/10.1177/0891241602031004002 |
| [11] | Combs, J. G. (2010). Big samples and small effects: Let’s not trade relevance and rigor for power. Academy of Management Journal, 53 (1), 9–13. https://doi.org/10.5465/amj.2010.48036305 |
| [12] | Davern, M. E. (2008). Representative sample. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (pp. 720–722). Sage. |
| [13] | Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013). Influence of trial sample size on treatment effect estimates: Meta-epidemiological study. British Medical Journal, 346. https://doi.org/10.1136/bmj.f2304 |
| [14] | De Winter, J. C. (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research, and Evaluation, 18 (1), Article 10. https://doi.org/10.7275/e4r6-dj05 |
| [15] | Emerson, R. W. (2015). Convenience sampling, random sampling, and snowball sampling: How does sampling affect the validity of research? Journal of Visual Impairment & Blindness, 109 (2), 164–168. https://doi.org/10.1177/0145482X151900215 |
| [16] | Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine. |
| [17] | Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26 (3), 213–224. https://doi.org/10.1002/bdm.1753 |
| [18] | Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13 (3), e1002106. https://doi.org/10.1371/journal.pbio.1002106 |
| [19] | Hedt, B. L., & Pagano, M. (2011). Health indicators: Eliminating bias from convenience sampling estimators. Statistics in Medicine, 30 (5), 560–568. https://doi.org/10.1002/sim.3920 |
| [20] | Hendriks, T., Warren, M. A., Schotanus-Dijkstra, M., Hassankhan, A., Graafsma, T. Bohlmeijer, E., & de Jong, J. (2019). How WEIRD are positive psychology interventions? A bibliometric analysis of randomized controlled trials on the science of well-being. The Journal of Positive Psychology, 14 (4), 489–501. https://doi.org/10.1080/17439760.2018.1484941 |
| [21] | Hennink, M., & Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, 114523. https://doi.org/10.1016/j.socscimed.2021.114523 |
| [22] | Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33 (2/3), 61–83. https://doi.org/10.1017/S0140525X0999152X |
| [23] | International Monetary Fund (2018). IMF annual report. https://www.imf.org/external/pubs/ft/ar/2018/eng/ |
| [24] | Johnston, M. P. (2017). Secondary data analysis: A method of which the time has come. Qualitative and Quantitative Methods in Libraries, 3 (3), 619–626. |
| [25] | Kaufmann, R., & Tatum, N. T, (2017). Do we know what we think we know? On the importance of replication in instructional communication research. Communication Education, 66 (4), 479–481. https://doi.org/10.1080/03634523.2017.1342849 |
| [26] | Kraemer, H. C., & Blasey, C. (2016). How many subjects: Statistical power analysis in research. Sage. |
| [27] | Lannutti, P. J., Butauski, M., Rubinsky, V., & Hudak, N. (2021). Setting the agenda: LGBTQ+ and SGM family communication. Journal of Family Communication, 21 (2), 138–143. https://doi.org/10.1080/15267431.2021.1912048 |
| [28] | Levine, T. R., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings. Communication Monographs, 76 (3), 286–302. https://doi.org/10.1080/03637750903074685 |
| [29] | Maleki, F., Ovens, K., McQuillan, I., & Kusalik, A. J. (2019). Size matters: How sample size affects the reproducibility and specificity of gene set analysis. Human Genomics, 13 (Suppl 1), 42. https://doi.org/10.1086/s40246-019-0226-2 |
| [30] | Martino, M. L., Lemmo, D., & Gargiulo, A. (2021). A review of psychological impact of breast cancer in women below 50 years old. Health Care for Women International, 42 (7–9), 1066–1085. https://doi.org/10.1080/07399332.2021.1901901 |
| [31] | McEwan, B., Carpenter, C. J., & Westerman, D. (2018). On replication in communication science. Communication Studies, 69 (3), 235–241. https://doi.org/10.1080/10510974.2018.1464938 |
| [32] | Minarik, T., Berger, B., Althaus, L., Bader, V., Biebl, B., Brotzeller, F., Fusban, T., Hegemann, J., Jesteadt, L., Kalweit, L., Leitner, M., Linke, F., Nabielska, N., Reiter, T., Schmitt, D., Spraetz, A., & Sauseng, P. (2016). The importance of sample size for reproducibility of tDCS effects. Frontiers in Human Neuroscience, 10, 453. https://doi.org/10.3389/fnhum.2016.00453 |
| [33] | Moher, D., Dulberg, C. S., & Wells, G. A. (1994). Statistical power, sample size, and their reporting in randomized controlled trials. Journal of the American Medical Association, 272 (2), 122–124. https://doi.org/10.1001/jama.1994.03520020048013 |
| [34] | Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6 (7), Article e1000097. https://doi.org/10.1371/journal/pmed1000097 |
| [35] | Murphy, K. R., & Aguinis, H. (2019). HARKing: How badly can cherry-picking and question trolling produce bias in published results? Journal of Business and Psychology, 34 (1), 1–17. https://doi.org/10.1007/s10869-017-9524-7 |
| [36] | Muthukrisha, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B. (2020). Beyond Western, educated, industrial, rich, and democratic (WEIRD) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31 (6), 678–701. https://doi.org/10.1177/0956797620916782 |
| [37] | Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716. https://doi.org/10.1126/science.aac4716 |
| [38] | Parsons, A., Turner, R., Ingleton, H., Dubrow-Marshall, L., Kefalogianni, M., Omylinska-Thurston, J., Karkou, V., & Thurston, S. (2021). Flowing towards freedom with multimodal creative therapy: The healing power of therapeutic arts for ex cult-members. The Arts in Psychotherapy, 72, Article 101743. https://doi.org/10.1016/j.aip.2020.101743 |
| [39] | Peterson, R. A., & Merunka, D. R. (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67 (5), 1035–1041. https://doi.org/10.1016/j.busres.2013.08.010 |
| [40] | Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo Sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences, 1145 (45), 11401–11405. https://doi.org/10.1073/pnas.1721165115 |
| [41] | Radler, B. T. (2014). The Midlife in the United States (MIDUS) series: A national longitudinal study of health and well-being. Open Health Data, 2 (1), e3. https://doi.org/10.5334/ohd.ai |
| [42] | Rains, S. A., Levine, T. R., & Weber, R. (2018). Sixty years of quantitative communication research summarized: Lessons from 149 meta-analyses. Annals of the International Communication Association, 42 (2), 105–124. https://doi.org/10.1080/23808985.2018.1446350 |
| [43] | Roser, M., & Ortiz-Ospina, E. (2013, May 25). Global extreme poverty. Our World in Data. https://ourworldindata.org/extreme-poverty |
| [44] | Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2 (2), 107–114. https://doi.org/10.1177/2515245919838781 |
| [45] | Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96 (5), 1055–1064. https://doi.org/10.1037/a0023322 |
| [46] | Soliz, J., & Phillips, K. E. (2018). Toward a more expansive understanding of family communication: Considerations for inclusion of ethnic-racial and global diversity. Journal of Family Communication, 18 (1), 5–12. https://doi.org/10.1080/15267431.2017.1399890 |
| [47] | Statista. (2021, January 20). Median age of the resident population of the United States from 1960 to 2019. https://www.statista.com/statistics/241494/median-age-of-the-us-population/ |
| [48] | Thalmayer, A. G., Toscanelli, C., & Arnett, J. J. (2021). The neglected 95% revisited: Is American psychology becoming less American? American Psychologist, 76 (1), 116–129. https://doi.org/10.1037/amp0000622 |
| [49] | The Economist Intelligence Unit. (2020). Democracy index 2020 in sickness and in health? https://pages.eiu.com/rs/753-RIQ-438/images/democracy-index-2020 |
| [50] | UNESCO (2021). Access. Global Education Monitoring Report. https://www.education-progress.org/en/articles/access |
| [51] | United Nations. (2019). Population. United Nations. https://www.un.org/en/global-issues/population |
| [52] | U.S. Census Bureau. (2019, December 3). Census Bureau reports nearly 77 million students enrolled in U.S. schools. https://www.census.gov/newsroom/press-releases/2019/school-enrollment.html#:~:text=Enrolled%20students%20make%20up%2025%25%20of%20the%20U.S.,Hispanic%2C%2015%25%20black%20alone%20and%205%25%20Asian%20alone. |
| [53] | U. S. Census Bureau. (2019). Quick Facts/United States. https://www.census.gov/quickfacts/fact/table/US/PST045219 |
| [54] | U. S. Census Bureau. (2020, December 15). Census Bureau releases 2020 demographic analysis estimates. https://www.census.gov/newsroom/press-releases/2020/2020-demographic-analysis-estimates.html |
| [55] | U. S. Census Bureau. (2022). Datasets. https://www.census.gov/data/datasets.html |
| [56] | Wang, H. J., Kjellstrand, C. M., Cockfield, S. M., & Solez, K. (1998). On the influence of sample size on the prognostic accuracy and reproducibility of renal transplant biopsy. Nephrology Dialysis Transplantation, 13 (1), 165–172. https://doi.org/10.1093/ndt/13.1.165 |
| [57] | Zhang, Z., Xu, X., & Ni, H. (2013). Small studies may overestimate the effect sizes in critical care meta-analyses: A meta-epidemiological study. Critical Care, 17 (1), R2. https://doi.org/10.1186/cc11919 |
APA Style
Kory Floyd, Nathan T. Woo, Jeannette Maré, Kaylin L. Duncan. (2023). Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals. Communication and Linguistics Studies, 9(2), 27-41. https://doi.org/10.11648/j.cls.20230902.12
ACS Style
Kory Floyd; Nathan T. Woo; Jeannette Maré; Kaylin L. Duncan. Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals. Commun. Linguist. Stud. 2023, 9(2), 27-41. doi: 10.11648/j.cls.20230902.12
AMA Style
Kory Floyd, Nathan T. Woo, Jeannette Maré, Kaylin L. Duncan. Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals. Commun Linguist Stud. 2023;9(2):27-41. doi: 10.11648/j.cls.20230902.12
Share This Article
Twitter
Linked In
Facebook
Copy
Download@article{10.11648/j.cls.20230902.12,
author = {Kory Floyd and Nathan T. Woo and Jeannette Maré and Kaylin L. Duncan},
title = {Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals},
journal = {Communication and Linguistics Studies},
volume = {9},
number = {2},
pages = {27-41},
doi = {10.11648/j.cls.20230902.12},
url = {https://doi.org/10.11648/j.cls.20230902.12},
eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.cls.20230902.12},
abstract = {Background: The ability to draw accurate inferences from research depends heavily on the quality and representativeness of research samples. Research samples in the social sciences, including communication, are frequently criticized for being small and unrepresentative, yet there is substantial variation in sample characteristics. Objective: The objective of this project was to undertake a systematic examination of the characteristics of human samples used in communication research in major communication journals, in order to respond to the criticisms that such samples are small, underpowered, and lacking in external validity. Method: To ascertain the status of human samples in communication research, this project examined every empirical study published between 2010 and 2019 in four top communication journals—Communication Monographs, Communication Research, Human Communication Research, and Journal of Communication—that reported data from human subjects. The data set included 1,264 individual studies and a total sample size of 932,060 participants. Results and Conclusion: Sample sizes ranged from 10 to 57,847 participants, with an average of 740.12 participants, and were larger for non-experiments than experiments, quantitative than qualitative studies, and secondary than primary data analyses. Ninety-four countries were represented in the samples, although more than 70% of samples were recruited exclusively from the United States. Compared to U. S. demographics, such studies oversampled younger participants, female participants, and white participants.},
year = {2023}
}
TY - JOUR T1 - Sampling Practices in Communication Studies: A Decade of Research in Four Top Journals AU - Kory Floyd AU - Nathan T. Woo AU - Jeannette Maré AU - Kaylin L. Duncan Y1 - 2023/06/15 PY - 2023 N1 - https://doi.org/10.11648/j.cls.20230902.12 DO - 10.11648/j.cls.20230902.12 T2 - Communication and Linguistics Studies JF - Communication and Linguistics Studies JO - Communication and Linguistics Studies SP - 27 EP - 41 PB - Science Publishing Group SN - 2380-2529 UR - https://doi.org/10.11648/j.cls.20230902.12 AB - Background: The ability to draw accurate inferences from research depends heavily on the quality and representativeness of research samples. Research samples in the social sciences, including communication, are frequently criticized for being small and unrepresentative, yet there is substantial variation in sample characteristics. Objective: The objective of this project was to undertake a systematic examination of the characteristics of human samples used in communication research in major communication journals, in order to respond to the criticisms that such samples are small, underpowered, and lacking in external validity. Method: To ascertain the status of human samples in communication research, this project examined every empirical study published between 2010 and 2019 in four top communication journals—Communication Monographs, Communication Research, Human Communication Research, and Journal of Communication—that reported data from human subjects. The data set included 1,264 individual studies and a total sample size of 932,060 participants. Results and Conclusion: Sample sizes ranged from 10 to 57,847 participants, with an average of 740.12 participants, and were larger for non-experiments than experiments, quantitative than qualitative studies, and secondary than primary data analyses. Ninety-four countries were represented in the samples, although more than 70% of samples were recruited exclusively from the United States. Compared to U. S. demographics, such studies oversampled younger participants, female participants, and white participants. VL - 9 IS - 2 ER -
Email: