Increasing Validity of Online Research by Implementing New Tools and Techniques
DOI:
https://doi.org/10.51480/compress.2021.4-1.248Keywords:
careless responses, fraudulent IP manipulation, Amazon MTurk, poor quality data, validity improvementAbstract
The aim of this paper is to present new techniques that increase the quality of data obtained through online research, basing on the example of Amazon MTurk. Using a critical analysis of the literature, preceded by a query, we identified the main sources of low-quality data such as careless responses, form-fill bots activity, and fraudulent behavior of manipulating IP addresses. The techniques implemented in the study offer several practical implications in terms of validity improvement. The results proved that 60% of observations were recognized as poor quality data. This confirms the applied control techniques to be useful and the proposed theoretical model of time operationalization to be an important indicator of careless responses.
References
Anduiza, E., Galais, C. (2017). Answering Without Reading: IMCs and Strong Satisficing. Online Surveys International Journal of Public Opinion Research. 29(3). 497–519. Pobrano z: https://academic.oup.com/ijpor/article-abstract/29/3/497/2669464 (10.01.2021).
Becker, R., Möser, S., Glauser, D. (2019). Cash vs. vouchers vs. gifts in web surveys of a mature panel study – main effects in a long-term incentives experiment across three panel waves. Social Science Research. 81. 221–234. Pobrano z: https://www.sciencedirect.com/science/article/pii/S0049089X18305581 (10.01.2021).
Bell, T. (2001). Extensive reading: Speed and comprehension. The Reading Matrix. 1(1). Pobrano z: https://www.semanticscholar.org/paper/Extensive-Reading%3A-Speed-and-Comprehension.-Bell/12f10caba81be9aa363fe1d92d4aac97fc025d55?p2df (10.01.2021).
Bosnjak, M., Poggio, T., Becker, K.R., Funke, F., Wachenfeld, A., Fischer, B. (2013). Online survey participation via mobile devices. W: The American Association for Public Opinion Research (AAPOR) 68th Annual Conference. Pobrano z: http://www.aapor.org/AAPOR_Main/media/AnnualMeetingProceedings/2013/Session_I-5–2-Bosnjak.pdf (9.01.2021).
Chandler, J., Shapiro, D. (2016). Conducting Clinical Research Using Crowdsourced Convenience Samples. Annual Review of Clinical Psychology. 12. 53–81. Pobrano z: https://www.annualreviews.org/doi/full/10.1146/annurev-clinpsy-021815–093623 (9.01.2021).
Everett, A.C.J., Earp, B. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology. 6. Pobrano z: https://www.frontiersin.org/articles/10.3389/fpsyg.2015.00621/full (9.01.2021).
Hargittai, E., Shaw, A. (2020). Comparing Internet Experiences and Prosociality in Amazon Mechanical Turk and Population-Based Survey Samples. Socius. 6. Pobrano z: https://journals.sagepub.com/doi/pdf/10.1177/2378023119889834 (3.01.2021).
Huang, J.L., Curran P.G., Keeney, J., Poposki, E.M., DeShon, R.P. (2012). Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business and Psychology. 27(1), 99–114. Pobrano z: https://journals.sagepub.com/doi/pdf/10.1177/2378023119889834 (7.01.2021).
Kees, J., Berry, C., Burton S., Sheehan, K. (2017). An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon’s Mechanical Turk. Journal of Advertising. 46(1). 141–155. Pobrano z: https://www.tandfonline.com/doi/abs/10.1080/00913367.2016.1269304 (3.01.2021).
Kung, F.Y.H., Kwok, N., Brown, D.J. (2018). Are Attention Check Questions a Threat to Scale Validity? Applied Psychology: An International Review. 67(2), 264–283. Pobrano z: https://iaap-journals.onlinelibrary.wiley.com/doi/full/10.1111/apps.12108 (2.01.2021).
Lovett, M., Bajaba, S., Lovett, M.M., Simmering, M.J. (2018). Data Quality from Crowd sourced Surveys: A Mixed Method Inquiry into Perceptions of Amazon’s Mechanical Turk Masters. Applied Psychology: An International Review. 67(2), 339–366. Pobrano z: https://iaap-journals.onlinelibrary.wiley.com/doi/full/10.1111/apps.12124 (14.01.2021).
Mancosu, M., Ladini, R., Vezzoni, C. (2019). ‘Short is better’. evaluating the attentiveness of online respondents through screener questions in a real survey environment. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 141(1), 30–45. Pobrano z: https://journals.sagepub.com/doi/full/10.1177/0759106318812788 (2.01.2021).
Meade, A.W., Craig, S.B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. Pobrano z: https://pdfs.semanticscholar.org/41b7/7840bf309358ecf45b16d00053ed12aea5c0.pdf (2.01.2021).
Munger, K. (2020). Knowledge Decays: Temporal Validity and Social Science in a Changing World. Pobrano z: https://files.osf.io/v1/resources/3mnzu/providers/osfstorage/5d6d45b980f9b5001763c4d2?action=download&version=1&direct (dostęp: 2.01.2021).
Moss, A., Litman, L. (2018). After the Bot Scare: Understanding What’s Been Happening With Data Collection on MTurk and How to Stop It. CloudResearch. Pobrano z: https://www.cloudresearch.com/resources/blog/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it/ (2.01.2021).
Peyton, K., Huber, G.A., Coppock, A. (2020). The Generalizability of Online Experiments Conducted During The COVID-19 Pandemic. Pobrano z: https://osf.io/s45yg/download (2.01.2021).
Shamon, H., Berning, C. (2020). Attention Check Items and Instructions in Online Surveys with Incentivized and Non-Incentivized Samples: Boon or Bane for Data Quality? Survey Research Methods. 14(1). 55–77. Pobrano z: https://ojs.ub.uni-konstanz.de/srm/article/view/7374/6874 (2.01.2021).
Tourangeau, R., Rips, L.J., Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Tourangeau, R., Sun, H., Yan, T., Maitland, A., Rivero, G., & Williams, D. (2017). Web Surveys by Smartphones and Tablets: Effects on Data Quality. Social Science Computer Review, 36(5), 542–556. Pobrano z: https://www.researchgate.net/profile/Ting_Yan3/publication/318613541_Web_Surveys_by_Smartphones_and_Tablets_Effects_on_Data_Quality/links/597b6f040f7e9b880281afae/Web-Surveys-by-Smartphones-and-Tablets-Effects-on-Data-Quality.pdf (2.01.2021).
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Polish Communication Association
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Content of the articles is licensed under a Creative Commons Attribution 4.0 International license
