Jenny Baker / Wednesday, March 23, 2022 / Categories: 594 Ask Me Anything (AMA) About Graduate School: How Online and Brick-and-Mortar Programs Are Viewed on the I-O Psychology Reddit Sayeed Islam, Talent Metrics Consulting/Farmingdale State College; Gordon Schmidt, Purdue Fort Wayne; Haroon Pervez, University of Central Florida; and Jennifer DeRosa, Vicki Carpenter, Yashica Albert, Carole Ingram Keiser University Industrial-organizational (I-O) psychology continues to be one of the fastest growing fields (U.S. Bureau of Labor Statistics, 2014). As the field continues to grow, so do the number of graduate programs offering graduate degrees in I-O psychology. There are now 28% more master’s programs in the field of I-O psychology than a decade ago (Bailey, 2020). With this unprecedented increase in programs and students, there has been an increase in the number of graduates from online programs. As noted by Landers (2018) with current trends, the future composition of SIOP membership may soon include more online graduates than graduates from traditional brick-and-mortar programs. Data from the Society for Industrial and Organizational Psychology indicate that the number of new student members from online or distance programs far outstrips the number from traditional brick-and-mortar schools for the discipline (Landers, 2018). As online learning becomes more of a norm, students continue to search for information regarding online I-O psychology graduate programs. The relative success and educational quality of online and traditional higher education programs has been hotly debated (Wagner et al., 2011; Wickersham & McElhany, 2010), with some studies indicating little differences between online and traditional courses in higher education. Other research indicates concerns around long-term career prospects of students in online graduate programs (Bendersky et al., 2008) as well as degree completion rates (Gazza & Hunker, 2014). The increase in online programs granting master’s- and doctoral-level degrees in I-O psychology has created an influx of online graduates joining SIOP. This trend has prompted internal research from SIOP for I-O-psychology programs in particular. A subcommittee of the SIOP Education and Training Committee found that although universities attempt to maintain rigor in their programs’ educational standards, concerns remain about the faculty of the online programs (Dahling et al. n.d). Specifically, the relevant background of the faculty teaching in the online programs was difficult to determine. The results of a survey conducted by the committee indicated a slight negative view of online degree holders. These results were not a critique of the quality of education provided by these online programs but a recognition that SIOP had to develop relationships with these institutions to better understand the quality of education offered. Without clear guidance on program quality, it remains up to students to decide which programs best meet their needs. Students tend to find much of their information about potential graduate programs online (Constantinides & Stagno, 2012) and are often swayed by modality in their choice of graduate school (Leach et al., 2017). Many students use social media sites like Twitter and Reddit. Reddit is one of the world’s most popular websites with 52 million users visiting the site per day (Kastrenakes, 2020). Reddit is a social news-aggregation, web-content rating, and discussion website (Schmidt, 2019). Reddit contains multiple subreddits, which are “user-created areas of interest where discussions on Reddit are organized” (Anderson, 2015). These subreddits allow interested individuals to discuss topics of interest. A growing area of Reddit is the industrial-organizational (I-O) psychology Reddit, where users come to learn more and discuss topics about the field of I-O psychology. I-O psychology Reddit currently has 19,760 members (Reddit, n.d.), which has grown substantially from when the site first began in 2011 with 76 members (Wayback Machine, n.d). In comparison, SIOP has 9,000 members (SIOP, 2022), so this subreddit may have more members than there are active SIOP members. Reddit is one of the most popular resources for online information. Many users (also known as Redditors) use the site to learn about a variety of topics including potential graduate programs. Reddit has proven useful as a data source in studies of mental health (Boettcher, 2021) and news sources (Ovadia, 2015). Reddit has been cited as a rich data source comparable to Amazon MTurk (Jamnik & Lane, 2017). Although assessments of online education have been primarily conducted in individual courses, most research does not look at the evaluation of online programs versus traditional brick-and-mortar schools. Parahoo et al. (2016) developed a student satisfaction model in higher education to assess online programs. Their model has been used to assess both online and traditional forms of higher education (Lazarevic & Bentz, 2021). Their data indicated that student satisfaction in online education was evaluated by students using the following criteria: Student interactions with faculty, other students, administration, and IT Faculty empathy, defined as faculty’s level of caring for students University reputation, defined as the overall reputation of the university Faculty feedback, defined as the quality of feedback students receive from faculty Physical facilities, including location and physical facilities offered Each of these criteria was found to be related to overall student satisfaction, making this model an ideal framework to use to assess student satisfaction through relevant variables students use when evaluating programs. Although this model of student satisfaction is commonly assessed using a Likert scale, for the present study we propose to evaluate Reddit comments using this model qualitatively. Previous research has used this model as the basis for qualitative research (O’Malley, 2020). The present study attempted to evaluate the qualitative feedback about I-O psychology graduate programs posted to Reddit to assess differences across online, in-person, and hybrid-learning-modality programs. The authors attempted to address the following research questions: Research Question 1: What factors are most often discussed in relation to I-O psychology graduate schools? Research Question 2: What differences in terminology are there between brick-and-mortar and online-only programs? Research Question 3: Are there differences in language and sentiment over time? Method Graduate student coders collected comments from Reddit threads on the I-O psychology Reddit page. The I-O psychology subreddit contains threads devoted to a variety of topics related to the field of I-O psychology. The I-O psychology Reddit page has Graduate School Q&A Mega Threads, which are threads dedicated to answering questions about graduate school admissions and programs. Users ask questions about graduate school admissions and programs in these threads. Other users then post answers or responses. The threads are organized by school year. Four research assistants were asked to collect comments related to specific brick-and-mortar (face to face), online, or hybrid programs. Before data collection began, the research assistants went to the SIOP Graduate Training Programs page and identified graduate programs as face to face, online, or hybrid. Hybrid was defined as offering both online and face-to-face components of the program (Abdelrahman & Irby, 2016). Each research assistant was assigned a thread and then collected relevant comments. Using the name of the schools, the comments were coded as belonging to schools that were (a) face to face, (b) online only, or (c) hybrid. Some institutions appeared on multiple lists of programs because they offered face-to-face, online, and hybrid programs. General comments about online or brick-and-mortar education were not included in the dataset because the Parahoo et al. (2016) model is specifically related to student satisfaction, which has been used as a measure of program quality (Braskamp et al., 1979). Comments that were not associated with a program were deemed to be outside of the scope of the present study. Using sample comments, the research assistants were trained until they reached consensus on the coding for sample comments. 294 relevant comments were collected and coded. Using the Parahoo et al. (2016) model of student satisfaction, comments were categorized as being related to any of six topics: (a) IT/Admin/student interactions, (b) faculty empathy, (c) faculty feedback, (d) university reputation, and (e) physical facilities. Because Reddit users may write comments that touch upon a variety of issues, each post could be coded as more than one category. Any comments that did not fit into any of these categories were categorized as “other.” Results Data from the entire corpus were analyzed using Voyant tools (Sinclair & Rockwell, 2016). Voyant tools is a computer-assisted text-analysis (CATA) program that provides researchers count measures of the words in a corpus of text. Figure 1 contains a word cloud of all the terms produced by Voyant tools. The qualitative data were organized by year and modality for analysis. The word “program” was central to the corpus across years and modality. Figure 1 Word Cloud of Entire Corpus Table 1a contains sentiment scores by year and modality. The corpus was analyzed for sentiment scores using Daniel Soper’s sentiment analyzer (Soper, 2006). The sentiment analyzer scores the corpus from -100 (completely negative sentiment) to +100 (completely positive sentiment) using a common dictionary of terms to categorize language as positive or negative. The overall corpus had a slightly negative sentiment score of -15. The only positive score was for online programs discussed in 2018–2019. All the other scores were negative with the sentiment scores for online for 2017–2018 being the lowest at -85.7. Hybrid programs from 2017–2018 had no programs listed, and there was no analysis conducted. Overall, these sentiment scores indicate that the majority of comments were negative in nature. Average sentiment scores for face-to-face programs was -23.73, for online programs it was -45.53, and for hybrid programs sentiment scores were -40.65. Sentiment scores varied considerably over time without consistency. Table 1a Sentiment Scores by Year and Modality Modality F2F Online Hybrid Entire corpus across all years Year 2016–2017 -43.1 -15.1 2017–2018 -9.6 -85.7 -45.2 2018–2019 -23 27.5 2019–2020 -19.2 -78.4 -33.1 Average -23.73 -45.53 -40.65 Table 1b contains most frequent words by year and modality. The term “program” appears most often. All program names were removed. Table 1b Most Frequent Words by Year and Modality Modality Year F2F Online Hybrid Entire corpus across all years 2016–2017 program (11); good (6); programs (6); phd (5); - - program (185); programs (80); good (78); phd (56); students (53) 2017–2018 program (57); good (28); programs (28); io (21); students (20) course (3); master’s (2); nonsense (2); offer (2); poor (2) program (7); programs (7); good (6); i’ve (5); master’s (5) - 2018–2019 program (26); students (11); phd (9); programs (8); good (7) program (6); field (5); ’ve (4); applied (3); fairly (3) - - 2019–2020 program (59); phd (23); good (20); programs (19); like (15) professional (4); chicago (2); equal (2); faculty (2); good (2) program (12); dc (6); phd (6); faculty (5) - *word counts appear next to each term in parentheses Table 1c contains number of posts, percentage of posts, and unique terms by year and modality. Unique terms are words that only appear in that specific year and modality’s corpus not the entire corpus. This list included some program names and locations that have been redacted and replaced with the term “program name” by the researchers. Some negative terminology like “nonsense” and “negative” appear in the online comments. Among the face-to-face program language there are terms such as “TA” and “OHP” and “IO” reflecting some discussion of the subfields of I-O. Table 1c Unique Terms by Year and Modality Year Modality Number of posts % of year’s posts Unique terms (in comparison to corpus) 2016–2017 F2F 13 100% program name (4), position (2), org (3), focused (3), PhD (2) 2017–2018 F2F 98 86% exec (9), io (21), program name (6), aren’t (6), apparently (6). 2018–2019 F2F 44 78.5% program name (5),TA (4), reviews (3), ra (3), apps (3). 2019–2020 F2F 92 82.8% health (9), ohp (5), occupational (7), io (14), probably (6) 2017–2018 Online 4 3.5% nonsense (2), course (3), negative (1), wouldn’t (1), vanguard (1) 2018–2019 Online 12 21.5% talked (2), structured (2), solution (2), quit (2), gig (2) 2019–2020 Online 3 2.7% equal (2), faculty name (1), joke (1), program name (1), program name (1). 2017–2018 Hybrid 12 10.5% program name (4), program name (3), program name (4), leadership (2), program name (2) 2018–2019 Hybrid 2019–2020 Hybrid 16 14.4% program name (7), creation (2), computational (2), online (5), dc (6) *word counts appear next to each term in parentheses Table 2 contains percentages of the coded categories from the Parahoo et al. (2016) model. The most common category discussed in the posts by coders were university reputation across multiple years with it accounting for 100% of the comments from 2016–2017. Table 2 Percentage of Comments Relating to Student Satisfaction Topics by Year: Face-to-Face Programs 2016–2017 2017–2018 2018–2019 2019–2020 Faculty empathy 0 0 0 1% Faculty feedback 0 11% 0 17.5% Student, IT/admin staff interactions 0 1% 6.8% 3.1% University reputation 100% 88% 63.7% 78.3% Physical facilities 0 0 0 0 Other 0 0 29.5% 0 Table 3 contains percentages of the coders categories for online programs. Across all years, university reputation was the most prominent category for online programs with at least 80% of posts discussing university reputation across all years. Table 3 Percentage of Comments Relating to Student Satisfaction Topics by Year: Online Programs 2017–2018 2018–2019 2019–2020 Faculty empathy 0 0 0 Faculty feedback 20% 0 20% Student, IT/admin staff interactions 0 0 0 University reputation 80% 90% 80% Physical facilities 0 0 0 Other 0 10% 0 Table 4 contains percentages of the coded categories from the student satisfaction model regarding hybrid programs; university reputation was the most common category across all years with 92.3% of posts discussing reputation in 2017–2018 and 81.3% in 2019–2020. This indicates that the most common topic identified among these posts is the reputation of the university. Table 4 Percentage of Comments Relating to Student Satisfaction Topics by Year: Hybrid Programs 2017–2018 2019–2020 Faculty empathy 0 0 Faculty feedback 7.7% 18.7% Student, IT/admin staff interactions 0 0 University reputation 92.3% 81.3% Physical facilities 0 0 Other 0 12.5% Table 5 contains representative comments by modality that represent some of the common content found in these threads. Program names and city names have been removed. These comments seem to highlight the focus of many of the threads on university reputation as a driving factor. Table 5 Representative Quotes by Modality Modality Representative Quote F2F “Most of the programs around [program 1] have smaller faculties and are very much focused on professional students. None aside from [program 2] offer a PhD in straight I-O psych. Consequently, they don't usually show up on the "best schools" lists because most of the ranking methodologies emphasize research productivity at PhD programs rather than professional training at terminal master's programs. I would not recommend [program 3], but [program 1] and [program 2] are both good programs. (I'm not sure that [program 1] is worth the cost relative to your other options, but the quality is good.) Online “I'm doing the online master’s program at [program 4] and am about three quarters of the way through. I've had a fairly positive experience with it. I work full-time and found it a good solution to obtaining my masters without having to quit my FT gig” Hybrid “I am having difficulty choosing where to attend. No programs seem to be top-notch in training and seem to have good opportunities for internships. They both say that there are opportunities for paid internships to help pay for the total cost. According to the professors who reached out to me from both schools, they don't give any funding to masters' program students or any tuition waivers.” Discussion Based on these initial findings, it appears that the focus of much of the discussion about master’s and PhD programs in I-O on the I-O psychology subreddit revolved around university reputation. Many of the comments defaulted to university reputation as the key issue of quality. Overall, the sentiment of the corpus appeared to be negative. This indicates that the language used when discussing specific graduate school programs through the I-O psychology subreddit is largely negative, either by discussing barriers to entry or the reputation of programs. PhD and MA programs should be mindful of reputation because much of the online discussion on Reddit discussed university reputation. It would be useful to study university reputation as a heuristic used by students in their decision-making process. Future research should assess brand recognition of universities as a way students may evaluate brick-and-mortar and online programs. The greater amount of negative sentiment attributed to online programs should encourage online-only programs to assess their university branding and work to improve this brand recognition. If online-only programs hope to compete for top students for their MA and PhD programs, it appears reputation is key. Both online and traditional programs could consider focusing on communicating student successes as well as their research productivity to attract higher quality students. Student placements in internships and jobs may well also improve university reputation overall. This study has several limitations. First, the I-O psychology Reddit page is not a representative sample of all I-O academics or practitioners. A survey study assessing the views of SIOP members or I-O practitioners may prove valuable to understand the driving forces of university reputation. Second, the use of open source qualitative data allowed the present research to draw general conclusions about many programs without generating the predictive capability between quantitative variables. Qualitative studies should be conducted about the process by which students are assessing online and face-to-face programs. This type of study may help online, hybrid, and brick-and-mortar programs better understand the factors driving such decisions. Another limitation was that there was little distinction made between online and hybrid programs in the SIOP graduate training programs database. Thus, there was significant overlap in language used between online and hybrid programs because some programs were included in both categories. In the future, a clearer distinction should be made between online and hybrid programs. Future research should analyze quantitative student satisfaction data longitudinally. Tracking changes of opinions of online and face-to-face programs may prove beneficial to MA and PhD program directors in the field of I-O psychology across all modalities. Collecting data on this topic more consistently and across programs will help programs address student concerns. Data of this type may also provide more objective metrics for students struggling to identify a program that they wish to attend. A deeper analysis of the differences between online, hybrid, and face-to-face programs would benefit SIOP and graduate school programs. A major future concern of those working in I-O psychology is maintaining the vitality of the field (Woo et al., 2022) through higher education quality. The COVID-19 pandemic has shown that online education will remain a viable option for the foreseeable future (Ali, 2020). By helping online and traditional programs serve students, SIOP and graduate programs will maintain a robust pipeline of talented I-O professionals. The goal should be that these professionals receive a high-quality education regardless of modality. References Abdelrahman, N., & Irby, B. J. (2016). Hybrid learning: Perspectives of higher education faculty. International Journal of Information Communication Technologies and Human Development (IJICTHD), 8(1), 1–25. Ali, W. (2020). Online and remote learning in higher education institutes: A necessity in light of COVID-19 pandemic. Higher Education Studies, 10(3), 16–25. Anderson, K. E. (2015). Ask me anything: What is Reddit? Library Hi Tech News, 32(5), 8–11. https://doi.org/10.1108/LHTN-03-2015-0018 Bailey, D. (2020). Enticing new faces to the field. Monitor on Psychology, 51(1), 60. https://www.apa.org/monitor/2020/01/cover-trends-new-faces Bendersky, K., Isaac, W. L., Stover, J. H., & Zook, J. M. (2008). Psychology students and online graduate programs: A need to reexamine undergraduate advisement. Teaching of Psychology, 35(1), 38–41. Boettcher, N. (2021). Studies of depression and anxiety using Reddit as a data source: Scoping review. JMIR Mental Health, 8(11), e29487. Braskamp, L. A., Wise, S. L., & Hengstler, D. D. (1979). Student satisfaction as a measure of departmental quality. Journal of Educational Psychology, 71(4), 494. Constantinides, E., & Stagno, M. C. Z. (2012). Higher education marketing: A study on the impact of social media on study selection and university choice. International Journal of Technology and Educational Marketing (IJTEM), 2(1), 41–58. Dahling, J., Fitzpatrick, T., Johnson, S., Lassiter, A., Samuelson, C. D., Wentworth, D. K. (n.d.). Summary of online industrial/organizational psychology programs. Society for Industrial and Organizational Psychology. Retrieved from https://silo.tips/download/summary-of-online-industrial-organizational-psychology-programs Gazza, E. A., & Hunker, D. F. (2014). Facilitating student retention in online graduate nursing education programs: A review of the literature. Nurse Education Today, 34(7), 1125–1129. Jamnik, M. R., & Lane, D. J. (2017). The use of Reddit as an inexpensive source for high-quality data. Practical Assessment, Research, and Evaluation, 22(1), 5. Kastrenakes, J. (2020, December 1). Reddit reveals daily active user count for the first time: 52 million. The Verge. https://www.theverge.com/2020/12/1/21754984/reddit-dau-daily-users-revealed Landers, R. (2018). Dominance of SIOP membership by online degree holders coming. NeoAcademic. https://neoacademic.com/2017/07/19/dominance-of-siop-membership-by-online-degree-holders-coming/ Lazarevic, B., & Bentz, D. (2021). Student perception of stress in online and face-to-face learning: The exploration of stress determinants. American Journal of Distance Education, 35(1), 2–15. Leach, L. F., Winn, P., Erwin, S., & Benedict, L. P. (2017). What 21st century students want: Factors that influence student selection of educational leadership graduate programs. In Information Resources Management Association (Ed.), Educational leadership and administration: Concepts, methodologies, tools, and applications (pp. 1877–1891). IGI Global. O’Malley, V. E. (2020). Promises to keep: How institutional brand aligns with the adult learner experience[Doctoral dissertation]. Northeastern University. Ovadia, S. (2015). More than just cat pictures: Reddit as a curated news source. Behavioral & Social Sciences Librarian, 34(1), 37–40. Parahoo, S. K., Santally, M. I., Rajabalee, Y., & Harvey, H. L. (2016). Designing a predictive model of student satisfaction in online learning. Journal of Marketing for Higher Education, 26(1), 1–19. R/iopsychology. Reddit. (n.d.). Retrieved from https://www.reddit.com/r/IOPsychology/ Schmidt, G. B. (2019). Reddit. In Debra Merskin (Ed.), The SAGE international encyclopedia of mass media & society, (pp. 1468–1470). Sage. DOI: http://dx.doi.org/10.4135/9781483375519.n563 Sinclair, S., & Rockwell, G. (2016). Voyant tools. http://voyant-tools.org/ SIOP. (2022, January 28). SIOP member benefits. https://www.siop.org/Membership/Benefits Soper, D. (2006). Sentiment analyzer. https://www.danielsoper.com/sentimentanalysis/default.aspx U.S. Bureau of Labor Statistics (2014). Occupational outlook handbook. Retrieved from http://www.bls.gov/ooh/fastest-growing.htm Wagner, S. C., Garippo, S. J., & Lovaas, P. (2011). A longitudinal comparison of online versus traditional instruction. MERLOT Journal of Online Learning and Teaching, 7(1), 68–73. Wayback Machine. (n.d.) https://www.reddit.com/r/IOPsychology/. Retrieved from https://web.archive.org/web/20110401000000*/https://www.reddit.com/r/IOPsychology/ Wickersham, L. E., & McElhany, J. A. (2010). Bridging the divide: Reconciling administrator and faculty concerns regarding online education. Quarterly Review of Distance Education, 11(1), 1. Woo, V., Lall-Trail, S., & Islam, S. (2022). Delivering on DEI: An analysis of coursework & research in graduate programs. The Industrial-Organizational Psychologist, 59(3) https://www.siop.org/Research-Publications/Items-of-Interest/ArtMID/19366/ArticleID/5540/preview/true/Delivering-on-DEI-An-Analysis-of-Coursework-and-Research-in-Graduate-Programs Print 3102 Rate this article: 4.7 Comments are only visible to subscribers.