Background and Goals

• During the Covid-19 pandemic, Columbia University’s Social Psychology lecture course was taught remotely on Zoom for the first time.
• The course had always included small-group discussion activities, but the teaching team wasn’t sure how they would translate to a remote-learning environment.
• In the first lecture, students experienced an initial prototype of the activity:
  • Groups of 5-6 students were placed in Zoom breakout rooms.
  • Students were given 10 minutes to discuss an assigned topic that was stated verbally in the lecture and sent as a text-based “Broadcast” reminder to all breakout rooms.
• After students experienced the activity prototype, I conducted evaluative research to determine how to improve the activities in future class sessions.

Institutional Objective

Optimize Zoom discussion activities to facilitate students’ deep, effective engagement with course content

Research Questions

• Is 10 minutes the right amount of time for these discussion activities? If not, how much time would be appropriate?
• What do students like and dislike about the current format, and how can we improve it?

Hypotheses

• While some students will think that they could benefit from shorter or longer discussions, students will tend to agree that 10 minutes is appropriate.
• Students will dislike the need to speak up due to feeling nervous; they will want the teaching team to find ways to help them feel more comfortable doing so (I had a sense of this based on the number of cameras off in class and in the first discussion activity).

Methods

I administered an anonymous survey comprising both closed-ended and open-ended questions.

Reasoning

Because teaching involves evaluating students’ performance through course grades, I was especially concerned about social desirability bias in this context. Because students may not have been comfortable providing critical comments if their identities were known, and given the need to collect both quantitative and qualitative data in order to answer the research questions, I proceeded with a survey that could be completed anonymously.

Participants

74 undergraduate and post-baccalaureate students currently enrolled in the course¹

Study Materials

[Click to enlarge]

Analytic Strategy

• Conducted a chi-square goodness of fit test to examine the distribution of students who indicated that the 10-minute length was “Too short”, “Just right”, or “Too long”
• Performed a thematic analysis on qualitative (open-response) data to understand student (i.e., “user”) goals, needs, and frustrations

Results

Quantitative

The majority of students thought that the 10-minute length was just right.²

Qualitative

My thematic analysis revealed three distinct student segments.³

Segment 1: Anxious and Disconnected

Segment 2: Unclear and Frustrated

Segment 3: Engaged and In Community

Recommendations and Impact

Based on the results of this research, the teaching team implemented the following changes to discussion activities in subsequent class sessions:

• Facilitated community by explicitly asking all students to introduce themselves to their fellow group members at the beginning of subsequent discussion activities
• Encouraged more active participation from all students by instructing everyone to share an initial answer to the discussion question when introducing themselves
• Provided the discussion question and more detailed instructions in the chat before breaking students out into groups

Success Metrics: Students’ feedback on how the optimized experience compared to the prototype in a follow-up survey

• Nearly half of students reported a better experience.⁴

• Students tended to agree that the optimized activity was enjoyable.⁵

Limitations and Learnings

While this research met my research goals and the primary institutional objective, it had some important limitations:

• Not all students in the course participated in the research; only 65% completed the original survey and 38% completed the follow-up. As a result, these results could have been impacted by selection bias.⁶
• Given the importance of maintaining student anonymity, I felt that it was not appropriate to conduct interviews to probe students’ goals and pain points more deeply. This was a trade-off that the teaching team was willing to make for ethical reasons because our roles involved evaluating students’ performance in the course.
• The teaching team likely could have improved the course further through a continued, iterative process of feedback and optimization. However, given the rate of attrition between the original survey and the follow-up, I was concerned that students would have become frustrated with requests to provide even more feedback. (We did note verbally in class and office hours that we always welcomed additional comments, though!)

The process also revealed a few unexpected learnings.

• From my perspective as a member of the teaching team, it was surprising to learn that students viewed course discussion activities as such an important opportunity for socialization, rather than just an opportunity to engage with course content.
  • Without having conducted this research, I wouldn’t have realized that classroom discussions served this critical role during pandemic times. Thankfully, we were able to optimize the activity structure to ensure that students were able to meet one another before diving into the discussion.
• In retrospect, it likely would have been useful to offer students the opportunity to provide more detailed (and non-anonymous) feedback on the activities after the course had finished and grades had been released.
  • I do believe that by probing deeper into students’ motivations, needs, and frustrations through a method like user interviews, we could have uncovered even more useful insights that could have been put to use in future semesters.

Footnotes

1. Because this research was conducted anonymously, I am not able to provide more specific demographic details for this sample. (The class was small enough that with this information, individual participants may have become identifiable.) ↩︎
2. A chi-square goodness of fit test indicated that this pattern of responses differed significantly from chance (χ² (2, N = 74) = 35.22, p < .001). Post-hoc paired comparison tests indicated a significant difference between the number of “Too short” vs. “Just right” responses (p < .001), and a marginally significant difference between the number of “Just right” vs. “Too long” responses (p = .098). ↩︎
3. While I originally showcased the findings of this analytic process in a simpler bullet-point format, I have converted these qualitative results into empathy maps and user personas for reference in this case study. ↩︎
4. A chi-square goodness of fit test indicated that this pattern of responses differed significantly from chance (χ² (2, N = 43) = 9.12, p = .010). Post-hoc paired comparison tests indicated significant differences between the number of “Worse” vs. both the “Better” and “Same” responses (ps = .004), and no difference between the number of “Better” vs. “Same” responses (p = 1.00).  ↩︎
5. Students rated enjoyment on a five-point Likert scale (1 = Strongly disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Strongly agree). A one-sample t test indicated that the mean level of agreement differed significantly from the neutral scale midpoint (t(42) = 2.24, p = .030). ↩︎
6. For instance, students with more extreme opinions about the discussion activities—or simply those with the most negative opinions—could have been especially motivated to respond, which would affect the validity of these findings. With this said, responses to questions with Likert-style scales in the follow-up survey appeared normally distributed with relatively few “Strongly agree” or “Strongly disagree” responses, which eased this concern to some extent. ↩︎