OC Assessment 2009-11
(Excerpts from WASC EER Report 2012 PDF)
The ULO Project on Oral Communication began in September 2009. The ULO Oral Communication Committee adopted an operational definition from AAC&U’s Oral Communication VALUE Rubric: “a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners’ attitudes, values, beliefs, or behaviors.” Based on this definition, the committee designed a five-point rubric with seven traits: verbal delivery, nonverbal delivery, presence of a central message, organization, language use, use of supporting material, and use of visual aids.
Method. In the first year, the committee sought to establish a benchmark of students’ performance toward the beginning Cal Poly, San Luis Obispo, Educational Effectiveness Review Report 6 of their academic careers. The assessment entailed videotaping oral presentations delivered by a sample of 102 freshmen enrolled in COMS 101 and 102 during Spring 2010. The sample was 51% female and 49% male and represented all six colleges: Engineering (24%), Agriculture (23%), Science and Math (20%), Liberal Arts (15%), Business (13%) and Architecture (7%). Frequencies for both gender and college distributions did not differ significantly from what would be expected. Three faculty members from Communication Studies observed and evaluated the speeches. Training sessions ensured norming of scores and provided evaluators the opportunity to discuss, modify, and clarify the rubric as needed. Following these sessions, each evaluator scored a selection of speeches on each rubric trait on a scale of 1 to 5 with 1 being insufficient and 5 being excellent.
Results. Figure 1.5 below shows the overall scores, with the rubric traits presented in order from highest to lowest means. In addition, the figure shows the percentages of students scoring at each level of the rubric. Because so few had scores of 1, percentages for scores of 1 and 2 (insufficient and below average) were added together.
Figure 1.5 An Overview of the Descriptive Statistics
| N | BELOW AVERAGE | AVERAGE | GOOD | EXCELLENT | MEAN | STD. DEVIATION | |
|---|---|---|---|---|---|---|---|
| Supporting Materials | 102 | 13.7% | 35.3% | 45.1% | 5.9% | 3.42 | .83 |
| Language | 102 | 7.8% | 56.9% | 29.4% | 5.9% | 3.33 | .71 |
| Central Message | 102 | 11.8% | 47.1% | 37.3% | 3.9% | 3.31 | .78 |
| Organization | 102 | 10.8% | 57.8% | 27.5% | 3.9% | 3.24 | .70 |
| Nonverbal Delivery | 102 | 23.5% | 49.0% | 25.5% | 2.0% | 3.06 | .75 |
| Verbal Delivery | 102 | 22.5% | 50.0% | 25.5% | 2.0% | 3.03 | .83 |
| Visuals | 75 | 16.7% | 26.5% | 22.5% | 7.8% | 3.27 | .99 |
Oral Communication Separate chi-square analyses confirmed that the observed frequencies for both the gender distribution and the college distribution did not differ significantly from the expected frequencies. Differences in the mean scores for traits were analyzed in two separate repeated measures ANOVAs— one looking at all seven traits for the 75 students who had scores on all of the traits, and the second for the 102 students who had complete data on the six traits excluding the Use of Visual Aids. Both analyses were significant, F(6, 444) = 5.70, MSE = .430, p < .01 for the seven-trait comparison; F(5, 505) = 6.62, MSE = .383, p < .01 for the six-trait comparison. Follow-up pairwise comparisons using a Bonferroni adjustment showed the same basic pattern in both sets of analyses. Students’ trait scores were significantly higher for Language Use and Use of Supporting Materials than for Verbal and Non-Verbal Delivery scores. In addition, the presence of a Central Message was significantly higher than the Verbal Delivery score in both analyses; in the seven-trait analysis the presence of a Central Message was also significantly higher than the Non-Verbal Delivery score (all p’s ≤ .05). There were no other significant differences.
Because Use of Visual Aids was not a component of all speeches, two different statistical analyses were run on the differences in mean trait scores. One considered all 7 traits for the 75 students who had scores on all 7, while the second considered all 102 students but excluded Use of Visual Aids. A follow-up comparison showed the same basic pattern in both analyses: students’ trait scores were significantly higher for Language Use and Use of Supporting Materials than for Verbal and Non-Verbal Delivery and for Presence of a Central Message than for Verbal Delivery. In the seven-trait analysis, scores were significantly higher for Presence of a Central Message than for Non-Verbal Delivery. There were no other significant differences.
These data suggest that the vast majority of Cal Poly freshmen meet an average (3) or better level of competence in oral communication, even with only introductory instruction. This is good news, but the data also suggest that students’ verbal and nonverbal delivery could be developed further; only a quarter of the sample achieved a score of good (4) or excellent (5). Improvement in these areas would likely occur over time as students received further instruction and additional speaking opportunities. However, given that Cal Poly requires most students to take only one course focusing on oral communication, instructors of that course should consider spending additional time on improvement of verbal and nonverbal delivery.
During the second year of the project, the committee presented these results to the University Assessment Council and the Communication Studies faculty. In addition, the committee delivered a ULO-based oral communication workshop through the CTL in which twelve participants applied the rubric after watching both a below average speech and a good speech. The first speech received an average score of 2.2 and the second received an average score of 4.4. This consistency indicates that the participants used the rubric to make reliable distinctions of quality between the two speeches.
The committee originally planned a third year of activity to assess senior-level presentations perhaps in connection with senior projects, but budget cuts curtailed this aspect of the project. Recommended Action Items 3. Identify areas of the curriculum outside the GE oral communication requirement in which the Communications Studies faculty can partner with other faculties to develop students’ oral communication skills (p. 6). 4. Complete the ULO Project on Oral Communication by collecting data on upper-division student performance and making a value-added comparison to lower-division results.
Recommended Action Items in WASC EER Report
- Identify areas of the curriculum outside the GE oral communication requirement in which the Communications Studies faculty can partner with other faculties to develop students’ oral communication skills
- Complete the ULO Project on Oral Communication by collecting data on upper-division student performance and making a value-added comparison to lower-division results