Cal Poly Academic Program Review (APR) for Graduate Programs
Revised Oct 27, 2004

Guidelines for Self-Studies

"First and foremost, the purpose of academic program review (APR) at Cal Poly is to acknowledge the strengths and seek ways of further enhancing the quality of academic programs."

In general terms, this is accomplished by reviewing the program's mission and goals as they relate to the mission of the institution; the curriculum through which the program's mission and goals are pursued; the extent to which the program is achieving its objectives for student learning; the quality and diversity of the faculty and staff and their contributions to achieving the program's mission and goals; and the quality of the infrastructure supporting the program (e.g., library and other educational resources; physical facilities, etc.).

To support the review, all programs are expected to include in their self-studies:

1. General information about the program, e.g., data on students, faculty, staff, facilities, etc. (most of which is supplied on request by Institutional Planning & Analysis).
2. A statement of intended student learning outcomes at the program level; methods for assessing them, including the use of direct measures; assessment results to date; and documentation of the use of assessment results in efforts to achieve program improvement.

   Note that student learning is as important in graduate programs as it is in undergraduate programs. Moreover, graduate programs have the responsibility to ensure that learning occurs at an advanced level and is not simply an extension of knowledge beyond the undergraduate curriculum. The Cal Poly catalog makes this point as follows: "In graduate courses, students cope with more complex ideas, problems, techniques and materials than in undergraduate courses. Graduate study requires searching and exhaustive analysis, identification and investigation of theories and principles; application of theory to new ideas, problems and materials; extensive use of bibliographic and other resource materials, with emphasis on primary sources for data; and demonstration of competence in scholarly presentation of the results of independent study." For more information on assessing student learning outcomes at the graduate level, see attachment.

3. The results of a focused inquiry addressing issues of particular interest/concern to the program itself, in the context of what is currently important to the college and university.

• For 2005/06 reviews of graduate programs, items of special importance to the university include:
  
  The existence of an up-to-date strategic plan at the program, department, or college level, in which the program's mission is defined and the context for program development & improvement is supplied.
• Evidence of an appropriate culminating experience
  • As graduate programs have evolved to meet both marketplace and student demands, many have become more interdisciplinary and encourage team approaches and cooperative learning. This has led to innovative ways to provide culminating experiences for students. When employing these new methods, programs must still demonstrate how they comply with Title V and Cal Poly catalog requirements for the culminating experience for each student. Specific guidance on culminating experiences is provided on Cal Poly's website.
• Discussion of program's fit within a polytechnic university.
  • Cal Poly has an important role to play in master's education in technical and professional fields. In addition, graduate programs with technical emphases in disciplines not traditionally considered technical - for example, music and art - provide opportunities for advanced study that are distinctive and highly relevant to Cal Poly's mission. Finally, when appropriate, our graduate programs should incorporate the same quantitative and practical approaches that characterize our undergraduate programs and are expected at a polytechnic university.
• Evidence of faculty engagement in research and professional development, as appropriate.

Programs are strongly encouraged to involve their faculty in all stages of APR, including design of the self-study. The Program Head/Chair, the Dean, and the Vice Provost all sign off on the self-study proposal and the time-line for its preparation. The Office of Academic Programs is available to provide consultation and assistance, as feasible, at all stages in the process.

For more information, contact Valene Mathews in the Office of Academic Programs.

Back to top

ATTACHMENT

Assessing the Quality of Graduate Programs

The primary focus of the assessment should be on  student learning outcomes.  Programs are encouraged to employ direct measures whenever possible, although indirect measures (such as alumni satisfaction) may also be used. All Master's programs incorporate a thesis, project or comprehensive exam that results in a written document that can be evaluated. Such capstone experiences can provide an ideal basis for assessing learning outcomes and reporting results in the self-study.  For example, a faculty panel could select a representative sample of student theses (or equivalent), and assess these  using a scoring rubric that links directly to the educational objectives of the program.  Then, the self-study might report the results as evidence of student learning in the program.  Rubrics could include any, or all, applicable learning outcomes from the list below (programs may add others).  Scores could be based on scale of 0-4, where 4 is excellent and 0 indicates failure to meet program expectations, as outlined in the sample rating scale for rubric scoring  (see below). 

Sample Masters Thesis/Project Learning Outcomes:

• Demonstrates original thinking or project design
• Takes creative approach to issues or problems
• Addresses a sufficiently complex problem or question
• Articulates the problem or question clearly
• Places problem or question in proper historical/scholarly context
• Formulates an original hypothesis.
• Designs and conducts relevant experiments to test hypothesis
• Uses appropriate research methods and/or technologies
• Shows evidence of advanced technical achievement
• Analyzes findings in adequate depth
• Reports findings clearly
• Draws appropriate, reasoned conclusions from the findings

Similarly, samples of comprehensive exams could be assessed for evidence of learning outcomes appropriate to that capstone experience.

Sample Rating Scale for Rubric Scoring

While the primary focus of program assessment should be the measurement of student learning outcomes, attention may also be paid to certain characteristics of graduate programs exhibited by many that are successful. In its 2000 report entitled Graduate Education at Cal Poly, Cal Poly's Task Force on Graduate Education provided a set of features that define quality graduate programs:

Governance

• The department or program has a graduate coordinator.
• The graduate coordinator has adjustment of teaching load in order to effectively administer the program.
• The department or program has a committee to deal exclusively with graduate issues.
• Graduate students are involved in program evaluation processes.
• The program coordinator works cooperatively through a campus-wide graduate committee.

Curriculum

A core curriculum exists that emphasizes integration of knowledge and preparation for specialization and the intellectual rigor of this curriculum is clearly beyond that of the baccalaureate level.

At least four regular graduate courses are available per year in addition to supervised and independent study.

No more than 20% of unit requirements for graduate level work involves independent study (excluding thesis).  In disciplines which are research-intensive, 30% is allowable.

Graduate level (i.e., 500-level) courses should make up 70% of a program.

Availability of graduate level coursework is increased by "pooling" or cross-registration between related departments.

Faculty

• Teaching faculty has Ph.D. or other appropriate terminal degree and relevant professional experience.
• Faculty has a strong commitment to professional development in the scholarship of discovery, the scholarship of integration and/or the scholarship of application.
• Faculty interacts with students in a professional and civil manner in accordance with University policies.
• Faculty members create supervisory relations with students that stimulate and encourage in the classroom, lab, or studio.
• Advisors develop a clear understanding with their students about their specific research responsibilities, including the time lines for completion of research and the thesis or project.
• Faculty members create an ethos of collegiality so that learning takes place within a community of scholars.
• Faculty encourages graduate students to participate in professional meetings or perform or display their work in public settings.
• Faculty acknowledges student contributions to research presented at conferences, in professional publications, or in applications for copyrights and patents.

Resources

• Space, facilities, equipment, and support staff are committed to carry out meaningful research and/or creative activities.
• The department or program is able to provide opportunities for research, scholarship, and creative activities.
• Each graduate student has a major professor and faculty committee.
• All students have the opportunity to be involved as teaching assistants where appropriate to the discipline.
• Some form of financial support (TA's, RA's, scholarship, fellowship, aid) is available for all students.
• Offices or study areas are provided within the department to provide interaction with faculty.

Access and Assessment

• Discriminatory barriers to present and potential students are absent.
• Graduate student population has high diversity (e.g., ethnic, geographic, etc.).
• Certificate programs are used to respond to the need for occupationally related graduate coursework.
• Program is reviewed and evaluated regularly to ensure that the graduate program is given specific attention separate from the other offerings of the department.
• Students make timely progress toward degree completion.