Unit 4: Course Development Standards
In order to complete the development tasks identified for your course, you’ll need to be familiar with our development standards. These standards are based on our experience providing open courses, feedback received from peer and external reviews, and principles of instructional design and professional test development processes.
Completing this unit should take you approximately one hour and 30 minutes.
4.1: Content Standards
As you identify and vet instructional resources for use in your course, act as a filter — though lots of resources may address the same topic, you want to pick the one that will best enable learners to meet the course’s stated learning outcomes. If you are working from a list of suggested open resources, feel free to raise questions or suggest alternatives if the Saylor-identified resources are not appropriate for whatever reason. We rely on the experience and subject matter expertise of our consulting faculty, so please let us know if something won’t work, and we can collaboratively find a viable solution. A few criteria to consider as you’re evaluating open resources include:
- Is the content immediately relevant to the course learning outcomes?
- Is it pitched to the appropriate level of expertise?
- Does the content include some level of interaction?
- Is the content easy to access, easy to read/see/hear, and appears polished?
- Is the content culturally sensitive and free of inappropriate bias?
- Is the content accessible to all users, inclduing those with visual/hearing impairments?
Once you’ve identified the best content, you’ll need to incorporate it into your course. As you’re pairing content, you’ll be asked to provide some or all of the following information about your course resources: hyperlink to content, instructions, time estimate, license, and aligned learning outcomes. Each of these is explained in more detail below.
- Is the content immediately relevant to the course learning outcomes?
4.2: Assessment Standards
This article describes best practices in multiple-choice item design, and reflects Saylor Academy's approach towards these kinds of auto-graded questions. Please read this article, as any assessment items created for Saylor Academy must adhere to these standards.
4.2.1: End-of-Unit Assessments (EOUAs) & Final Exams
End-of-Unit Assessments (EOUAs) and Final Exams must be auto-gradable within our current Moodle environment. For final exams, Saylor Academy exclusively uses multiple-choice items. With EOUAs, there are a few additional typical item types. Some items types will be more appropriate to assess particular learning objectives than others. You are welcome to use a variety of types in EOUAs, but you must ensure that the item type is an appropriate match to the aligned objective. You will likely be asked to develop multiple items aligned to each learning outcome for use in EOUAs or Final Exams.
Typical EOUA Item Types
- Multiple choice items should have 4 responses (one correct answer).
- Multiple response items should have 5 responses (2 correct answers).
- Open ended (fill-in the blank) questions for numerical input, dates, short vocabulary terms. Stems of this type should indicate the appropriate answer format (e.g. Provide your answer rounded to the nearest tenth.). This item-type should only be used when there is a very limited set of acceptable answers. If there are many possible answers that could be correct, a multiple choice/response is more appropriate.
- Questions with exhibits (scenarios, images, graphs) are used to expand assessment options, but exhibits must be relevant and openly licensed.
Minimum Assessment Item Standards
- Each item should measure a single, clearly defined objective
- Items should be the best measure of whether a learning objective has been achieved. In other words, items should be relatively easy/straightforward for students who have learned the material. They should be difficult for students who have not learned the material.
- Items should be written at a language that is at or below the average reading level of the students targeted by the assessment (about 40% of Saylor students identify a language other than English as their first)
- Items should be stated positively (i.e., avoid “which is NOT”, “EXCEPT”, and similar)
- Items should be clearly written, unambiguous, and free of cultural bias
- Items should not rely on "tricks" or "gotchas"
- Responses should be of equivalent length and arranged in a logical order (by length, alphabetically, etc.)
- Items may not use true/false; yes/no; all/none of the above, A and C, etc. as answer choices
- Incorrect choices should all be plausible and of equivalent plausibility
- Multiple response items should indicate the number of answers in the stem
- Exhibits should relate directly to the test question and not provide distracting or extraneous information
- Items should not disclose answers to other items in the same test
- Multiple choice items should have 4 responses (one correct answer).
4.2.2: Other Active Learning Components
Outside of the Final Exam and EOUAs, other active learning components are great opportunities to engage students in low-stakes practice, or guided problem solving, to break up otherwise dense readings and to apply concepts, among other things. Additionally, we can use this other category of assessment to encourage reflection, discussion, or self-assessment. Because we are not tied to auto-graded assessments here, there is greater flexibility to align these activities to a variety of different types of learning outcomes that may not be best assessed by the auto-graded options. This category also affords you the opportunity to have students create learning artifacts in modes or media other than written assignments (with the caveat that we serve a variety of students whose technology, time committment, or location may not make the most sophisticated media/modes possible).
As you evaluate active learning components to include in your course, consider whether there are clear right or wrong answers or attributes to a best answer, and plan for these by including rubrics or guides to responding (see example guide here). If an activity might elicit a variety of acceptable answers or approaches, consider posing it as a discussion prompt and encouraging students to use our discussion forum to discuss various perspectives or approaches with their fellow Saylorites. An activity may elicit a variety of acceptable answers but subject-matter expert voice may contribute something as well; in that situation, consider including, a “this is how the subject matter expert may have answered” exemplar.
You will need to scaffold the assignment in some way to make students understand what is expected of them and where to demonstrate their learning. See discussion topic 4 here and notice how Discourse user traveler1972 began the thread by quoting verbatim the course activity and then following up with his own response to it. User Helendsc searched for and found the thread and added her response.
Below you’ll find a few examples of ways we’ve approached various active learning components. This list is not conclusive, and we’re always happy to support the development of alternatives, so long as there is feedback where appropriate and the activity supports student learning and engagement.
Problem sets with extra practice problems from Saint Anselm College are included throughout this unit of Introduction to Psychology.
This activity asks students to identify an artwork from a list of links to online galleries and describe it according to concepts in the unit content
In a different activity within ARTH101, all students are asked to look at the same artwork as if they were a professional art critic. In this case, a subject-matter expert's viewpoint on the artwork will help students’ understanding.
Students are prompted to create and post a video of themselves making a sales presentation according to a provided framework.