Student Evaluation, Assessment, and Testing

An effective assessment provides critical information to both instructor and learner (AIH ch. 6):

• The learner learns how well they are progressing, with practical and specific feedback including direction and guidance on how to raise performance
• The instructor sees where more emphasis is needed — if several learners falter at the same step of a weight-and-balance problem, that step needs a better explanation or another demonstration
• Most importantly, a well-designed assessment provides an opportunity for self-evaluation that enhances the learner's ADM and judgment skills

Eight characteristics (AIH ch. 6):

• Objective — focused on actual performance, free of personal opinion, likes, dislikes, or bias. Over-identification with a learner to the point it influences objectivity is halo error
• Flexible — fit the tone, technique, and content to the occasion and the learner
• Acceptable — learners need confidence in your qualifications, teaching ability, sincerity, competence, and authority
• Comprehensive — covering strengths as well as weaknesses; not necessarily long, and not every aspect in detail
• Constructive — pointless unless the learner benefits; identify a mistake and give positive guidance for correction
• Organized — any logical pattern; the sequence of the performance itself works well
• Thoughtful — respectful of the learner's need for self-esteem, recognition, and approval. Ridicule, anger, or fun at the learner's expense has no place; deliver criticism in private
• Specific — "your second weld wasn't as good as your first" has little constructive value; say why and how to improve. At the end the learner should have no doubt about what was done well, what was done poorly, and specifically how to improve

Traditional assessment is written testing — multiple choice, matching, true/false, fill in the blank — with a set time and a single correct response. It relies largely on rote memorization and assesses at the rote and understanding levels, though carefully crafted scenario questions can reach higher. It lends itself to instructor-centered teaching, and a satisfactory grade on one lesson may not reflect the ability to apply knowledge in a different situation.

Authentic assessment requires the learner to perform real-world tasks and demonstrate a meaningful application of skills — to generate a solution instead of merely choosing a response — reaching the application and correlation levels. Specific performance criteria are known to the learner in advance, and scoring uses a rubric.

Traditional testing still has a place: it assesses grasp of information, concepts, terms, processes, and rules — the factual foundation needed before advancing (AIH ch. 6).

A guide used to score performance assessments in a reliable, fair, and valid manner. It is generally composed of dimensions for judging learner performance, a scale for rating performance on each dimension, and standards of excellence for specified performance levels (AIH ch. 6).

Collaborative assessment uses two broad rubrics: one for maneuvers and procedures, one for SRM — the cognitive, decision-making side of flight training.

A four-step series of open-ended questions that guide the learner through a complete self-assessment (AIH ch. 6):

1. Replay — the learner verbally replays the flight or procedure. You listen for places where the account doesn't seem accurate and, at the right moment, discuss the discrepancy. This validates the learner's perceptions and gives you critical insight into their judgment
2. Reconstruct — identify the key things the learner would have, could have, or should have done differently
3. Reflect — invest perceptions and experiences with meaning: What was the most important thing you learned today? What was easiest, what was hardest? Did anything make you uncomfortable? How would you assess your performance and decisions? How did your performance compare to the standards in the ACS?
4. Redirect — relate the lesson to other experiences: How does this relate to previous lessons? What might mitigate a similar risk in the future? What personal minimums should be established, and what additional proficiency flying or training might be useful?

The self-assessment is followed by an in-depth discussion comparing your assessment to the learner's.

• Describe — the learner can describe the physical characteristics and cognitive elements of the activity, but needs assistance to execute it
• Explain — can describe the activity and understand the underlying concepts, principles, and procedures, but still needs assistance to execute
• Practice — can plan and execute the scenario; coaching, instruction, or assistance corrects deviations and errors identified by the instructor
• Perform — can perform the activity without instructor assistance, identifying and correcting errors expeditiously, and at no time is successful completion in doubt
• Not observed — any event not accomplished or required

(AIH ch. 6)

• Explain — the learner can verbally identify, describe, and understand the risks inherent in the scenario, but needs to be prompted to identify risks and make decisions
• Practice — can identify, understand, and apply SRM principles to the actual flight situation; coaching quickly corrects minor deviations; the learner is an active decision maker
• Manage-Decide — can gather the most important data inside and outside the flight deck, identify possible courses of action, evaluate the risk in each, and make the appropriate decision. Instructor intervention is not required for safe completion of the flight
• Not observed — any event not accomplished or required

Grading should be progressive: automation management might be a "describe" item on flight one, "practice" by flight three, and "manage-decide" by flight five (AIH ch. 1, ch. 6).

To be effective, questions must (AIH ch. 6):

• Apply to the subject of instruction
• Be brief and concise, but also clear and definite
• Be adapted to the ability, experience, and stage of training of the learner
• Center on only one idea — limited to who, what, when, where, how, or why, not a combination
• Present a challenge to the learner

Fact questions (who, what, when, where) test memory and recall. HOTS questions (why, how) require combining facts with analysis, problem-solving, and conclusions. Devise and write pertinent questions in advance — one method is to put them in the lesson plan — and supplement with impromptu questions as the lesson progresses.

Start with yes/no questions like "Do you understand?" and "Do you have any questions?" — those aren't quizzing at all. Then (AIH ch. 6):

• Puzzle — "What is the first action if a conventional gear airplane with a weak right brake is swerving left in a right crosswind during a full flap, power-on wheel landing?"
• Oversize — "What do you do before beginning an engine overhaul?"
• Toss-up — "In an emergency, should you squawk 7700 or pick a landing spot?"
• Bewilderment — a question with so many nested conditions the learner loses the thread
• Trick questions — the learner feels engaged in a battle of wits and the significance of the subject is lost
• Irrelevant questions — asking about tire inflation during a test on magneto timing

Three steps (AIH ch. 6):

1. Be sure you clearly understand the question before attempting to answer
2. Display interest and frame an answer as direct and accurate as possible
3. Determine whether the learner is satisfied with the answer

If the honest answer is more complicated or advanced than the learner needs at this point in training, explain that the question was good and pertinent but that a detailed answer would unnecessarily complicate the learning task now — and invite the learner to reintroduce it later at the right point.

If you don't know, freely admit it and promise to get the answer, or offer to help the learner look it up in available references. Bluffing destroys learner confidence quickly (see also AIH ch. 8, "admit errors").

• Diagnostic — assesses learner knowledge or skills prior to a course of instruction
• Formative — not graded; provides a wrap-up of the lesson and sets the stage for the next one. Limited to what transpired during that lesson, it tells you what to reinforce
• Summative — used periodically throughout training to measure how well learning has progressed: a stage check, a chapter quiz, an end-of-course test

Separately, an assessment can be formal (usually documented — a quiz or written examination, used to measure and document whether course objectives were met) or informal (as needed, including verbal critique, and not part of the final grade) (AIH ch. 6).

• Reliability — consistent measurement on repeated application
• Validity — the extent to which the test measures what it is supposed to measure. The most important consideration in test evaluation; items that don't pertain to the objectives should be modified or eliminated
• Usability — functionality: readable type size, clear and concise wording, clean graphics, easy grading
• Objectivity — singleness of scoring. Essay questions are hard to score objectively; selection-type items are much easier
• Comprehensiveness — measures the overall objectives via a representative cross-section, not one cylinder of the engine
• Discrimination — distinguishes between learners at differing levels of achievement, via a wide range of scores, all levels of difficulty, and items that separate achievement levels

Note the caveat: discrimination suits academic achievement, but minimum standards are far more important in assessments leading to pilot certification (AIH ch. 6).

A four-step, general-to-specific process for criterion-referenced tests (AIH ch. 6):

1. Determine level-of-learning objectives — state them as general objectives in the cognitive, affective, or psychomotor domain. "Describe how to perform a compression test" is understanding level; it does not ask the learner to perform, compare, design, or interpret
2. List indicators or samples of desired behaviors — measurable behaviors that give the best evidence of learning
3. Establish criterion objectives — add the conditions under which the behavior is performed and the criteria to be met. If you wrote performance-based objectives for your lesson plans, these already exist
4. Develop criterion-referenced assessment items that measure the behaviors in the criterion objectives

Timing caution: authentic assessment may not be as useful as traditional assessment in the early phases of training, because the learner doesn't yet have enough knowledge to participate fully. Learners first memorize, then organize, and only then can they analyze, synthesize, and evaluate.

Because the objective is for all successful applicants to meet the high standards of knowledge, skill, and safety required by the regulations — not to spread applicants across a curve (AIH ch. 6). The ACS/PTS delineates the standards by which FAA inspectors, DPEs, and DMEs conduct tests, reflecting 14 CFR parts 61, 65, 91, the AIM, and pertinent ACs.

A detail worth knowing on checkride day: private pilot applicants are evaluated in all Tasks of each Area of Operation; flight instructor applicants are evaluated on one or more Tasks in each Area of Operation, with certain Tasks required by notes immediately following the Area of Operation titles — which is exactly the Area I note requiring Tasks E, F, and one other.

Since the evaluator may cover every Task, evaluate all of them before recommending an applicant for the practical test. That evaluation need not be formal, but it should adhere to criterion-referenced testing.

A critique is an instructor-to-learner assessment, used with either traditional or authentic assessment, individually or in a classroom. It should come immediately after the performance while details are easy to recall, and it considers good as well as bad performance — the individual parts, their relationships, and the overall performance (AIH ch. 6).

• Instructor/learner critique — the instructor leads a group discussion in which classmates offer criticism. Control it carefully with a clear purpose; do not let it become a free-for-all
• Learner-led critique — a learner leads. Inefficient because of inexperience, but generates interest and learning
• Small group critique — the class divides, each group analyzes an assigned area and reports; furnish criteria and guidelines
• Individual learner critique by another learner — one learner presents the whole assessment
• Self-critique — the learner critiques their own performance, still under instructor control and supervision
• Written critique — three advantages: you can devote more time and thought to it, the learner can keep and refer to it, and if the whole class writes one the performer has a permanent record of everyone's input. Disadvantage: the rest of the class doesn't benefit

Whatever the type, resolve disagreements, correct erroneous impressions, make allowance for learner inexperience, and reserve time at the end to cover what was omitted or under-emphasized.

The risk is delivering an assessment poorly — the failure modes mirror the characteristics of an effective one (FI.I.D.R1):

• Halo error — sympathy or over-identification distorts objectivity, and the learner is signed off for something they can't do
• Criticism without correction — if a learner made an earnest effort and is told the work is unsatisfactory with no explanation, frustration occurs, and unidentified errors get perpetuated through faulty practice (AIH ch. 8)
• Public criticism — a critique before the class can benefit everyone, but avoid embarrassing the learner in front of the class; deliver criticism in private
• Inventing deficiencies — deficiencies should not be invented solely for the learner's benefit; unfair criticism immediately destroys a learner's confidence in the instructor (AIH ch. 8)
• Grade inflation — praise given too freely becomes valueless

The learner must also understand the purpose of the assessment, or they are unlikely to accept the evaluation and little improvement results (AIH ch. 9).

Assessment of demonstrated ability must be based on established standards of performance, suitably modified to apply to the learner's experience and stage of development — and it must consider the learner's mastery of the elements involved in the maneuver, rather than merely the overall performance (AIH ch. 8, ch. 9).

Tools available to the flight instructor: the review, collaborative assessment (learner-centered grading), written tests, and performance-based tests.

An assessment can also be a tool for reteaching: if you observe a deficiency and decide a task needs reteaching, demonstrate the maneuver, let the learner practice under direction, then evaluate by observing performance.

Postflight critiques should be in a written format, such as notes, to help you cover everything you noticed during the flight.

Two key advantages (AIH ch. 6): it actively involves the learner in the assessment process and establishes the habit of healthy reflection and self-assessment that is critical to being a safe pilot; and the grades are not self-esteem related — they describe a level of performance, not a level of prestige. The learner cannot flunk a lesson; they demonstrate a particular level of flight and SRM skill.

Two cautions. Instructors trained under traditional grading must not force the rubric dimensions into the A-through-F mold — the antidote is remembering that evaluation should be progressive across lessons. And a learner who has never self-assessed may be reluctant, so you may need to teach the learner how to be an active participant in collaborative assessment.

The handbook's illustration: Brian rates himself "Perform" because he made no mistakes; Linda rates the same item "Practice." The ensuing discussion is where Brian learns what "Perform" actually requires.