Evaluation of e-Learning Experience
Summary
This entry documents my approach to evaluating user and learning experience through both formative and summative evaluation methods. The formative evaluation employed continuous discovery and learner interviews to understand the learning process in real-time, while the summative evaluation analysed end-of-course feedback forms to assess overall effectiveness. This dual-method approach provided comprehensive insights into the learning experience and identified areas for improvement.
Role: learning experience manager
Task: LX evaluation
Methods: interviews, dashboard analysis
Tools: miro board (brainstorming ideas with UX designer), Confluence (project documentation), Teams (conducting interviews)
Employer: IU International University of Applied Sciences
Understanding Evaluation Methods
There are three types of instructional design or LX evaluation - formative, summative, and confirmative. In my work as a learning experience manager, I conducted formative and summative evaluations. Based on the data obtained through those analysis, I created initiatives for improving the learning experience of online students.
Formative Evaluation
Formative evaluation is an ongoing process conducted during the design phase that allows for immediate adjustments and improvements. This method provides real-time insights into learner experiences and challenges, enabling instructors to make timely interventions. The iterative nature of formative evaluation makes it particularly valuable for identifying and addressing issues before they impact learning outcomes.
Summative Evaluation
Summative evaluation occurs at the course's conclusion and measures overall effectiveness and achievement of learning objectives. This method provides a comprehensive view of the course's success and areas for future improvement. It focuses on gathering concrete data about learning outcomes and student satisfaction.
Process
1. Setting up continuous discovery (formative and summative evaluation)
Continuous discovery is an ongoing research and feedback process that helps teams understand user/learner needs, behaviours, and pain points throughout a product or service lifecycle. Rather than conducting research only at specific points, it involves regular, systematic interactions with users.
We used this method throughout formative and summative evaluation.
I established a structured framework for collecting ongoing learner feedback. More specifically, I:
- Created interview protocols
- Developed documentation templates for consistent data collection
2. Conducting interviews (formative and summative evaluation)
As a team, we conducted over 50 interviews with online learners from different markets and collected feedback on the learning experience. My focus were online students from the UK (The London Institute of Banking & Finance).
3. Creating opportunity solution tree (formative and summative evaluation)
We created an Opportunity Solution Tree (OST) during both formative and summative evaluation. This way, we mapped the relationship between user needs, opportunities for improvement, and potential solutions in a structured way.
4. Implementation and analysis of the evaluation forms (summative evaluation)
Our Course Evaluation team designed forms collecting feedback form learners after finishing the course and our LX team implemented the form in the learning journey. As an LX manager, my focus was primarily on the feedback learners provided for the learning experience.
The Analytics team created PowerBI dashboards which served as a data source for further evaluation.
Based on the data in the dashboards, I:
- Extracted specific suggestions for improvement of the learning experience
- Documented success stories and areas of concern
- Added opportunities in our OST
LXD Reflection
As a Learning Experience Manager overseeing this evaluation project, I've gained valuable insights into both the process and impact of our systematic approach to course evaluation. The implementation of both formative and summative evaluation methods, supported by continuous discovery, has proven to be transformative for our understanding of the learning experience.
The formative evaluation process, particularly our structured learner interviews via Microsoft Teams, provided immediate actionable insights that we could implement in real-time. Our decision to conduct regular interviews rather than waiting for end-of-course feedback allowed us to identify and address issues as they emerged, significantly improving the learning experience during the design and implementation of our learning solution.
The summative evaluation phase focused on analysing feedback forms from course participants, which provided comprehensive insights into the overall learning experience, platform usability, and content effectiveness. Even though I often required help from our analytics team, the dashboards have proven to be a valuable source of feedback on the learning experience.