I typically prefer a rapid-prototyping approach to the design process. However, I have used a number of methodologies over the years, depending on the requirements and needs of the situation. With that said, conducting a needs assessment is vital to the success of the program. So, whether conducted at the outset, or throughout the design process, this is an essential component of any program.

A needs assessment should be multi-tiered, and focus not only on the wants, or desires of the administration, but on the needs of the learners themselves, ie., educators. To gain buy-in at all levels, and most importantly with adult learners, the program must contain opportunities for learners to take learning into their own hands, to own their learning. For example, this might include the ability for learners to choose their own pathway programs, enabling a common completion of the program or certification by all learners, while providing various avenues for learners to show what they know by completing what interests them or pertains to their specific needs and goals for learning.

The results of the needs assessment will vary from partner to partner. The methods and tools used will be similar and contain core elements, with the ability to refine tools based on the business need. However, there are likely common themes from partner to partner which will enable operational efficiencies and still provide a focus on the unique needs of learners, at scale. The first step, should involve identifying the specific business need of the partner, followed by a detailed gap analysis, assessment of options, and then finally a program recommendation.

The gap analysis should include core surveys, questionnaires, self-assessments, and observations. Self-assessments are critical as they will provide data points for comparison, pre- and post-program completion. This first tier should be automated for efficiency and should fuel the next steps in the analysis. Next steps, should include interviews, and focus groups with a variety of audiences based on the feedback elicited from the surveys, questionnaires, and observations.

Based on the list of needs generated by the gap analysis, an assessment is conducted to determine the best course of action based on the business need identified previously, the total cost of Professional Learning, and the anticipated Return on Investment. This assessment provides a list of options that are then positioned with the partner for consideration.

Key stakeholders should be involved during the entire process. This includes not only the teachers (the learners), administrators, parents, and students (if appropriate) but also the core n2y sales, product, and customer success teams that are managing the account. For example, external focus groups should make up a cross-functional team of stakeholders including administrators, teachers, and parents. It may be beneficial for these stakeholders to be separated into individual focus groups depending on the dynamics of each partner.

Research shows that peer-to-peer learning is a powerful tool to enhance the learning experience and increase retention of new learning. Fostering collaboration and support among educators participating in professional learning programs will vary from partner to partner. So, one size does not fit all, but an arsenal of tools at your disposal is the key to a successful implementation. Often in a school setting, especially in such a tight knit group, like special education, team teaching is a common practice, and a means for providing a more inclusive approach while still providing individualized instruction. This may include not only teachers, but coordinators, and education technicians.

There are a number of strategies to consider, which would encourage peer-to-peer learning, reflection, and sharing of best practices. With learning platforms becoming more common in every school, as well as the delivery mechanism for learner (student) content, its use and implementation is a must and should be at the core of all interaction. This is where micro-credentialing and badging are important components to leverage as well, as it will drive increased usage and retention. Not that all interactions should be online, but that the collection of communication, content, and data should be centrally located and shared, for all learners to mine.

Peer learning in practice, among others, should be multi-tiered as well and include Train-the-Trainer programs, Learning Communities, collaborative projects, and cohorts. Each of these can be powerful in their own way and have the ability to be centrally tied to one learning platform. Single-sign-on (SSO) using a federated sign-in from either a common platform like Gmail or Microsoft, or a partner's school email (which often is Gmail or Microsoft) is ideal. This enables ease of use, and ensures CES (Customer Effort Scores) are kept low. More on this in "Evaluation and Feedback".

Each program has its own merit and should be leveraged based on the needs of the partner and the business outcomes. Train-the-Trainer models have existed for a number of years. In some cases, this model is dated, requiring a new lens to be applied. Often its the requirements put on becoming a trainer, rather than the actual goal or outcome that is being achieved that is the issue. At the core, when a trainer has "mastered" the content and then returns to the district or school, the trainer becomes the expert with the objective that other learners seek out the trainer for coaching. Maine has an established 1:1 program that has existed for more than 20 years now. Its train the trainer program did not label experts as trainers but rather coordinators. These coordinators attended a number of professional learning opportunities that focused on collaborative, pick your own path, projects, and cohorts. These MLTI (Maine Learning Technology Initiative) coordinators became the core experts that all learners came to, whether student, teacher, technician, or administrator. So, not all Train-the-Trainer models need to look the same, nor have the same objectives.

Learning Communities should drive discussion and collaboration and should offer opportunities for celebration through various rewards. It might include opportunities for earning points, becoming the recognized expert, badging, or cohort gatherings. The sky is the limit but should cater to the audience and the needs of the partner.

Collaborative projects can have disadvantages, including unequal contributions, the quality of interactions, and level of comfort some learners have in this type of setting. However, typically the advantages far outweigh the disadvantages including improved retention, increased motivation and engagement, diverse perspectives, improved communication, and the development of lifelong partnerships. Collaborative projects should offer choose your own path options for learning and for sharing what you know or have learned.

Last, cohorts can enable peer learning across schools, districts, cross-functionally with various faculty, teachers, technicians, and administrators, or even cross-functional support with very diverse groups, including learners in other parts of the country or globally. The more diverse the cohort, the more likely perspectives will vary and provide greater insight and learning from each other. Some learners may find solace and comfort in a less diverse group. However, perspectives may not be as diverse. The use of a learning platform for blended delivery of content and communication may help these learners and provide a broader perspective.

Measuring the effectiveness of professional learning programs, not too dissimilar to conducting a needs assessment, should involve a multi-level approach. There are a number of models for measuring the success of training effectiveness used throughout the Learning and Development industry including the Phillips ROI Model, Kaufman’s Five Levels of Evaluation, Anderson’s Model of Learning Evaluation, and Summative vs. Formative Evaluation, to name a few. The model itself is not the key to the success of measuring training effectiveness but rather employing a multi-level approach. For the purposes of illustrating a multi-tiered approach, I will leverage the Kirkpatrick's Four Level Training Evaluation Model. Reaction, Learning, Behavior, and Results make up the four levels.

Although there are four levels to this model, implementing at all levels often is viewed as not practical or cost effective. Most companies reserve Level 3 and 4, Behavior, and Results respectively, for large company wide initiatives, while choosing to implement Key Performance Indicators (KPIs) associated with levels 1 and 2, Reaction and Learning, alternatively. However, Levels 3 and 4, should be employed in some capacity, whatever appropriately meets the needs of the partner to assure the most complete view of training effectiveness. Collection of Behavior and Results data may be achieved by employing new more sophisticated and automated avenues. More on this later. A balance of quantitative and qualitative metrics at Levels 1 and 2 are essential to mine Voice of the Customer (VoC) insight, whether from internal learners or external customers.

At the reaction level (1), data points should include participation rates, completion rates, satisfaction rates (CSAT), and Net Promoter Scores (NPS), for example. Typically, NPS is reserved for a longer-term view of training, while more often CSAT is used at the micro level to assess training interactions. A qualitative question at level 1 might be, "What topic/section did you find the most valuable?" This provides invaluable insight into the learners' reactions but also provides an opportunity for learners to reflect on what they have learned. At Level 2, Learning, data points should include pre- and post-assessment scores, formative quizzes, and pass rates, for example. Pre-assessments offer an opportunity for learners to test out of content they already know, while also providing stakeholders a glimpse at what learners have learned post-training by comparing pre-assessment scores to post-training assessment scores. A qualitative question at this level might be, "What did you learn from your training to help you perform at a higher level in your role?"

Metrics to show training effectiveness should also include a much wider array of data points, including for example, likes/dislikes, engagement rates; time spent learning, learner drop off rates, and badges earned through gamifying learning. Typically, these data points are housed within the Learning Management System (LMS) and ultimately provide additional context to stakeholders. Although not tied directly to training effectiveness, a Customer Effort Score (CES) should also be employed to target the ease with which trainees are able to learn. For example, points are awarded for the ease of signing on to the LMS, navigating assigned programs, and courses, accessing additional supporting resources, completing the overall training pathway, or asking questions and receiving answers from facilitators. The more difficult professional learning is to complete, the lower the success rates, and effectiveness of the training.

Levels 3 and 4, of the Kirkpatrick's model, although more time consuming to document and collect for further analyzing, are a major component of understanding overall training effectiveness. Level 3, Behavior, requires observations and reporting by administrators or department/chair leaders. Observers will report the degree to which a new skill has been applied as a result of the training. This is not overly difficult but does require an avenue for observers to quickly report their findings and they must be present to make the observation. Reporting can be done through a simple follow up survey conducted at a specific interval after the completion of professional learning. Level 4, Results, are the most difficult to record or analyze, especially at the district, school, classroom, or unique learning needs level. A true measure would include an evaluation of standardized student assessment scores. This is definitely a delicate subject but is an avenue for understanding the effectiveness of training. As an alternative, during the reporting of observations at level 3, the degree to which a student's productivity or learning has increased due to the implementation of the new skill may be recorded using the same survey tool.

Today's modern learner, learns on the fly, anywhere, anytime. Research shows learning rates as high as 56% at the point of need (mobile), 48% on evenings and weekends, 41% at their desks, 30% during breaks and at lunch, and 28% on the way to or from work. This poses a tracking challenge but if captured provides a more fluid image of the overall success of a program initiative. With the advent of improved standard automated data collection, alternative methods for collecting data for Levels 3 and 4 are now possible, or automations might be used in conjunction with all levels to provide a more complete view of overall training effectiveness. The addition of a Learner Record Store (LRS) to collect various experiences using Experience Application Program Interface (xAPI) data points can provide a much broader view of all learning. This additional metric collects data on a wide range of experiences that a learner has online. Activities are recorded using simple secure statements in the form of "Noun, verb, object" or "I did this." These additional data points can also help measure the overall effectiveness of training by looking at increases in self-directed learning experiences. However, the addition of an LRS requires a capital investment as well as ongoing supporting capital but can be invaluable to providing a complete picture of training effectiveness.

The feedback provided during the various stages of evaluation are invaluable to the ongoing success of the program. Feedback will be integrated, as appropriate, when the lifecycle of the program is being updated. Critical errors or omissions should be addressed immediately, not at the next update. Updates in some cases may be completed on the fly as they don't adversely affect the operational efficiency. However, the customer's insight is critical and should be implemented as soon as it makes sense. This increases the overall effectiveness of the professional learning and shows the customer that their feedback matters. 

There are several approaches to measuring the success of training. The key to success is to decide what your goals are for your partners, in terms of learning and retention, and what measures your partners believe will provide the most comprehensive picture, with the most reasonable lift to achieve the desired outcomes.