Showing posts with label Assessment. Show all posts
Showing posts with label Assessment. Show all posts

Friday, 4 December 2015

The future of work and learning


Photo by LandOreports
The final day of Online Educa Berlin continued with challenging, but even more diverse offerings. One of the best sessions was on “Future Work and Future Workers.” Several futurists posed their visions of how the world of work is changing. There seems to be little doubt that, without a major natural or other disruption to progress, artificial intelligence (AI) and robotics will reduce the human contributions to many tasks. Some jobs will be changed and some will fade away. One presenter thought technology-enhanced jobs will be the most common change, where AI and other tools will just modify and improve what we already do. Another felt that many jobs will disappear, and that it is too soon to see how people will adapt to the change. Can all the replaced people become knowledge workers? Unlikely, based on current definitions of that term, but in a world of connected knowledge, perhaps we all have a place.

All seemed to feel that work has already changed. Work is now more interconnected, multidisciplinary, multi-generational, freelance, and project-oriented. The days of having a single job are fading—we increasingly fill many roles and may eventually all become entrepreneurs. The workplace of the future (already here for many) will be spread over multiple locations, if not ubiquitous. We can all contribute from anywhere and at anytime using tools already available.

The world of learning is also changing rapidly. Virtual reality workshops could become a radical disrupter and challenge those most die hard in insisting on the need for classroom training. People are learning new skills each day via YouTube videos. Do you want your doctor trained this way? Maybe a better question is, if this is where they can learn new techniques quickly, do you want them to NOT use them?

What skills are most required in this new world of work? In addition to technical skills, some posed were
  • Analytical skills
  • Intuition (creative) skills
  • Information brokering
  • Multi-cultural perspectives
  • Multi-disciplinary flexibility
Concerns were raised that the rapid pace of change is counter productive, and some warned that if we do not make it a habit to slow down, we might lose too much of our humanity.

At a session dedicated to participants from UN agencies, 23 participants discussed challenges to training design, development, and delivery. Concerns were raised about the need for faster, agile instructional design processes. In a discussion about uses of blended learning, it was clear that the term has many meanings. Many saw that at least 3 phases are needed—an online course phase, followed by a classroom phase, completed by a mentoring phase. (This was heartening to see!) Others talked about using a variety of tools and modes in the blend, not just single conceptions of “online.” We shared the tools we use, and agreed to keep a community together as much as possible to share experiences and strategies. ECBCheck, a quality management system for online learning, will be taught in a MOOC starting 15 February,2016.

In a session on learning analytics, the challenges of sorting through piles of data to turn them into useful information for diagnosis and prediction to improve learning was highlighted. In university settings, they are first tackling the task of gathering analytical data for predicting positive learning experiences, engagement and completion. Decision makers want single indices to use as predictors to help learners at risk, but with so many contributing factors, this is difficult. Learning quality, which will be much more complex to analyze, will have to come later, in the eyes of the presenters.

by Bryan Mathers
In a visually inspiring session, Bryan Mathers provided a graphical tour through ideas and concepts about learning. His blog might be of interest to you.

A final session on “2500 years of Learning Theory in 25 minutes” by Donald Clark, left few theorists unscathed. It was a challenging and thought-provoking session, but one full of hyperbole. However, the emphasis on practical theory (an oxymoron?) was refreshing.

The vendors also offered some interesting food for thought. Open Assessment Technologies offers an open source testing system (TAO) that provides 17 interaction types, including a customizable assessment type that could include an embedded and graded simulation. Online assessment can be much more than multiple choice, even though we haven't exhausted that mode yet. TAO tests can be served online from any server.

The Open Badge Factory offers an inexpensive service to allow the creation of badges that are easily transferable to many platforms, and stored in a cloud for future application. With the growing world of competency frameworks, and a WMO Global Campus, a robust badging system might serve an important role. Other vendors focused on content creation or course management systems, for example. Online assessment tools were well represented.

Patrick Parrish (WMO) and Maja Kuna (EUMETSAT)

Thursday, 29 October 2015

What Determines Quality in Training?

A workshop on identifying quality criteria was offered at the CALMet Conference, September, 2015, in Seoul, Korea. The participants were presented with this scenario: You are reviewing a training announcement from an unknown provider in a topic area you are interested in, but you would like to know about the potential quality from a reading of the announcement before investigating further.
Participants were asked to work in small groups to develop criteria within three areas: Organizational, Processes, Product. They were also asked to assign priorities to the criteria as time allowed.

The resulting draft criteria lists compiled from their decisions was then compared to a previously prepared draft list of criteria, which was developed from a review of several existing quality management systems for educational resources. These included ISO/IEC 19796, ISO 29990, Open ECBCheck (for eLearning courses), EFQUEL (for eLearning programmes), Commonwealth of Learning (UNESCO), and the WMO Competency Requirements for Education and Training Providers. Approximately 6 additional systems were examined.

New suggestions for criteria from the workshop were added or merged with the existing ones.

What came out on the top? See the list below, as well as the more complete report. Many more important criteria are contained there, as well as a discussion of the results.

Organizational

•How trainers are hired, trained, and evaluated
•Existing accreditation that covers vocational training, or existing quality assurance processes
•Numbers, roles, and qualifications of training staff members (including access to content expertise)
•Location/country, accessibility (including online presence)
•Facilities, resources, and technology infrastructure

Processes

•how learning outcomes are specified and how they inform the training content and activities
•Good development and delivery processes, documentation of processes, ISO certification (if used)
•How learning needs are determined
•how training is evaluated and improved

Products (including both courses and resources)

•currency of the content (up to date)
•learning activities match learning outcomes, opportunities for students to apply what they learn
•the relevance of course content to job competencies
•assessment methods, matched to learning outcomes
•performance-based learning outcomes
•Timing of courses (Duration and scheduling)
•the results of past evaluations of this course or resource (or related ones), student recommendations


Conclusion

Thanks to those who participated in developing these draft criteria!

As work continues to decide how quality information might be shared, in a WMO Global Campus Shared Calendar and future learning resource Registry, for example, the discussions to date should provide some useful input. But they also raise several issues for further debate.

•How does accreditation relate to quality assurance of vocation training and individual courses and resources? Is additional QA still useful?
•Does ISO 29990 certification cover the required criteria? Is more quality information required for specific resources and courses?
•How can we apply quality standards that are inclusive of many types of training centers and units?
•Is QA information voluntary, or mandatory? How will it be communicated?

Wednesday, 11 June 2014

Assessment tool validity (more from #EUMETCAL )

Hi again from lovely Poland,

The discussion this morning concentrates on assessment of forecasters in the content of the CAeM AMP competencies.

Many of us are thinking about what resources and training we need to fulfil this assessment task competently.

The presentations so far so what it happening in a couple of places (Ireland and Russia) in their assessment of AMP. It is really impressive to see the leaps that are being made to prove that the AMP are genuinely competent.

Some of the discussion revolves around what are the right tools for the assessment and how do we make sure we have the right/skilled people doing the assessment.

There is some information on the validity of different ways of assessing from the world of recruitment. There is some research available on how good different methods such as interview, observation etc are at predicting if people will be able to perform in a job.  This resource may be useful: OPP validity of assessment. It comes from the world of recruitment but makes an interesting comparison for our work.

( This reminds me of the Dunning–Kruger effect documented in this paper (unskilled and unaware) which talks about the way that people in the lower levels of skill may grossly over-estimate their skill. The paper starts with a funny story about a thief and some lemon juice.  )

Other questions raised have been:
  • What can be / is the role of peer assessment and what are the issues that need to be managed to make this useful
  • What is the "frame of the assessment" do we describe people who are incompetent in terms of AMP competencies or people who need to develop their competency. 
In implementing the AMP competencies successful organisations seem to have taken in to account the feelings of everyone impacted and had a change process that respects those feelings.

Assessor training is really important, some organisations are in countries that have vocational assessor training practice / training to draw on.

I am curious how the implementation of competencies will play out outside of aviation. This is a question of change management. Outside of aviation, the decision to mandate the competencies will be national rather than come from an external entity. It may well be fine as the process to adopt AMP competencies has broken the taboo about competencies for all of us.

The discussion continues ....