Cyber Assess Western Australia 2009

Cyber Assess (Mechanical Fitting) - Lecturer point of view

South West Regional College of TAFEWA


This case study provides the perspective of a lecturer and coordinator who were involved in the Cyber Assess project who supervised the mechanical fitting component.
As a Vocational Education and Training (VET) Provider delivering trade training in a regional area, the South West Regional College (SWRC) is often faced with the challenge of small class sizes. With small class sizes comes the problem of low funding and the related difficulties of delivering quality training and assessment to the learners.
This was particularly relevant in the final stages of an engineering tradesperson mechanical apprenticeship and the groups need to be split into specialisation areas. During the project, a class of 15 needed to be split to accommodate a group of six fitting apprentices.
The project enabled the trial of new delivery and assessment strategies and technologies with apprentices and employers and in doing so, provided a new model for delivery and assessment in the workplace.

What was done

The first step was to clarify the goals of the project and align them with the problems faced in small class sizes. The second step of the process was to look at the technologies and strategies available and how they could be integrated into the project. The technologies the project decided to utilise were:
• Moodle - is a learning management system (LMS) that enables training providers to load their own content, link to other websites, design and manage quizzes and assessments and keep track of the learners activities and progress.
• Elluminate - Elluminate is a virtual classroom used for synchronous on line learning, similar to MSN but with much more functionality for the learning environment.
• POV (Point of view) cameras and MP4 recorder - A POV camera is a device used for recording assessment evidence from the point of view of the candidate. The camera captures video and audio and records to a small pocket MP4 Recorder.

The third step was to look at the training schedule for the final phase of the Fitter course to see what subject/s would lend themselves to delivery and assessment in the workplace.
The units chosen for delivery as part of this project were MEM18004B Maintain and overhaul mechanical equipment and MEM18011C Shut down and isolate machines/equipment.
These units were selected because of the high level of practical component involved and the fact that they are both based around tasks that the students carry out at work in a routine fashion. Once the class group and the strategies were decided upon, it was time to contact the apprentices and employers to discuss with them the implications of delivering these units in the workplace. It was very important to approve the use of POV cameras as many industrial sites have bans on using cameras on site. Fortunately, none of the employers who had apprentices in this part of the project had any issues with allowing the POV camera to be used.
After talking to the students, it was decided to first use the POV cameras to assess a simulated isolation and shutdown at the college. This served two purposes. Firstly it gave the participants the opportunity to practice in using the technologies in a safe and controlled environment and secondly, it provided additional information when assessing. It also guided the assessor in knowing what sort of evidence they could expect to get from the POV cameras.
The project had to ensure that the resources that the students required were loaded onto the Moodle site and were accessible. Only existing resources were used for this part of the project. In order to introduce the assessment medium to the students, it delivered as part of the shut down. The isolated machinery unit assessment was completed using the POV cameras at the TAFE.

CyberAssess1.jpgThe students were given a brief lesson on how to use the POV cameras and on the assessment task of isolating part of the compressed air system at the college. This proved a very positive exercise, both as a way of providing a valid and fair assessment at the college and in training the students in the use of POV cameras. Lecturers were no longer required to stand over the shoulder of students during the assessment (carried out at a different location within the college). Lecturers were able to give attention to the remainder of the class and mark the assessment at a later time when it was convenient.
Before the project could be taken to the workplace, the issue of how we controlled the use of the POV cameras and responsibility of the cameras when used by the student had to be solved. It was decided that the cameras and recorders would be controlled by the library on a type of closed reserve system. Additional forms (Appendix 1) were used for the student to complete prior to taking the camera.
The lecturer also completed a form on the return of the equipment, which stated that the video captured had been copied off the device and the memory had been deleted. It was important to ensure the use of this equipment was not putting undue pressure on the staff in the library. The lecturer had envisioned sending each set of cameras out to the work places and having all of the evidence back within four or five weeks. Even though it was a ’nice plan’, it was not realistic.
Within the first week of use out in the field we had one of the MP4 recorders malfunction - the backlight failed and it could no longer be used. This resulted in an average turnaround of six to ten days. Fortunately the supplier of the cameras was able to replace the MP4 recorder quickly and we were back in business. By now, some of the assessments from the student had been returned and the lecturer was pleasantly surprised at both the amount and quality of evidence they had videoed. The cameras allowed a first person perspective of some of the ancillary tasks that students were doing on a daily basis as well as the tasks that were being directly assessed. A lot of supplementary evidence for other units the students had done previously was also received.
Unfortunately we had another MP4 failure and we were again down to one camera. This proved to be a bit of a problem as in some cases the student may have had the glasses for two weeks waiting for the right work to turn up for them to video. However, this was still preferable to lecturers waiting around for the right task.
Lecturers were surprised at the ease of assessing when watching a video.One of the downsides of doing a live practical assessment is that the elements or criteria that you are assessing may be separated by long periods of non-assessable work. This may mean that you need to spend an hour watching for observable assessment points that may only add up to five minutes.
With the POV footage you are able to fast forward through the footage to the key areas that are of interest for the assessment. Assessors are also able to go back and review parts of the process that you weren’t sure the student covered, which is something that can not be done during a live assessment.

Patrick (student) identifying himslef for POV Assessment
Patrick demonstrating the lock he has put on equipment

©©As the POV assessments were occuring, the lecturer piloted connecting with the students on a weekly basis using Elluminate. The plan was that instead of the students coming to TAFE each week, there would be a meeting once a week in a virtual classroom to discuss the unit, what they were doing at work and how the POV assessments were going.
On the first planned session only four out of the six students were able to log in (one student was sent to a remote mine site and didn’t have access to a computer and the other was unable to download software on his PC at work due to site policies). The session proved successful and it was agreed that another attempt would be made in the following week.
Over the next week, an attempt to install Java on the PC of the student was made, however due to the unavailability of IT support on site for the apprentice and the relatively low priority it had in the scheme of a large multinational company, it was not solved.
Lecturers still met online with the rest of the group but decided to opt for a day at the TAFE every three weeks rather than continue with Elluminate so the student who couldn’t access it wouldn’t be unfairly disadvantaged. However those who could access Elluminate enjoyed using the technology and believed that it would have been worthwhile to continue if all of them could access it.
Early on in the project, it was decided that there would be a need to communicate with students quickly and regularly, especially to facilitate the timely movement of the cameras between the students’ workplaces.
In consultation with the group, the best way to do this was with the use of SMS (text messaging). Lecturers knew how much time a young person spends texting, however they were unaware of the power of a text to reach students. Within minutes of sending a group text, 90% of students replied within the day. The students also seemed to place more importance on a text message, for example if a request or instruction was sent via SMS they would treat it with some urgency, something that they don’t often seem to do even if you are giving them face-to-face instruction.

Benefits of experience

A major benefit of the project was that the lecturers were able to get the students to video their tasks and being able to review the recordings at their own convenience, and the ability to focus on the assessment. The POV cameras provided for greater focus on the assessment process by enabling the student being assessed to work independently on the task they are being assessed on.
There was no distraction from either the assessor or their classmates. This would be very hard to achieve in conventional assessment in the TAFE environment. Any observational assessment calls for one-to-one assessment, which is very difficult to properly assess and maintain a level of supervision and/or instruction over the rest of the class.
Another benefit of using POV cameras to capture the assessment task was being able to fast forward and rewind a recording to get to the core evidence. Often during a conventional assessment you may be distracted momentarily and miss a crucial piece of evidence.
Lecturers have learned a great deal during this project, and have been able to pass on their experiences to other staff at the college through delivering information sessions at professional development (PD) days at the college on the use and benefits of POV cameras. This has been a rewarding experience; the participants of these sessions have all been very excited at the prospect of being able to use POV Cameras.
The coordinator of the project has been able to raise the profile of e-learning in the college and resources are now being used to deliver regular PD to lecturing staff on the use of alternative strategies including Moodle, Elluminate, POV Cameras, and SMS.
As a spin off from the project, the coordinator has introduced podcasting as a way to deliver content to students and is currently involved in some PD in developing and loading podcasts onto a LMS.
The coordinator intends to use this technology to educate more of the staff in the benefits of POV cameras. As part of the project, the college has committed funds to purchase eight pairs of POV cameras, which means that many more lecturers will be able to access them to conduct workplace assessments.
The project has given the coordinator an opportunity to trial technologies in a way that they wouldn’t have been able to do under normal circumstances. They were able to meet with other staff on a regular basis and share our experiences and ideas. The coordinator, who was a trade lecturer, found that it is easy to get wrapped up in their own world and forget that there are many other people around the college with similar problems and some fresh and alternative ways to deal with those problems.
In participating in the project, the coordinator has had an opportunity to get to know students in a different context and place a bit more value on what they are doing at work. The scope and depth of the work done by students was surprising and a better understanding of where TAFE lecturers, students, workplace supervisors and trainers could sit in the modern training environment was achieved.

Lessons learnt

During the project, the coordinator learned that there was a lot of information that could be gathered from the workplace if given the appropriate tools and time. One of the issues experienced was with tools malfunctioning, namely the MP4 recorders. Unfortunately there were not many of these on the market and as the suppliers of POV cameras have found the earlier models had some issues.
These are being addressed and the reliability of the equipment is improving. One of the major lessons learned was that a lot of lead time and testing is required to ensure apprentices can access the technologies in the workplace. This was particularly important when dealing with larger companies with IT departments and policies that may lock students out.
When using POV cameras some of the video footage was missed due to the camera not being correctly adjusted for the wearer. The older glasses were difficult to adjust and therefore the students would not always capture the required evidence or it would be just out of shot. The newer cameras are fitted with a hinge to enable easy adjustment but it was still very important to ensure the candidates know how to check and adjust the field of view with the cameras.
If done again, more time developing the quantity and quality of the asynchronous materials on Moodle would be desirable. The more beneficial learning opportunities that can be given to students, the better the experience and performance they will have. Also, the project learned that if you want students to know or do something, send it via SMS.

The results

The project allowed the trial of some alternative strategies for delivery and assessment in the workplace. It enabled the coordinator to meet with other participants on a regular basis and discuss in a safe and supported environment different methods of using the technologies to better support workplace learning.
By managing the project, the coordinator has further developed research techniques and strategies for dealing with the huge volume of work that accompanies any large project. These included tasks like managing the budget, keeping participants engaged, liaising with outside organisations and getting reports written and submitted within the project deadlines.
The students from the fitting class have all said that they enjoyed being a part of this trial and that they believe that all of the technologies used, helped when delivering and assessing in the workplace. Even the student who was unable to participate in the Elluminate sessions has stated that they believe that given enough time they would be able to overcome the technical problems and be keen to use this platform in the future.
The coordinators skills in using the POV cameras, Moodle and Elluminate have all developed considerably during the project. They have also picked up some new ideas for delivering in the workplace using podcasting.

Reflections and suggestions

All of the participants were better off as a result of being part of Cyber Assess. The only regret the coordinator had was that because of the short lead time they was unable to ensure all of the participants were able to access the required software from work. When introducing a new delivery strategy you need to be very clear with all of the stakeholders, to ensure that they clearly understood:
• what was being proposed
• how the changes would be implemented
• how they would benefit
• what was required of them
• implications regarding infrastructure ie computer availability and policies.

The Framework connection

The national training system’s e-learning strategy, the Australian Flexible Learning Framework (Framework ) funds and supports E-learning Innovations projects, which aim to embed e-learning into the national training system by supporting and enabling innovation in training design and delivery at the state and territory level.
Workplace delivery and assessment, while being a desired outcome of all training packages, are difficult to address by training organisations.
The major reason for this is economics, delivering to small groups of workers in many different locations is a logistical nightmare with many additional costs and with limited resources, training providers are forced to go down the traditional institutional modes of delivery.
The Cyber Assess Project aimed to embed a new pedagogy of delivery and assessment into the SWRC that enabled the VET practitioner to offer meaningful workplace delivery and assessment to existing workers, apprentices and trainees using innovative e-learning practices.
It aimed to embed virtual classroom platforms, such as Elluminate, as a real alternative to traditional face-to-face delivery that has enabled students to access their classes without needing to leave their workplace or, in some cases, their home. In engaging in this e-learning Innovations project, the following products and resources were used:
• Moodle
• Elluminate
•Wimba Create


This is a Western Australian E-learning Innovations project output, developed by the South West Regional College of TAFE with seed funding from the Framework.

For more information

For more information on South West Regional College of TAFEWA
Guy Truss
Lecturer and E-learning Coordinator
Phone: (08) 97807135

For more information on the Australian Flexible Learning Framework:
Phone: (07) 3307 4700