East and Southeast Asia are experiencing a rapid pace of economic development that call for continuous review of education and training systems. TVET reform processes at the national and regional levels are addressing the question of quality and standards. During the past decades a vast variety of different approaches in TVET were developed or introduced. These approaches were often inspired through transfer or adoption from other regions of the world. In consequence, a vast variety of TVET-systems were implemented in different countries in the region and may require continues review and adaptation. Some states have no coherent national TVET-system with a common underlying concept of quality. A vast variety of TVET approaches consequently employ different concepts of quality and quality management in TVET.

Standards in TVET differ from the concept of quality. In order to enhance the mobility of labour force, especially with respect to ASEAN Economic Community 2015 and ASEAN Free Trade Area (AFTA), standards will gain a wider importance at the regional level. Mutual recognition of learning outcome or certificates will only be achievable with standards that have similar underlying concept or structure.

The 4th issue aims at enhancing the discussion with respect to a coherent understanding of TVET quality and its underlying concepts. The editorial team of the 4th issue received and selected relevant articles comprising research-, practice-, and policy-based papers with respect to TVET quality and standards as covered in the following questions:

• What can be learned from the vast variety of TVET-approaches and their underlying concept and understanding of quality and/or standards with respect to the quality of labour force?
• What concepts of quality and/or standards in TVET exist?
• How can quality be measured and the results be used for further development and innovation?
• What systems of quality management are being employed and how are they used to enhance continuously the quality and to steer the TVET system?

The articles cover topics as described below:

GERT LOOSE and GEORG SPÖTTL establish a symbiosis of assessment-oriented and experience-oriented training in the course of implementing “intelligent standards” as reliable benchmarks. Fourteen main principles comprise orientation marks, as well as the importance of high flexibility to even out regional disparities. The application of practical experience provides the context of attaining the competences required by the labor market.

With the goal of accomplishing an effective governance of the TVET sector and thus improving the evolvement of highly-skilled human resources to stabilize Malaysia’s econo¬my, MOHAMAD SATTAR RASUL, ZOOL HILMI MOHAMED ASHARI, NORZAINI AZMAN and ROSE AMNAH ABDUL RAUF give a comprehensive insight into Malaysia’s TVET-sector transformation policy and governance issues and its resulting challenges and difficulties.

CAROLINA LASAMBOUW describes the current research management process in Bandung State Polytechnic (POLBAN), emphasizing the need to lower barriers and increase the efficiency of research in vocational higher education in Indonesia based on nine problematic aspects. These aspects are of mayor importance in order to support education and community service duty and call for the decentralization policy as a part of higher education autonomy.

MUSTAFA and STEFANIE PETRICK introduce QMSiS as a practical example from a voca-tional high school in Makassar, Indonesia that achieves a better TVET-quality. Considering the eight Indonesian education standards, QMSiS additionally combines characteristics of ISO 9001:2008 with the concept Q2E1, allowing vocational schools to develop their own quality management standards (QMS). The internal and external school processes as well as the industry linkage have been improved under QMSiS.

MOHD BEKRI RAHIM explains how e-portfolios, which are used in Malaysian Skill Certifi-ca¬tion (MSC), provide an efficient method for students’ competence evaluation, give descrip-tions of students’ development process, work as a storage of artefacts, facilitate assess¬ments and enable online learning. In a modified Delphi-study of three rounds, twenty two indicators for virtual learning and five indicators for competency assessment were identified as impor-tant to MSC.

THOMAS SCHRÖDER provides a theoretical basis for inter-institutional and cross-regional research activities on vocational education in East and Southeast Asia. The resulting concept of “Scientific Coaching” offers a holistic and work process-related approach and enables scientific capacity building in the field of vocational education on the grounds of targeted and experience-based competence development.

Thank you to the contributors. Enjoy reading!

The Editors of Issue 4

Paryono Paryono, Georg Spoettl, Phil Loveder, and Chin Wei Keh

Citation

Paryono, P., Spoettl, G., Loveder, P., & Chin, W.K.(2015). Editorial Issue 4: Mitigating TVET quality and standards as a basis for harmonising its systems at the national and regional levels. In: TVET@Asia, issue 4, 1-2. Online: http://www.tvet-online.asia/issue4/editorial_tvet4.pdf (retrieved 30.1.2015).

The technical and vocational education and training (TVET) system in Brunei Darussalam is currently undergoing major restructuring, with the establishment of the Institute of Brunei Technical Education (IBTE) and launch of the ‘White Paper’ on transformation of TVET in May 2014. One of the aims of the reform is to ensure graduates are job-ready and employable, by shifting from being a supply-driven to a demand-driven system. Two key performance indicators (KPIs) have been identified to assess the efficiency and effectiveness of the reform and the TVET system. The first KPI focused on the ‘employability rate’ of graduates, while the second KPI looked at employers’ level of satisfaction of graduates hired. This paper reports on an employment study (ES) and employers’ satisfaction survey (ESS) carried out by the Research and Statistic Division, IBTE. With response rates of 98% for ES and 30% for ESS, we found (1) for ES, employability rate of our graduates are below the targeted KPI, at 63.7%; and (2) for ESS, employers’ satisfaction level met the targeted KPI, at 96.5%. By drawing upon these outcomes, we shared the experience in developing and executing the surveys, and identified some of the challenges faced and made several recommendations.

Key words: Technical and vocational education and training (TVET), Brunei, IBTE, transformation, governance, TVET, reform, Key Indikators

1 Introduction

By 2035, Brunei Darussalam is aiming to be internationally recognized for the accomplishments of its well-educated and highly skilled people, its quality of life, and its dynamic, sustainable economy. These are the essences of Brunei’s National Vision 2035 as authorized by His Majesty the Sultan of Brunei in January 2008. One approach to realise Brunei Vision 2035 includes the transformation of technical and vocational education and training (TVET) in Brunei, with the establishment of the Institute of Brunei Technical Education (IBTE) and the launch of the ‘White Paper’, the ‘Upgrading Plan for Technical Education Brunei Darussalam 2013 – 2018’ and the “Institute of Brunei Technical Education Order 2014” in May 2014.

There are six key changes currently being implemented over the next few years for the modernization of TVET in Brunei, viz. (i) course restructuring, (ii) expansion of apprenticeship options, (iii) reviewing progression opportunities, (iv) upgrading the training environment, (v) introducing a new scheme of teaching service, and (vi) renaming of Department of Technical Education (DTE) and the seven vocational and technical institutes (VTIs) under it. Parallel to this transformation, there is also a move from output based performance measures to outcome-based measures. Outcome based measures are preferable and require a more sophisticated method of data collection. The new reform places employability of graduates and employer’s satisfaction as the two main outcome measures.

The new outcome measures also fit into two of the four Brunei’s Ministry of Education (MOE) 2012-2017 Strategic Plan checkpoints identified to achieve quality education for the nation (Ministry of Education [MOE] 2012). The four checkpoints as identified are reduction of unemployment rate by 80%, being in top 50 in PISA, improving Brunei’s rank in the Global Competitive Index (GCI), and obtaining employer satisfaction.

The move towards outcome based measures is aligned with TVET providers around the region as it is essential for vocational and technical education educators to constantly evaluate their programs to ensure that their content remains relevant, of high quality and is in tune with the demand of the job market (Peng & Zhang 1997).

1.1 IBTE’s Key Performance Indicators

The previous system put a lot of emphasis on output measures that included the number of learner enrolments and the number who complete their courses. The new outcome measures in the transformation are written into key performance indicators (KPIs). KPIs are a set of measures that enable an organization to focus on aspects of performance that are most critical for the current and future success of the organization (Parmenter 2012). KPI is not a new concept and has been utilized in variety of organizations, including educational institutions worldwide e.g. Canada (Colleges Ontario n.d.), Australia (NCVER 2014), & Singapore (MOE 2014).

The two KPIs of IBTE are as follows:

Employability rate – the proportion of graduates employed within six months after their convocation ceremony; and

Employers’ satisfaction rate – the proportion of employers (i.e. supervisors) being satisfied with the work and performance of our hired graduates.

Accordingly, while both KPIs can only be reported annually, they seem to satisfy other characteristics of effective KPIs (Parmenter 2012). For instance, the two KPIs are non-financial measures, which ties responsibilities to a team or a cluster of teams (i.e. team based), and yet are relatively simple to be understood by everyone – from the CEO to the front-liners, in this case, the instructors (i.e. strategic).    

Both KPIs are based on data generated by graduates of IBTE. Generally, educational institutions rely on their alumni for a continuous stream of information to examine and improve the outcomes of their programmes (Loadman, Freeman, Brookhart, Rahman, & McCague 1999; Sageev & Romanowski 2001; Richardson, Slater, & Wilson 2007). Under Brunei’s TVET transformation, therefore, IBTE had commissioned a project to develop the two KPI surveys framework and investigate the feasibility of such surveys to be conducted annually thereafter. This paper describes the conceptual foundation of the two surveys, present selected findings from the two pilot studies and discusses their implications.

2 Graduates Employment Survey (ES)

2.1 Conceptual Framework

The Graduates Employment Survey (ES) is an annual survey conducted by the Research and Development Division (R & D), IBTE to survey the employment outcomes of graduates six months after their convocation ceremony. This survey claims much of its lineage from the “tracer studies” by then R & D, DTE which was carried out annually from 2009 till 2013 with the exception of 2012.

Improvement from past practices

The previous tracer studies had surveyed graduates several months after completing their course, to find out students’ opinions on the programmes learnt, the quality of teaching-learning, the relevance of their industrial attachment and their employment status after graduation (DTE 2010).

Previously, the time period whereby the tracer study was conducted varied between two (2) months to eight (8) months after graduation (Amalina & Chin 2012). The timing of the survey for these tracer studies, therefore, was thought not to be consistent and optimal, especially for comparison with future ES. For instance, following this TVET transformation, some programmes will have more than one intake of students per academic year, thus resulting in different completion phases. This would have complicated the time period to be allocated for data collection, as different groups would have different time length being out in the job market. Therefore, we resolved to finalize and standardize the conceptual framework of ES, whereby the status of graduates will only be assessed six months after their convocation ceremony. Other than the basis that one may only effectively look for a job after obtaining their official qualification certificate, status of graduates six months after convocation or graduation seems to be the yardstick when compared to the employment surveys carried out in other educational institutions worldwide e.g. Canada (Cambria College 2014) and Hong Kong (HKUST 2014).

More often than not, the length of time to complete these tracer studies, as well as the type of report produced, also varied and depended mostly on the R & D personnel who were put in charge. The ES aims to minimize this variability by documenting its standard operating procedure to enable comparability with future ES, despite changes in the R & D personnel. While the ES owes much of its framework to the previous tracer studies reports published by then R & D, DTE, this new working framework is simpler (e.g. shorter number of questions), yet comprehensive (e.g. all graduates to be contacted). The survey should be able to be completed within two (2) months – one month of data collection and one month of analysis and report write-up. This shorter timeframe enables up-to-date information to be shared in a timely manner with the relevant stakeholders.

2.2 Development of current framework

The development of current ES framework started in late 2013, in line with other initial preparations made for Brunei’s TVET transformation. The sampling approach as used in the tracer studies was re-assessed and its survey instrument revamped and further tested. To better understand the methodology developed in this project, readers will need additional information with regards to the previous tracer study.

The previous tracer study can be divided into two different parts; (i) graduates’ experience, obtained using a questionnaire distributed to graduates a few months after completing their course, and (ii) graduates’ employment status, as compiled by Group Coordinators (GC) for each courses offered. GCs are akin to class/group tutor who are in charge of the students’ well-being and monitoring of academic progress. Most, if not all, GCs are also teaching staff, commonly being instructors to one or more units taken by the students. It can be assumed therefor that these GCs would know their group of students at personal level.

Accordingly, GCs would then be required to contact the graduates for the tracer study. However, it was noted that not all GCs might be fully committed to compile data on graduates’ employment status. This can be seen on the relatively low response rate for tracer studies, which ranged from 40.5% to 78.7%. Each VTI would also have a Research Coordinator (RC) whose task was to compile survey data at VTI level before submitting it to R & D for analysis. Previously, it would have taken even up to six (6) months for a tracer study report to be produced from start of the data collection phase, perhaps due to the lack of urgency and the low importance being placed on these studies in decision-making and planning of DTE and the seven (7) VTIs.

Enhancement of study instrument

Initially, there were only three items surveyed on graduates’ employment status for the tracer study; (i) graduate’s employment status (i.e. whether employed, unemployed or doing further study), (ii) employed graduates’ destination sector (i.e. whether working in government or private sector, or being self-employed), and (iii) relevance of job to courses taken for employed graduates. The questionnaire on graduates’ experience, on the other hand, is a bit more comprehensive whereby 20 items were included in the survey. This includes questions on graduates’ evaluation of the programme, perception on quality of teaching and learning, and perception on industrial attachment.

The tracer study instrument tried to capture too much information on the graduates’ experience as a student at the seven (7) VTIs, but not a lot on the graduates’ employment status. Yet, only limited analyses have been done to these data. As a result, most of the information in the data collected from the previous tracer studies remains to be fully extracted and systematically collated.   

Three (3) separate instruments were thus developed for the ES – each to be filled by different groups. First is the ES form, a short form to be filled in by graduates on their employment status. These forms are then returned to GCs who will then fill in the GC form, a Microsoft Excel form. Usually GCs would be given one month to distribute and collect the ES form. Lastly, the RC form, needed to be filled in by RCs who would then compile the GES forms from GCs for each VTI before submission to R & D.  

In the ES form, in addition to the three (3) items from the tracer study, graduates were also asked the following:

If status is unemployed, the reason for being unemployed will be asked. The rationale of this is to enable us to differentiate those who are unemployed by choice.

If status is undergoing further study, the place of study will provide more information on our graduates’ progression destination.

If status is employed, the new ES instrument will also ask for the employers’ affiliation, job title or post, graduates’ salary per month, and the duration to find job.

Questions on graduates’ student experience were no longer included in the survey as these aspects are being covered in a separate survey (Student Evaluation Survey) conducted at the end of their course. By doing so, the ES can be more focused in delivering its aims and objectives.

Improvements in study approach

Other than the survey instrument, issues in the data analysis approach and report layout and format were also addressed. For example, common highlights from the previous tracer studies was the employment rate, where the number of students employed was expressed as a percentage of the total number of graduates. The convention, however, considered only those who were actively entering the job market when calculating the employment rate (MOE 2014; NCVER 2014). 

The previous tracer studies also relied heavily on the RCs to manually produce a summary table on status of graduates by programme for each VTI, based on the hardcopies of the survey instruments filled in by GCs. This step is time-consuming and would have benefited greatly from simple spreadsheet automation. The pilot ES saw an opportunity to produce a better workflow for RCs whose task now is to cross-check the summarized data as keyed-in by the GCs, thus improving data accuracy.   

2.3 Procedures

2.3.1 Pilot Study

To ensure the validity of the instrument and that the questions are easy to follow, a pilot study was conducted on a group of 10 students whom are not part of the sample of this survey. Feedback included the need for the addition of Malay language translation to the instrument, and rephrasing of instructions, to give clearer instructions to the respondents.

2.3.2 Sampling approach and data collection strategy

The sampling frame for the ES would include all graduates from the seven (7) national VTIs (now combined under IBTE), by convocation cohort. Depending on the frequency of the survey being conducted, some other graduates’ employment surveys would consider a bigger sampling frame by combining several different convocation cohorts (Cambrian College 2014; Nel & Barnard 2010). Since our ES is an annual survey, the sampling frame is deemed appropriate and has been employed by other institutions for a similar survey such as Singapore’s GES (MOE 2014) and Australia’s ES (NCVER 2014). 

Prior to the data collection phase, R & D would call up a meeting with the seven (7) RCs to brief them on the list of programmes and groups of students to be surveyed. The RCs would then notify the GCs involved. The graduates are contacted by their respective GCs to provide the information required, either by filling in the ES form directly, over a phone, or face-to-face interview with the GC. The whole process flowchart is given below (Figure 1):

Figure 1: Framework of the administration of the Employment Study

2.3.3 Response rate

At the time of writing, IBTE is conducting the third round of ES since its commencement, which focuses on graduates from the 21^st National Convocation cohort. The first ES, based on 854 graduates of the 19^th National Convocation, saw a very high response rate of 97.7%. However, our most recent completed ES, which reported on 1013 graduates of the 20^th National Convocation cohort, had a response rate of 90.9%. In both ES, the response rates varied across different programmes. The discrepancies between the first and the second ES response rates may have been affected by a change in the appointment of RCs.

Considering the relatively small number of graduates to be surveyed each year (< ~2000 graduates in total), we therefore suggested for each VTI to ensure response rates of at least 90% for future ES. This is to ensure that results disaggregated by programme are equally reliable.

2.4 Data analysis and key findings

Data compiled by RCs from each VTI were assembled into one national database, to be stored and maintained by R & D. Descriptive analyses of data were done in Microsoft Excel. Results obtained can then be disaggregated by institution, programme, field of study and/or qualification level. Key findings of the last two (2) ES is given below (see Table 1).

Table 1:          Key findings for 2013 and 2014 Employment Study.

It should be noted that in calculating the employability rate, the number of graduates who were unemployed by choice (e.g. chose not to work, or waiting to further study) is not included. Prior to the Brunei TVET transformation process, the SMT together with senior officials from the Ministry of Education had set targets for both KPIs, by benchmarking against common practices in the region (SPU, pers. comm.). Accordingly, IBTE aims for:

• An employability rate of 80% by 2017, and
• An employers’ satisfaction rate of 70% by 2017.

Overall employability rate for the first ES was 67.3%, with more than half of the programmes (25 out of 31 programmes offered) having an employability rate of more than 50%. However, only five programmes reached the targeted KPI (i.e. employability rate >80%). The second ES witnessed a slight improvement in the overall employability rate for IBTE graduates, which was 71.2%. Subsequently, ten programmes managed to reach the targeted KPI for 2014.

Two (2) types of outputs are produced from ES, (i) full report, and (ii) brief report. The full report would document a comprehensive summary of our graduates’ status six (6) months after convocation, to be shared amongst the SMT and relevant stakeholders (e.g. curriculum division) for planning and decision-making. The brief report, on the other hand, is a less technical and more graphical summary of the survey with general key points, usually to be shared with the public (e.g. industry, parents, students, secondary school counselors, etc.).     

2.5 Challenges and improvement over time

Several challenges and limitations identified from the first ES include difficulties faced by GCs in getting in touch with their previous groups of students, and the difficulties for some GCs and RCs to fill in the simplified electronic forms (i.e. GC and RC forms). Also noted were the difficulties in tracking the graduates especially when the GC for a particular group is no longer available in the system – usually due to retirement, on study leave, or has been transferred elsewhere.

Also perhaps the main reason that lead to poor cooperation from the GCs in the first ES was the lack of a sense of importance of the survey. While the KPI itself is not a new concept and can be easily understood, prior to the survey there was no official statement relayed to the teaching staff regarding the integration of the KPIs as IBTE’s performance measures. Following this, efforts introduced to strengthen the system include issuance of official statement on the KPI, as well as sharing sessions with the teaching staffs regarding the survey.    

Accordingly, the ES relies on GCs to contact graduates whom by then would have left the VTI system. Hence, one way forward is for the GC to develop ongoing contact with students, to ensure a high response rate.

3 Employers’ Satisfaction Survey (ESS)

3.1 Conceptual framework

The Employers’ Satisfaction Survey (ESS) is another survey conducted by R & D, IBTE which measures the employers’ (i.e. supervisors) level of satisfaction with our employed graduates. Generally, the survey aimed to obtain employers’ or supervisors’ perception of our graduates, not to evaluate the graduate. Therefore, it was agreed that the ESS will be an annual survey based on follow-up with the employed group of graduates identified from the ES (i.e. convocation-specific group). This is to enable results from the ESS to be used as one of the institution’s KPI.

Indeed graduate employment is one of the main objectives of TVET. As such, employers become an important stakeholder in TVET – employer perceptions of graduates to enter the workplace forms an essential part of quality assurance. No such study has ever been done either by IBTE or any other higher educational institution in the country. The goals of this pilot ESS project, therefore, were (i) to increase knowledge of and responsiveness to the TVET needs of the employers of our graduates, (ii) to evaluate the performance of our VTIs in producing graduates with appropriate skills and knowledge, and (iii) to build a working framework for future ESS surveys.

3.2 Study approach

In developing the instrument to be used in the pilot ESS, the project was informed by a comprehensive literature review of similar documents and their execution in other educational institutions or agencies (Oliver, Freeman, Young, Yu, & Verma 2014; Lujan 2001; Colleges Ontario (n.d.); Hakobyan 2010; Saskatchewan Apprenticeship & Trade Certification Commission [SATCC] 2013). These are displayed in Table 2. Few studies reported that it can be extremely difficult to contact employers for a survey on graduates’ satisfaction without being able to refer to a specific graduate (Oliver et al. 2014). Thus, most ESS tend to contact its graduates first to identify the supervisors to be surveyed, although the alternative (i.e. directly approach supervisors without having to contact graduates) had also been employed in some surveys.

Table 2:          Comparisons of Employers’ Satisfaction Study (ESS) from several educational institutions worldwide.

The pilot ESS focused on direct supervisors of graduates of the 19^th National Convocation (who were involved in the first ES described in previous section). These direct supervisors are not necessarily the managers or head of departments of the company/organization, but rather those who are familiar with the day-to-day work and performance of the graduates.

3.2.1 Development of study instrument

The survey instrument was designed to be simple and short, to make it easy for the respondents to fill in the survey in less than ten (10) minutes. The instrument draft draws on commonalities with other ESS, but mostly guided by the five clusters of ESS framework as identified by the Australian ESS report (Oliver et al. 2014), namely foundation skills, adaptive skills, teamwork and interpersonal skills, technical skills and domain-specific knowledge and employability skills. Our survey instrument, therefore, consisted of 14 items related to indicators of perceived satisfaction on the work and performance of our hired graduate as well as employers’ views on TVET in Brunei in general. The questions were relatively highly reliable (internal consistency reliability, Cronbach’s alpha = 0.917), and the instrument underwent the process of triangulation with a group of IBTE staff who were supervisors to attachment students to further ensure reliability and validity of the survey questionnaire.   

3.3 Procedures

3.3.1 Pilot study

A pilot study was conducted on five (5) supervisors from a selected oil and gas company. Changes were made accordingly based on their feedback to ensure that the questions are easy to follow.

3.3.2 Sampling approach and data collection strategy

Prior to surveying, a list of employers’ contact details was compiled by calling the graduates individually. During the survey, where graduates could not be contacted, direct contact was made with the company, either through email, phone or by meeting the supervisors face-to-face.

To ensure the survey was representative, a minimum effective sample size was calculated. Initially, supervisors to 152 graduates were required for the “employed graduates” population to be representative. However, it later became apparent that some of the graduates were no longer employed for a variety of reason (for instance, to continue with further study, personal choice to not work, etc.). As a result, the total number of employed graduates with employers’ details was reduced to 325 graduates (i.e. population size). Allowing for non-compliance, a total of 97 respondents were obtained. Therefore, with a graduate representative rate of 30%, the sample size was representative at 90% confidence interval with an accepted error of 6.9%.

Surveying was conducted over five (5) weeks, from 16^th April 2014 till 24^th May 2014. This was around six (6) months after the ES (or about a year after their convocation ceremony). It was agreed that only employed graduates as identified from the ES would be contacted (i.e. not all graduates re-contacted), to ensure that graduates have been working for at least 6 months with the company/organization. This is to address the issue with some companies that have their own policy whereby supervisors can only evaluate employees after six (6) months of being hired by the company.

3.3.3 Response rate

Our pilot ESS had a supervisors’ response rate of 52.4%. The majority of respondents were employees of companies related to the oil and gas industry (29.9%), followed by those providing professional and technical services (16.5%) and companies in the construction or construction-related industry (10.3%) (Figure 2).

Figure 2: Distribution of respondents by field of industry of their employment (n=97)

Some supervisors were emailed at least three (3) times, first being the initial introduction email, a second follow-up email and a final reminder email. Where additional contact details were available, some of the follow-up was also made via phone, before being directly approached. Yet, still a number did not respond to the survey.

3.4 Data analysis and presentation of key findings

Data was keyed-in automatically via the built-in function when creating online questionnaire using Google Form. For data analysis, the dataset was transferred into Microsoft Excel to enable descriptive analyses to be carried out. 

Accordingly, almost all of the employers (except 3 which declined to answer) are either satisfied (64.9%) or very satisfied (35.1%) with the overall work and performance of hired graduates, resulting in an overall employers’ satisfaction rate of 96.5%. Figure 3 also shows their level of satisfaction based on the field of industry of employment.

Figure 3: Employers’ level of satisfaction on overall work and performance of graduates by industry division (n=97)

Similar to ES, results of the analysis from ESS were presented either in a full report or a brief report. The data collected also enabled results to be aggregated by field (industry sectors), and graduates’ previous programmes and previous VTIs, which is useful feedback for SMT to make informed decisions and assist planning. Additionally, this survey also provided an opportunity for employers’ feedback, comments and suggestions on TVET in Brunei, thus improving collaboration between IBTE and the industries.

4 Conclusion: Lessons Learned and the Way Forward

TVET in many countries, and some for quite a long time, is at a crossroad between a rapidly expanding education system and a fast-changing labour market. Redirecting planning from an institution-focused to a more client-focused approach is a priority objective of IBTE, embedded in its official mission, which is “to produce highly skilled and employable graduates that meet stakeholders’ expectation through holistic learning environment”. In pursuing international best practice, IBTE has developed and assigned two KPIs that are aligned with the institution’s strategy that work together to promote more effective decision-making in order to respond to the labour market needs. Results from the two pilot surveys have highlighted issues and potential limitations, shown direction to improvements, and set baseline values for future use. There is a need for TVET providers to invest in a systematic outcome based measures to ensure that the system is aligned to the labour market.

References

Amalina, H. & Chin, W. K. (2012). A longitudinal study on the employability of Business school graduates of vocational and technical education in Brunei Darussalam. Paper presented at the Brunei Darussalam Teacher’s Day International Conference 2012, “Crafting Innovative Instruction: A 21st Century Perspective”, ICC, Brunei Darussalam, October 10-11, 2012.

Cambrian College (2014). Graduate employment report 2013 – 2014. Online: http://docplayer.net/7664255-Cambrian-college-2013-2014-graduate-employment-report-cambriancollege-ca-sudbury-ontario.html (retrieved 09.01.2017).

Colleges Ontario (n.d.). Key performance indicators. Online: http://www.collegesontario.org/outcomes/key-performance-indicators.html (retrieved 09.01.2017).

DTE (2010). Tracer study report. Brunei Darussalam: R & D.

Hakobyan, N. (2010). American University of Armenia: Employer satisfaction survey of AUA graduates. Online: http://iro.aua.am/files/2012/03/2010-Employer-Satisfaction-Survey.pdf (retrieved 09.01.2017).

HKUST (2014). Hong Kong University of Science and Technology – Graduate employment survey. Online: https://career.ust.hk/web/page.php?sid=25 (retrieved 09.01.2017).

Loadman, W. E., Freeman, D. J., Brookhart, S. M., Rahman, M. A., & McCague, G. J. (1999). Development of a national survey of teacher education program graduates. In: The Journal of Educational Research, 93, 2, 76-89.

Lujan, S. S. (2001). The University of Texas Pan American – Employer Satisfaction Survey.

Ministry of Education, Brunei Darussalam (2012). The Ministry of Education Strategic Plan 2012-2017. Brunei Darussalam: Strategic Management Unit. Online: http://planipolis.iiep.unesco.org/upload/Brunei%20Darussalam/Brunei_Darussalam_Strategic_Plan_Book_2012-2017.pdf (retrieved 09.01.2017).

Ministry of Education (Singapore) (2014). Graduate employment survey. Singapore: Author. Online: https://www.moe.gov.sg/docs/default-source/document/education/post-secondary/files/nus.pdf (retrieved 09.01.2017).

Nel, H. & Barnard, A. (2010). Graduate Employability: A case study of Nelson Mandela Metropolitan University. Online: http://www.saqa.org.za/docs/pres/2010/nel_h.pdf (retrieved 09.01.2017).

NCVER (2014). Australian vocational education and training statistics – Student Outcomes 2014. Online: https://cica.org.au/wp-content/uploads/Student-Outcomes-NCVER.pdf  (retrieved 09.01.2017).

Oliver, D., Freeman, B., Young, C., Yu, S., & Verma, G. (2014). Employer satisfaction survey: Report for the department of education, June 2014. Online: https://docs.education.gov.au/system/files/doc/other/ess_final_report_june_14_0.pdf (retrieved 09.01.2017).

Parmenter, D. (2012). Key performance indicators for government and non profit agencies: Implementing winning KPIs. Hoboken, NJ: John Wiley and Sons, Inc.

Peng, F. & Zhang, J. (1997). On the education of information resource management professionals at high level in China. In Ward, P. L. & Weingand, D. E. (eds): Human development: competencies for the twenty-first century. Munchen.

Richardson, J. T., Slater, J. B., & Wilson, J. (2007). The national student survey: Development, findings and implications. In: Studies in Higher Education, 32, 5, 557-580.

Sageev, P. & Romanowski, C. J. (2001). A message from recent engineering graduates in the workplace: Results of a survey on technical communication skills. In: Journal of Engineering Education, 90, 4, 685-693.

SATCC (2013). Apprentice and employer satisfaction survey report. Online: https://dl.dropboxusercontent.com/content_link/CcA4ZZVLZhnDss9iX6wVG0ztiUSGpdKG2hd4NiBxiiayqV9ScbszZcZQDK9tB28r/file (retrieved 09.01.2017).

Citation

Ebil, S., Othman, N., Hjh Norhakimah, H.M.N., Marlinawati, H.A., Omarali, M, & Chin, W. K. (2017). Brunei TVET Transformation: The Development of the Institute of Brunei Technical Education’s two Key Surveys. In: TVET@Asia, issue 8, 1-15. Online: http://www.tvet-online.asia/issue8/ebil_etal_tvet8.pdf (retrieved 28.01.2017).

Life Skills are considered a critical aspect in engineering practice. Not only are these skills needed in order for an individual to get a job, but also to still remain employable. Despite all the efforts that have been done in trying to improve Life Skills assessment, the impact of its implementation in Brunei Darussalam still remains elusive. This study aims to help students develop these skills by integrating feedback within their assessments. Two groups were compared and analyzed during 4 weeks activity to see the impact of this strategy. Using both quantitative and qualitative method, the study sampled 48 students and 2 teachers with a Mechanical Engineering background. Paired t-tests were performed to determine participant group’s achievement of Life Skills at the end of the activity.  The results (calculated p=0.00) suggest there is a significant development in their Life Skills. However, in spite of the encouraging outcomes, there are still some major issues in its implementation. Qualitative analysis done with the instructors involved exposed feedback to being time consuming, labor intensive and it is dependent on the students to act upon it for improvement.

Key words: Life skills, employability skills, transversal skills, assessment, feedback assessment

1 Introduction

1.1 Background of the study

Life Skills have been defined as ‘psychosocial abilities for adaptive and positive behavior that enable individuals to deal effectively with the demands and challenges of everyday life’ (UNICEF 2015). Over the years, it has commanded a central place amongst engineering practice all around the world by industry representatives (Connelly & Middleton 1996) and engineering educators (Shuman et al. 2005). These skills are needed not only to get a job, but also to remain employable. Although a Life Skills programme has been implemented in Brunei’s Vocational and Technical Education (VTE) system over the past 12 years, the impact of this programme remains elusive which questions its potency. Further research revealed several barriers present in the current Life Skills programme (Kahar 2006; Rosli 2012; Paryono 2014). Since its implementation in 2001, the policy guide on life skills has been regularly reviewed and updated to meet the needs of both students and employers. From an implementation perspective, it was observed that there is room for improvement when conducting assessment for life skills, and that feedback to the students has not been stressed adequately in the current practice and is often over-looked.

Feedback is a vital part of assessment in terms of closing the skills gaps between students and industrial standards (Ramaprasad 1983; Sadler 1989). Providing students with feedback can help them develop and nurture these skills according to the set standards. The lack of feedback results in students not having any room for improvement which indirectly affects their employment performance. The importance of giving effective feedback should be emphasized among the teachers under Vocational and Technical Education (VTE) to ensure consistency across assessors and the quality of certification for graduate skills. Since the Life Skills programme in Brunei Darussalam is taken as part of “integrative’ courses to be embedded in each module, feedback offers an alternative teaching and learning tool for students to develop their skills. Although research done on the role of feedback in engineering education is sparse, promising findings on the use of feedback on writing (Srichanyachon 2012; Taylor 2011; Mackiewicz 2004) and communication skills (Bjorklund, Parente, & Sathianathan 2002) has prompted this research. 

1.2 Research Questions

Most research done on the assessment of Life Skills was conducted in western countries and in Australia mostly for Vocational Education and Training (VET) within industries but only a handful of studies were done in Vocational and Technical Institutions (VTI). Therefore, significant approaches between these two frames may differ. The main aim of this study is to investigate the impact of feedback in assessment of Life Skills in a Vocational institution in Brunei Darussalam. Hence, with this aim in mind, the researcher sought to answer the following research questions.

1. Does integrating feedback into assessment improve the students’ learning in Life Skills?
2. What are the perceptions of the teachers towards the use of feedback in assessment of Life Skills?

2 Literature Review

2.1 Defining Life Skills

With pressure from government and employer groups for change, the time for considering Life Skills in academic environments has come. These skills refer to non-technical skills needed to effectively function as an employee and work for the betterment of oneself and the organization (Bakar, Mohamed, & Hamzah 2013). In Brunei Darussalam, Life Skills are integrated through modules in which eight Life Skills are to be assessed. Figure 1 is a brief description of each component of Life Skills as stated in the curriculum.

Figure 1:         Description of the eight components of Life Skills

2.2 Feedback

Feedback is defined as “information provided by an agent (e.g. teacher, peer, book, self, experience) regarding aspects of one’s performance or understanding” (Hattie & Timperley 2007). The extant literature is clear that feedback is arguably the most important aspect of assessment in its potential to affect future learning and student’s achievement (Black & William 1998; Gibbs & Simpson 2004; Hattie & Timperley 2007). Although an assessment of hundreds of meta-analyses from 180,000 studies showed that “the most powerful single moderator that enhances achievement is feedback” (Hattie 1999), this is not always the case. Kluger and DeNisi (1996) found that more than one third of the effects indicated negative impact on learning. Based on further research, Nicol and Macfarlane-Dick (2006) found that effectiveness of feedback is dependent on the construct of teachers’ feedback. According to Villiers (2013) in-depth review, there are seven principles of highly effective feedback as illustrated in Figure 2.

Figure 2: Seven Principles of Highly Effective Feedback (adapted from Villiers (2013))

Feedback within assessment is an integral part of the learning process because it aims at to contribute to the progression of learning and hopefully will support students’ improvement in Life Skills. Feedback is one tool that can be used to help students master their difficulties. With this strategy, the researcher is able to target in which area the students are struggling with respect to eight skills set by BDTVEC’s (Brunei Darussalam Technical and Vocational Education Council) Guide (2012). The feedback intervention will be done according to the Seven Principles above to ensure the highest quality feedback given to the students.

2.3 Types of feedback

The aim of feedback is to increase student knowledge, skills, and understanding in some content area or general skill (e.g., communication, teamwork, problem solving), and there are multiple types of feedback that may be employed toward this end (e.g., response specific, goal directed, immediately delivered) (Shute 2007). It is therefore, the teacher’s responsibility to acknowledge the different types of feedback and to know when to use them. Gilbuena et al. (2015) has summarized them as shown in Table 1.

Table 1:            Types of Feedback

For the purpose of this study, feedback was considered to be directive whereby each intervention has unique learning outcomes that are tailored to each participant’s needs.

3 Methodology

This study is designed to investigate the impact of using feedback in Life skills’ assessment in Vocational and Technical students in Brunei Darussalam using both quantitative and qualitative data. In this study, students’ achievement will be measured as quantitatively whilst the teacher’s perceptions will be analyzed qualitatively to tie the results together. The convenience samples of this research are two groups (48 students) of Mechanical Engineering Skills Certificate 2 level students. The two teachers, who were interviewees for this research, have had years of teaching experience. They were interviewed at the end of the study.

The alignment of research questions and data collection instruments is presented in Table 2.

Table 2:            Outline of instruments in relation research questions

Quantitative Data

• Life Skills Grading Criteria Assessment. The Grading Criteria Assessment was adopted from the BDTVEC’s 2012 policy guide. All the eight Life Skills components were assessed based on their performance in the activity. Data from both groups were measured and compared using the before-and-after design. The ‘before’ observation is established before the intervention, and was compared with the ‘after’ observation at the end of the study (Kumar 2005). In this study, their first semester’s Life skills results are taken as baseline results.

Qualitative Data

• Semi-structured interview. With relation the second research question investigating the teachers´ perceptions towards integrating feedback into grading of Life Skills, the researcher conducted semi-structured interviews with the two teachers who assessed the students during the activity. The findings from these interviews were beneficial as it could be used to elicit respondents’ perceptions, as well as having different perspectives than the researcher. The interview was done at the end of the activity. It was recorded and transcribed to validate and assist the evidence on the printed document.

Research procedures

• Feedback strategy. The design of this study consists of two groups, which are the ‘Feedback’ group and the ‘Conventional’ group. This type of ‘before-and-after’ design is used to compare and determine whether the intervention has an effect on the achievement of the students. The conventional group on the other hand, is assessed traditionally as per usual where the teacher records and grades without guiding the students. Reflecting from the procedures done in several case studies reviewed by Juwah et al. (2004), the standard procedure in implementing feedback into assessment is a step-by-step procedure as shown in Figure 3.

Figure 3: Schematic diagram of research design for both groups

It is crucial to dedicate the initial stage of learning towards briefing the students with what is expected from them. By introducing Life Skills at the early stage, it will enable them to know what the requirements are and goals they need to fill in order to do well in Life Skills.

The first part of the activity (practical session) requires the students to work in groups of four authentic learning was used to test their critical thinking skills in relating real-life environment with theoretical knowledge. The students were assessed using the rubric with respect to the eight Life Skills (see Figure 1 above) during the practical sessions. Each teacher assessed two groups of students simultaneously during the course of 4 weeks of activity.

Once the students have completed their lesson, the teachers administered a face-to-face feedback intervention for their respective groups. In order to ensure the reliability of the assessors, the researcher was present in assisting the two teachers when giving feedback to the students for consistency measures. The researcher focused on the areas within the eight Life Skills components that needed further improvement. Students provisional grades were shown to them according to the assessment criteria for the Feedback Group. Sadler (1989) argues that students must know: what good performance is (the concept of a goal or standard); how current performance relates to good performance; and how to act to close the gap between current and good performance.

4 Results and discussions

4.1 Quantitative data

Results of the students from their first semester were taken as baseline to compare the two groups. For both semesters, assessment of Life skills was done using the Rubric Evaluation provided by the BDTVEC’s 2012 policy guide. Analysis of the data for the Feedback group showed that the majority of the students improved their skills over the four weeks course of the activity as evidently shown in Table 3. It was observed by the assessors that the students were more aware of what was expected of them and knew they were being assessed throughout. This resulted in their putting in more effort to enhance and present their skills and this was reflected in their final and baseline results comparison.

Table 3:            Individual Baseline and Final Life Skill’s results (both groups)

Most of the students in Feedback Group managed to improve the performance criteria and scored higher than they did in Semester 1. The overall mean score for the group is 67% However, this was not the case for students F11 and F20, their mean scores did not improve as much as others and only made a point difference from Semester 1 for the case of Student F20. This student in particular was monitored closely as it was noticed in the beginning that he needed more attention in developing these skills. It may be argued that Life Skills are leaning more towards habits, and therefore take time to change. The student may not be able to perform and give their best as there is a limited time of four weeks to do so. Life Skills is known to be enhanced and practiced over a period of time in order for it to be embedded into our everyday lives. When Student F20 was confronted during one of the feedback sessions, he admitted that he did not comprehend the teacher’s feedback as the criteria are too generalized.

On the contrary, for students who were assessed conventionally (without feedback) through their activity, they did not improve as much and remained stagnant. In addition, most of them scored within the passing mark margin of 50%, which is only considered as ‘Satisfactory’. Although at the beginning of the activity both groups were briefed with necessary knowledge on the eight Life Skills components, it could be argued that the Conventional group were not able to perform as well as Feedback group because of the different instructional design given, where the first group had feedback from the assessors themselves and able to practice them throughout the activity. For example, ‘Self-management’ skill, the students from Feedback group were advised and encouraged to record what they have done into their laboratory book as usually done in industry. However, the Conventional group was not given any directive nor facilitative feedback to remind them to do so and hence, they did not carry out the task professionally and this was reflected in their scores. Therefore, this result suggests that there is a difference in the scores between the two groups.

To further validate the difference between performance achievements of the two groups, the results were further analyzed using IBM Statistical Package for Social Sciences (SPSS) Version 21.0. Both mean scores and standard deviation of score distribution were calculated using paired t-test at which scores of two different variables with the same group of cases are compared. The null hypothesis used is there is no difference between the mean results of the students exposed to feedback given by the teacher and the students who were assessed as usual i.e. both groups should perform similarly. The descriptive result of the paired t-test was summarized in Table 4 below. Results from the paired t-test on the baseline means scores revealed that there is not much difference between the two groups (XA=51.5; XB= 50.8). This result suggests that the students have the same level of Life Skills and this has been the trend in Vocational institutions in Brunei Darussalam. During Semester 1, teachers assessed the students without guiding them on how to improve and were merely dependent on the student’s own effort. Paired t-test was used because both groups were matched since the beginning of the study, both coming from Skills Certificate 2 level and performed satisfactorily in Semester 1.

Table 4:            Individual Baseline and Final Life Skill’s t-test results (both groups)

Results from the paired t-test on the baseline means scores as evidently shown in Table 4 revealed that there is not much difference between the two groups (XA=51.5; XB= 50.8). This result suggests that the students have the same level of Life Skills and this has been the trend in Vocational institutions in Brunei Darussalam. During Semester 1, teachers assessed the students without guiding them on how to improve and were merely dependent on the student’s own effort. Paired t-test was used because both groups were matched since the beginning of the study, both coming from Skills Certificate 2 level and performed satisfactorily in Semester 1.

Although the baseline results were almost similar for these two groups, their final results suggests otherwise. As seen from Table 4 above, there is a significant difference on their final Life Skills’ results obtained by the two groups (XA=66.9; XB= 50.9). The p value for Feedback group were calculated to be less than 0.05 (p=0.00) and hence validates the fact that there is a significant difference between their baseline and final results. In the case of Conventional group, the p value is more than 0.05 (p=0.857).  Hence, with these data in hand, the null hypothesis can be rejected and it can be claimed that there is an improvement in the Feedback group compared to the Conventional group.

Qualitative data

The qualitative data from the semi-structured interviews were conducted with the two teacher participants who were involved in assessing the students in order to find out their views and preferences towards the practice of giving feedback. This was beneficial as it would provide a valuable insight for this study in determining the limiting factors if feedback were to be integrated in the future. The following were some of the prominent findings noted by the teacher participants on the feedback intervention:

Positive influence

(a)        Teacher’s competency on Life Skills

Mainly, the most common reason for preferring teacher feedback is that the teacher knows the standards that determine students’ grades. The comment made by T1 when responding to the question ‘In what ways do you think feedback can be used to enhance students’ Life Skills?’

“…Personally speaking, I think feedback is important for students to improve their skills. Through teacher feedback, it actually act as a guide on how they are expected to perform when they work in industries. Teachers like us have years of trainings and we have been working close with the industries. I don’t think the student knows how Life Skills is like in working environment because they haven’t gotten any experience yet. If we were to just let them be and just see them perform without feedback and giving our perspective on how they perform tasks. They will be stuck on whatever they know…they don’t know where to go”.

The comment above shows how important is the role of teacher’s feedback in improving students’ Life Skills. She deduced that teacher feedback would be useful if it were to be integrated because teachers have years of experience and a better grasp on what is the standard expected by industry – the students’ potential employers. It is however, important to take note that new teachers may not be as competent as senior staff members.

(b)       Teacher feedback motivates students

Another main finding that was commented by T1 was the fact that teacher’s feedback motivates students in trying to improve their Life Skills. This is aligned with several studies done on feedback (Hattie 2009; Zacharias 2007; Srichanyachon 2012)

“Based from my experience in this study, I noticed that students are actually motivated to improve their skills. They feel as though it’s like a glimpse into their future profession. Through my feedback, they were able to act upon it as they see its importance and how it relates to them. As the end prize, they will have provisional grade for their effort during the feedback session and this then furthered their willingness to improve”

This is also supported by numerical analysis done where 92% of the students feel motivated from the teacher’s feedback. It was claimed by Gilbuena et al. (2015) that in such context, students are more likely to integrate feedback into their Life Skills because they will view these skills as an integral part of what an engineer does.

(c)        Teacher control grades

It was found that students preferred teacher feedback for the sole reason that teachers have control over their grades (Zacharias 2007). In this study, the teachers detected that students behave differently when around the assessors, even when it’s not during class. She explained that the students were aware that their grades may deteriorate if they behave differently when in lesson and outside of class as Life Skills covers an overall attitude of a student. This is a concern as it questions whether the students only behave professionally in front of the assessors; feel ‘motivated’ to improve in order to improve their grades. This finding is indeed worrying as the students are more interested in their grades rather than seeing the true light of Life Skills in their everyday practice.

Issues and limitations

Research has shown that there are still a number of issues and limitations existing in the integration of feedback in assessment, which has also been pointed out by the interviewees. There are two main limitations that were experienced by these teachers. As expressed by T1 and T2 respectively:

(a) Teacher feedback is time consuming and labor intensive

“ .. I would say that giving feedback is time consuming because we have these sessions where we provide students with feedback during lunch breaks. I wouldn’t mind doing it but it’s also taking up the students’ time… in terms of giving feedback, we have to take time and analyze and assess each student with what they have done during the activity… and it’s a bit tedious. Maybe it’s okay if it’s a smaller group of students I guess…but definitely not for a whole class. Another thing is when we had more than 1 assessors, because I don’t think it’s possible to do that in a long run”

“I think feedback is one sided… teachers make most of the work to give feedback and assess them according to the grading criteria. They somehow have to correlate these two together which takes a lot time and effort…Honestly speaking, I do not think most teachers would want extra work”

(b) Students’ initiative to use feedback

Secondly, the teachers were concerned in terms of ensuring the students use the feedback given to them. It is possible that students do not pay attention to comments because they don’t make sense to them (Duncan 2007) or that students simply do not see its importance in their education. T2 expressed:

“Giving them feedback is good. But there is a limit to that because it’s like spoon-feeding. There are times when there’s no initiative from them and feedback that is when our feedback becomes useless, because in the end, it’s up to the students whether they will use it or otherwise. Even if they do, results still vary with students.”

Discussions

With reference to the results to answer Research Question 1, the use of feedback in enhancing students’ Life Skills in Brunei Darussalam was achieved in this study. The independent t-test for their final result has shown that there is a significant difference (P < 0.05 (0.000)), which indicates an improvement in learning. Although they did not manage to achieve the highest industrial standard, their final results have given promising evidence that the feedback strategy aids the improvement of their scores. Perhaps over a longer period, they would be given more opportunity to improve their Life Skills beyond academic achievements.

In exploring the answer to Research Question 2, it gave an inside perspective as to some of the advantages, challenges and limitations in providing feedback to the students. Most of all, the assessors are in favour of the learning strategy. It was noted that feedback has a positive impact towards the student’s achievement as they have this cultural belief that teachers are more competent as they have more experience. This, therefore, reflects on their marked preference to have feedback embedded in the assessment of Life Skills. Interestingly, however, it was also shown, from the students’ view, that the students were aware that teachers control the grades. This is indeed worrying. Zacharias (2007) found teacher feedback contributes greatly to students’ emotional states mainly towards their motivation and attitudes. While too much feedback would make the students depressed, little feedback was considered more motivating because, for the students, it indicated fewer mistakes. The assessors have also pointed out that feedback is only effective if the students make the initiative to improve. Therefore, the construct of feedback plays an important role in determining the students’ attitude. Directive feedback was also said to be time consuming for both parties, therefore, it is crucial to connect both professional skills and technical skills together instead of having to sacrifice time. Nevertheless, the results from this study are limited in terms of time and opportunities. Hence, additional research is warranted.

5 Conclusion

It is widely recognized that feedback is a significant part of a learning cycle to improve performance. Life Skills on the other hand, is a collective of skills that needs to be embedded into students to ensure that they will become successful members of society especially in their working environment. In this study, this was attempted by integrating feedback in assessment of Life Skills. Overall review has shown positive impact in terms improvement in students’ achievement. Feedback on Life Skills helps students recognize how to incorporate and demonstrate these skills into their activity. In addition, these students have shown positive attitudes towards this strategy in their learning. Nevertheless, a few have shown otherwise due to unclear feedback provided. However, there are still limiting factors present in integrating feedback as noted by their assessors such as it being time consuming, labor intensive and it is dependent on the student’s willingness and initiative to improve. This study has shown a valuable insight towards the power of feedback and its implications on Life Skills.

References

Bakar, A. R., Mohamed, S., & Hamzah, R. (2013). An Assessment of Workplace Skills acquired by students of Vocational and Technical Education Institutions. In: International Education Studies, 6, 11, 15-20.

Bjorklund, S. A., Parente, J. M., & Sathianathan, D. (2002). Effects of faculty interaction and feedback on gains in student skills. In: Journal of Engineering Education, 93 (2), 153-160.

Brunei Darussalam Technical and Vocational Education Council (BDTVEC) (2012). BDTVEC Guide: Life Skills Implementation and General Teaching Practice, BDTVEC, Brunei Darussalam: Government printing department.

Black, P. & William, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5 (1), 7-74.

Connelly, J. D. & Middleton, J. C. R. (1996). Personal and professional skills for engineers: One industry’s perspective. In: Engineering Science and Education Journal, 5 (3), 139-142.

Duncan, N. (2007). Feed-forward: improving students use of tutor comments. In: Assessment & Evaluation in Higher Education, 32 (3), 271-283.

Gilbuena, D. M., Sherrett, B. U., Gummer, E. S., Champagne, A. B., & Koretsky, M. D. (2015). Feedback on professional skills as enculturation into communities of practice. In: Journal of Engineering Education, 104 (1), 7-34.

Gibbs, G. & Simpson, C. (2004). Conditions under which assessment supports students’ learning. In: Learning and Teaching in Higher Education, 3-31.

Hattie, J. A. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement.  Oxford: Routledge.

Hattie, J. A. & Timperley, H. (2007). The power of feedback. In: Review of Educational Research, 77 (1), 81-112.

Hattie, J. A. (1999). Influences on student learning. Inaugural lecture, University of Auckland, New Zealand.
Online: http://projectlearning.org/blog/wp-content/uploads/2014/02/Influences-on-Student-Learning-John-Hattie.pdf (retrieved 16.02.2017).

Juwah, C., Macfarlane-Dick, D., Matthew, B., Nicol, D., Ross, D., & Smith, B. (2004). Enhancing student learning through effective formative feedback. The higher Education Academy Generic Centre.

Kahar, I. N. (2006). Common skills: a hidden curriculum policy in its implementation stage. Unpublished M. Teach Research Exercise, Universiti Brunei Darussalam, Sultan Hassanal Bolkiah Institute of Education, Bandar Seri Begawan, Brunei Darussalam.

Kluger, A. N. & DeNisi, A. (1996). The effects of feedback intervention on performance: a historical review, meta-analysis and a preliminary feedback intervention theory. In: Psychological Bulletin, 119 (2), 254-284.

Kumar, R. (2005). Research methodology (2^nded.). Pansheel Enclave, New Delhi: SAGE publications.

Mackiewicz, J. (2004). The effects of tutor expertise in engineering writing: A linguistic analysis of writing tutors; comments. IEEE Transaction on Professional Communication, 47 (4), 316-328.

Nicol, D. J. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. In: Studies in Higher Education, 31 (2), 199-218.

Paryono (2014). Transferable skills in Technical and Vocational Education and Training (TVET) in Brunei Darussalam. In: TVET@Asia, issue 3, 1-15. Online: http://www.tvet-online.asia/issue3/paryono_tvet3.pdf (retrieved 09.01.2015).

Ramaprasad, A. (1983). On the definition of feedback. In: Systems Research and Behavioural Science, 28, 4-13.

Rosli, M. R. (2012). A study on the implementation of VTE common skills: A case study. Unpublished M.Teach Research Exercise, Universiti Brunei Darussalam, Sultan Hassanal Bolkiah Institute of Education, Bandar Seri Begawan, Brunei Darussalam.

Sadler, R. D. (1989). Formative assessment and the design of instructional systems. In:  Instructional Science, 18, 145–165.

Shuman, L. J., Besterfield-Sacre, M., & McGourty, J. (2005). The ABET ‘Professional Skills’ – Can they be taught? Can they be assessed? In: Journal of Engineering Education, 94 (1), 41-55.

Shute, V. J. (2008). Focus on Formative Feedback. In: Review of Educational Research, 78, 153-189.

Taylor, S. S. (2011). “I really don’t know what he meant by that”: How well engineering students understand teachers’ comments on their writing? In: Technical Communication Quarterly, 20 (2), 139-166.

United Nations International Children’s Emergency Fund (UNICEF). Definition of Terms. Online: www.unicef.org/lifeskills/index_7308.html (retrieved 26.05.2015).

Villiers, R. D. (2013). 7 principles of highly effective mangerial feedback: Theory and practice in managerial development interventions. In: The international Journal of Management Education, 68-72.

Zacharias, N. (2007). Teacher and Student Attitudes toward Teacher Feedback. In: RELC Journal, 38 (1), 38-52.

Citation

Pg Mahmud, S. N. A. & Chin, W. K. (2017). A Study on the Use of Feedback to Improve Assessment of Life Skills in Mechanical Engineering Course. In: TVET@Asia, issue 8, 1-14. Online: http://www.tvet-online.asia/issue8/pg-mahmud_chin_tvet8.pdf (retrieved 21.02.2017).