Reforming Assessment - Time for Change

The recent CBSE paper leak presents an occasion to consider wide-ranging reforms in the process by which we assess student learning in our schools to make it more pedagogically appropriate, less stressful, and relevant to modern society. At a time when public discourse on education commonly references 21^st century skills, a Board examination that only emphasises rote learning and the ability to memorise and reproduce facts is an anachronism that has outlived its utility.

The purpose of assessment should be to illuminate where a learner is on her path of learning at a given point in time. This implies that learning is a continuous, ongoing process that does not end at a particular stage, in keeping with the internationally accepted principles of lifelong learning. Equally importantly, the assessment of learning should lead to an understanding of a student’s ability to construct and apply learnt knowledge as opposed to her ability to memorise factual information that will not be retained after the examination.

A conscious move away from the current situation was made when the Right to Education Bill, 2005, was drafted on the basis of recommendations of a sub-committee of the Central Advisory Board of Education (CABE). While recommending the removal of all Board examinations at grade 8 level and a no-detention policy, the sub-committee also recommended the introduction of Continuous Comprehensive Evaluation (CCE) to ensure that learning was assessed regularly through the school year in order to track student progress. This came from the belief that labelling children examination failures at such a young age was harmful to their self-esteem and their ability to stay engaged with learning. The sub-committee recognised that the majority of children in the schooling system come from less privileged backgrounds, many of them being first generation school-goers, and failing them would have the immediate consequence of pushing them out of school.

As the then Director in the Ministry of HRD who assisted the sub-committee and worked on the draft, I can vouch for the fact that CCE at the elementary stage (grades 1 through 8) was expected to be driven by NCERT and customised by states based on their own specific requirements. It was never envisaged that CBSE, a Board of secondary education, would be responsible for its design and implementation, as eventually happened. The manner in which CCE was implemented carried within itself the seeds of its own downfall, increasing pressure on teachers and establishing a complicated assessment system that was never fully understood by those most affected by it.

Assessment of children’s learning should be an in-built mechanism, with teachers undertaking regular, ongoing monitoring of what has been learnt by the individual child so that corrective action can be taken in time – in other words, assessment for learning. This is of course, easier said than done, given the conditions under which many teachers function, especially in government schools, with multi-grade classrooms, and pupil-teacher ratios in excess of 100:1 in many states. Yet a portfolio containing work done by each child over a period of time remains the best indicator of student progress and can help to scaffold and support individual learning.

This in-classroom assessment should be accompanied by diagnostic, low-stakes, external assessments to indicate the health of the education system. The results of such assessments, such as the National Achievement Survey (NAS) recently carried out by NCERT, provide policy makers with clear information about the functioning of the system and help to identify shortcomings for correction. After the poor showing of Himachal Pradesh and Tamil Nadu in the Programme for International Student Assessment (PISA) 2009, India has stayed away from international learning assessments like PISA, TIMSS, and PIRLS, but there has been some discussion lately around the possibility of participation in PISA 2021. In such a situation, undertaking national diagnostic assessments that provide comparative data over time and across regions acquires renewed importance.

Using Board examination results for selection to higher education institutions has become an annual farce, with students scoring unrealistically high marks, pushing up the cut off levels each year. Eliminating the Board examination altogether and replacing it with an individual entrance test or a standardised assessment like an indigenously developed GRE- or GMAT-like test would remove stress and provide a reliable mechanism for selection. Combined with a portfolio describing a student’s progress over the years, this would provide a far more accurate picture of abilities than the current Board results.

With the establishment of the new National Testing Agency announced in the last Budget, there is an historic opportunity to initiate far-reaching reforms in the way in which we assess student learning, and to drag our education system into the 21^st century. It is unfortunate that the committee set up to review the CBSE leak has its mandate confined to the leaks only, instead of focusing on the big picture. The earlier experiment of CCE may not have been fully successful, but that should not deter us from putting in place more carefully designed assessment systems that help all stakeholders, and most of all, our children.

The Reality of Skilling India

Background

In the last few years, India has attempted to put in place policies and mechanisms to take advantage of her apparent demographic dividend, in recognition of the importance of skilling her young people before they reach the job market. The World Youth Skills Day (July 15^th) provides an opportunity to reflect on our current situation and the additional steps we need to take in order to achieve our goals.

The numbers are well known, but bear repetition – India’s strength over the next couple of decades is expected to lie in her demographic dividend, i.e., the 65% of her population that is under the age of 35, who are or will be the next productive generation.  Each month, just over a million young people join the workforce, yet only a minuscule number are formally trained. These young people can potentially fill a requirement for skilled personnel not only in India, but also in other countries across the world that will be dealing with ageing populations during this period.

But this significant demographic dividend brings challenges and opportunities in equal measure; any failure to address its obligations now will have significant consequences in the future, including some that may have an impact on our very social fabric.  Signs of this impact are already visible in daily newspaper headlines, which describe a sharp increase in petty and serious crimes, all too often perpetrated by poorly educated and unemployed youth.  An interesting analysis on this situation by my friend Amitabh Pandey can be found here.

Policy Interventions

On the policy side, the national skill development policy was revised in 2015 in the form of the National Policy on Skill Development and Entrepreneurship 2015.  A National Skill Qualification Framework separately prescribes competencies required at various levels of skill in the form of knowledge, skills and aptitude.  However, the supply side of training and skill delivery needs to be accompanied by expanding opportunities available to youth on the demand side; this implies bringing about an investment climate that encourages the creation of jobs.  FDI norms have recently been eased in 15 sectors, which have the potential to generate some 4 million additional job opportunities, yet much more needs to be done. The 2016-2017 Budget had also proposed steps to supplement the Make in India initiative, reinforcing the need to build capacities of Indian youth, yet one of the contradictions with which policy makers continue to grapple is the fact that while on the one hand there is unemployment, on the other, industries deal with large scale vacancies. Clearly this mismatch between employment seekers and employers is one that needs to be addressed on priority; the question to ask if we're seeing enough action on the ground.

Role of Industry

This process cannot be left to government alone – industry too needs to step up its engagement and accept responsibility as part of its national and social obligations.  There is also a need for stronger industry-academia interaction; we need to see high quality industry engagement with the actual process of skill development.  Creation of industry cells at ITIs and polytechnics to enable exposure to current methodologies and processes is one possible form of engagement.  In today’s world, technology changes rapidly, making it difficult for technical institutes to keep up. This is where industry can step in to complete the learning process and to help training institutes to stay abreast of current developments.  The example of the German dual system is often cited, precisely because it involves such a strong relationship between industry and training, with trainees completing part of their course on the job in their chosen fields.

Situation on the Ground

In a recent study that was carried out on behalf of CII Northern Region, it was found that young people are in general, quite willing to pay in order to acquire skills and skill certification, believing that such certification can lead to better opportunities; this willingness does not however, find immediate reflection among employers, who are not ready to pay a premium for skill training.  This contradiction arises partly from the fact that employers are not fully comfortable with or sure of the reliability and integrity of the assessment and certification processes, which underlines the need to strengthen these systems so that we have in place robust and dependable mechanisms for assessing and certifying the skill levels of employment seekers.  Nothing illustrates the criticality of such mechanisms better than the recent scandal involving exam toppers of the Bihar examination board.  However, the importance of industry recognising skill certification through a wage premium in order to encourage the acquisition of skills cannot be emphasised strongly enough – failure to do so could become one of the most important causes of our eventual failure in this area.

In order to achieve sustainable progress in vocational training and skill-intensive education, these will also need to be seen not as parallel options, but as supplementary streams to mainstream education.  This requires closer integration with the school system, something that the CBSE has recently started with its vocational education courses in grades 9-12.  What we need to see also is greater awareness generation about these options at the school level, not just among students but also among teachers and educational administrators. Skilling cannot be an ad hoc activity; it needs structure and clear targets in order to take up the massive task of skilling 400 million people. Holistic planning that involves all stakeholders thus becomes essential to bridge the demand and supply gap of the workforce.

Conclusion

The state of education in India is well known; systematic neglect over the years, prioritising certain forms of higher education over school education and general literacy, and overall mismanagement have led to a situation where more than half the children in grade 5 cannot read a grade 2 text.  We need to be certain we do not repeat this experience in the skill development sector.  Poor learning levels in school, combined with often questionable skilling courses and a reluctance to pay wage premium are a recipe for disaster.  We have only the next 2-3 years to get the formula right; if we don't, we're staring at a guaranteed demographic meltdown instead.

ASER 2014 - Looking Back Over a Decade

(This piece appears in ASER 2014, which was released earlier today in New Delhi; reproduced here by kind permission of the publishers, ASER Centre.)

Ten years ago, an ambitious and audacious idea was floated – why not have a people’s audit of government expenditure on education and produce a report for the common man?  In a conversation shortly after the 2% Education Cess was  introduced in 2004, I recall Madhav (Chavan, co-founder and CEO of Pratham) first proposing the concept, arguing that the people had a right to know where the Cess was being spent, and how effective it really was.  At the time, I was with the Ministry of Human Resource Development (MHRD) and grappling with the concept of the Prarambhik Shiksha Kosh (PSK), a non-lapsable fund we were trying to convince the Finance Ministry to create, in order to ensure that Cess revenues remained with MHRD to support elementary education.  In that context, Madhav’s idea seemed like a good one, but I had no inkling then of the scale at which he was proposing to execute it.

When it was finally carried out, the Annual Status of Education Report (ASER) 2005 covered some 490 districts and 3.3 lakh children, who were tested by volunteers from all walks of life, at home, in school, on the streets and in the fields, and just about everywhere in between.  Today the scale and scope of the reports have widened significantly, but for a first time exercise, it was bold and unparalleled in scale; quite simply, nobody had ever attempted anything like it before.  Most striking of all was its intent, captured by the “Preamble” to the report.  “We, people of India, from different states and regions, speaking different languages, sat with our children and looked within, inside our homes, at our villages, into our schools, and prepared this report for ourselves, to build a better India”.[1]  This was what set ASER apart from donor-funded or government surveys – it was a report of the people, by the people, for the people.

In 2004, Sarva Shiksha Abhiyan (SSA), the government’s programme for universalising elementary education, was in its third year of implementation.  Our concerns in MHRD at the time were primarily around provisioning and ensuring that all children were enrolled in school, in order to meet the first goal of the programme, viz., all children in school/EGS centre/bridge course by 2003.  It was already evident that the 2003 enrolment milestone had not been met, and all our efforts were thus concentrated on catching up.   When the findings of ASER 2005 were shared with us, some days before the formal release of the report, it was heartening to learn that the survey had estimated that just over 93% children of the appropriate age group were enrolled in school; this accorded well with the data being reported by the states, and seemed to indicate that SSA was having a successful impact on the ground.

The focus of most of the debate after ASER 2005 was more on the national enrolment figures; learning levels had also been tested, and the results did not fully match either popular perception, or the available NCERT data.  Later reports, treating the question of enrolment as more or less settled, have emphasised learning levels of children, testing among other things, to see where they stand, and exploring the differences, if any, between public and private schools.  These reports have also examined the availability of facilities in schools vis-à-vis those mandated by the Right to Education Act, and collected some basic economic information about households, such as possession of mobiles and TVs, etc.

The numbers coming out of the ASER 2005 survey also validated the results of another study commissioned by government and carried out by IMRB.[2]  The latter study estimated that some 1.34 crore or 6.94% children were then out of school, which approximated the ASER estimates of 1.4 crore or 6.6%.  The importance of the ASER results lay of course, in the fact that unlike the IMRB exercise which had been funded by government, they were an independent and non-partisan estimation.  While the IMRB report could conceivably be questioned as a "government statistic", the results of ASER were not so easily open to multiple interpretations.  Both the IMRB report and ASER 2005 were used extensively by government to provide evidence of the impact and effectiveness of its universalisation programme, at least in terms of improved enrolment figures.

While the enrolment data emerging from ASER has generally been viewed as encouraging, this has not always been the case with figures related to learning.  NCERT and several state governments disputed many of the findings, questioning both the methodology and the process adopted to determine learning outcomes.  In some cases, the hostility extended to actively banning Pratham from working with their schools, a challenge the latter overcame by working directly with village communities instead.

Somewhere around the fifth ASER report, a suggestion was made that perhaps there should be fewer reports; possibly one report every 2-3 years instead of a regular annual publication.  Many others who work in this sector no doubt shared my relief that this was one suggestion Pratham did not accept.  An annual ASER exercise and report have now become an integral part of the education landscape, serving to educate and inform stakeholders in the system and the public at large.

What impact has ASER had on the Indian education environment?  First, just the introduction of the concept of a "people's audit of education" was a game changer in itself; the People's Report on Basic Education (PROBE) was a one-off exercise and had not, at the time, been repeated, nor was it anything like ASER in its scope.  ASER reports have regularly held up a mirror to society, informing us of how much (or how little) our children have gained in terms of improved education levels, access to better schooling, and removal of inequities.  Note that its original purpose has not changed – ASER is still aimed at anyone who has an interest in education, not just policy makers or academics or other standard stakeholders in the system.

Second, the single most significant finding of ASER year after year has been the fact that learning levels across the country, whether in public or private school, have not improved.  Clearly, even after spending crores of rupees on delivering a Right to Education, our efforts have not succeeded as well as they should have; the policy prescription for shifting attention away from inputs to outcomes could not be clearer.

Third, and directly as a result of the above finding, ASER has succeeded in bringing the issue of learning centrestage; from a focus on ensuring that children are enrolled in school and that adequate infrastructural and teaching facilities are provided to them, the debate has now moved to a place where inputs are assumed, but the interest is in outcomes.  For the first time, the 12th Five Year Plan acknowledged that "there is a need for a clear shift in strategy from a focus on inputs and increasing access and enrolment to teaching learning process and its improvement in order to ensure adequate appropriate learning outcomes”,[3] explicitly agreeing that a more-of-the-same approach focused only on provisioning will not necessarily work.  While there will always be discussion around methodological approaches and whether ASER follows this or that method as opposed to others, the fact is that successive ASER reports have compelled us all to sit up and take notice of what is really happening inside schools.

Additionally, ASER has pushed both the central and state governments into commissioning their own assessments and analyses of the status of education in their schools, often in a move to defend policy and/or practice.  In many cases, these assessments do not produce the same results as ASER, partly since they are not comparable in terms of what is measured and who is covered, and there is often much controversy and hand wringing over the discrepancies, yet it is a moot point if such assessments would today be considered so essential if public perception had not been influenced so profoundly by ASER.

Fourth, ASER has been successful in highlighting an important trend in school enrolment – from only 16% children enrolled in private schools when ASER 2005 was carried out, the percentage has increased to nearly 30% in the last report.  Present trends seem to indicate that this number will increase to 50% by the end of the current decade. Given that this increase has taken place in rural areas, where much of the money spent on SSA and other programmes has been concentrated, this is not an encouraging development, and is one that merits serious reflection on the part of policy makers.

What should one now expect after a decade of this exercise?  Ideally, the annual reports should continue to raise the uncomfortable questions that they do today.  Perhaps there is now a case for a somewhat more sophisticated analysis of learning; not necessarily one that substitutes for say, a PISA or TIMSS, but one that develops a more rigorous indigenous model of assessment, feeding even more closely into policy making and thus potentially making a difference to learning in schools.  For there is no doubt that unless we get this piece right, any illusions of benefiting from a "demographic dividend" in the future are unlikely to be realised.

Personally, I would also like to see greater dissemination of the results of the report, not just at the time of its release, but continuously through the year.  Pratham and ASER Centre have of course, been disseminating the results at district and State levels all these years, but what we need in this country is a continuous and sustained debate about the education of our children.  Data from ASER is used regularly by the media to illustrate their reports; perhaps the next question to ask could be around ways to deepen this engagement in order to keep a discussion going.

Whatever direction the report takes in the coming years, ASER can rightly claim the credit for having changed, over the course of the last decade, the manner in which school education is discussed and understood in India; for that one achievement alone, Pratham deserves our thanks.

---

[1] Pratham Resource Centre, Mumbai, Annual Status of Education Report (ASER) 2005 – Rural, 2006.

[2] http://bit.ly/1xCn3RO accessed 31.12.2014

[3] Planning Commission, Government of India, Twelfth Five Year Plan (2012-2017) – Faster, More Inclusive and Sustainable Growth, Vol III, 2012.