16
	Evaluation Based Planning for 
	Rehabilitation Programmes in India

	Maya Thomas and Maliakal Thomas

This chapter describes how the concepts of monitoring and evaluation in rehabilitation programmes were introduced into various non‑governmental organisations (NGOs) supported by the Disability Division of ACTIONAID‑India.   Most of the literature on evaluation deals with the methodology and outcomes of evaluation, rather than  the process preceding the evaluation. Here we attempt to delineate the process of attitude change that was brought about in the NGOs towards the concept of evaluation and describe how they  changed from a stage of resistance to one of acceptance and active involvement. 

What is Evaluation?

According to the World Health Organisation (WHO, 1981): 

	 "Evaluation is a systematic way of learning from experience and using the lessons learned to improve current activities and promote better planning by careful selection of alternatives for future action. This involves analysis of different phases of a programme, its relevance, its formulation, its efficiency and effectiveness, and its acceptance by all parties involved".

Another working definition proposed by Krefting (1994) states:

	"Evaluation is the systematic collection, analysis and interpretation of information about the activities and outcomes of CBR programmes in order for interested people to make judgements about what the programme is doing and how it can be improved." 

In many instances, the words monitoring and evaluation tend to be used together or interchangeably.  However we take monitoring to refer to a continuous mechanism of observation, recording and checking of programme implementation.  Evaluation, we view as a periodic exercise carried out during, or at the end of, a programme, to assess the worth and/or impact of a programme.  
	The data gathered for monitoring purposes are of value in an evaluation exercise.  Both monitoring and evaluation have to do with inputs, processes, outputs, outcomes, impact and effectiveness, though with differing emphasis.  Monitoring helps to follow‑up and watch over while evaluation helps to take stock of a programme.   

The Background

The field of rehabilitation is in an evolving stage in many developing countries, including India.  Over the last decade in India, many programmes and approaches are being promoted by the government and by the NGOs, to reach the large numbers of disabled people in the country.  The Community Based Rehabilitation (CBR) approach occupies an important position among the wide ranging activities.  
	There are two major reasons for the promotion of CBR. One is the availability of appropriate know‑how for the care of disabled people in the community, and the consequent shift from 'professionals' to 'non‑professionals' in community care programmes.  An important advantage of this shift towards decentralisation is the availability of services for the majority of the population within a reasonable period of time and in a form that is acceptable to them.  
	The second reason for the promotion of CBR is the national commitment for universal provision of services and the shift from 'everything for a few' to 'something for everyone', (Murthy, 1991).
	In 1990, the Disability Division of ACTIONAID‑India began to actively promote CBR projects through NGOs in  different parts of the country.  Currently the Division supports 15 CBR projects based in rural as well as urban slums.  These projects are managed by people from different backgrounds; some are disabled people, some are family members of disabled people, some are religious and charitable organisations, some are rehabilitation professionals and some are social workers.  Most of the people do not have professional qualifications in the field of rehabilitation.  
	The CBR programmes encompass various components including  prevention, medical rehabilitation, education, vocational training/income generation, social rehabilitation and awareness building (Thomas, 1993). 
	In the national commitment to, and in the promotion of CBR, it is  important to keep in mind that rehabilitation is culture dependent, especially if it is to take place in community settings. It will be difficult to have a centralised plan, particularly in a country as diverse as India.  A centralised national plan can only provide a framework.  There has to be specific decentralised planning for service provision in accordance with local cultures. 
	In order to ensure that such decentralised programmes are effective, especially if they are carried out by non‑professionals, it is necessary to equip the implementers of CBR with the skills of monitoring and evaluating their work.  The relevance of the CBR approach has to be evaluated in each area and community.  The only way to know that CBR benefits disabled people, their families and the wider communities is to continuously monitor and evaluate the different aspects of CBR programmes.  
	From the beginning, the Disability Division of ACTIONAID‑India recognised the fact that CBR was in an evolving stage and that it had to be evaluated in its different contexts in order to study its impact.  The Division started on the premise that monitoring and evaluation must be seen as integral parts of any programme.  We believed  that the projects supported by us, should develop the capability of integrating into their programmes procedures for monitoring and evaluation, and of using the results obtained  for their own  future planning.  Our goal was to upgrade the technical skills of project staff so that they could introduce their own  monitoring and evaluation into their programmes rather than have external evaluations mid‑way through the programme. 
	In the initial stages, the NGOs supported by ACTIONAID concentrated primarily on service provision. Their programme planning did not include monitoring and evaluation as integral components.  However, from ACTIONAID's point of view, the  concepts of monitoring and evaluation of programmes were important and we had to persuade the programmes of their worth.  It was not to be a centralised plan of monitoring and evaluation, but one that was done by the local programmes.  The aim was to have monitoring and evaluation as integral parts of the programmes.  Primarily to feed into the development and future planning of the programme and only secondarily for the purposes of the donor agency or other external bodies.  The stages of the process through which we attempted to achieve these outcomes, are detailed in the following sections.

Phase of Negotiation
A situation analysis was carried out as a first step at the different partner organisations in order to assess their opinions and impressions of the concept of evaluation.  Most NGOs were found to have resistance to evaluation for various reasons.  
	Many felt that they were too busy with programme implementation. Others voiced their ignorance and lack of skills to carry out evaluations.  Yet another group cited the lack of financial and technical resources.  Some resisted the idea because it came from a donor agency, fearing that adverse findings would affect future funding.  
	All this clearly pointed out the need for an attitude change towards evaluation and to promote its acceptance as important in programme planning and effectiveness.  It was important to convince the NGOs that:

	1. evaluation would address primarily  their need and not the needs of external agencies; 
	2. evaluation could lead to better and more cost‑effective methods of service delivery;
	3. the funds utilised for evaluation could be recovered by ensuring that failures were minimised;
	4. evaluation was not as difficult a process as they thought.

Phase of Passive Co-operation
One of the first stages in this process of bringing about attitude change was the starting of a newsletter to disseminate information on contemporary issues in rehabilitation. 
	ACTIONAID Disability News was started in 1990, as a biannual publication meant for concept clarification and highlighting of issues to be addressed in the field of rehabilitation.  The newsletter was aimed at implementing agencies, donor organisations, policy planners, administrators and professionals involved in rehabilitation. 
	The rationale for starting the newsletter was this: projects are more likely to get interested in trying to innovate in their own programmes when they knew where they were in relation to the global situation in rehabilitation, and what they should do to catch up with what was going on.  Once there are innovations in a programme, the implementers would also want to know how effective they were, thus generating an interest in evaluating  the approaches adopted. 
	In the first year, the initial issues of the newsletter attempted to provide a holistic picture of the field of rehabilitation and its various aspects, including methods of service delivery, manpower training, development of technical aids,  evaluation and research.  
	With the first few issues, there was a perceptible change in the NGOs, from active resistance against new ideas, to passive acceptance, though scepticism remained.  A readership survey which was conducted at that time showed that a majority of the respondents read more than 75% of the newsletter (see ACTIONAID Disability News, 1993).  The changes noticed in the NGOs supported by ACTIONAID included attempts at decentralised planning of programmes, more active promotion of community participation and examination of viable income generation activities as a contribution to the sustainability of programmes.

Phase of Active Co-operation
Evaluation and research in rehabilitation became the major focus of the information dissemination effort in 1991‑92. It was a pro‑ active and planned effort with definite goals, though the ideas were consolidated  through a democratic, bottom‑up and consensual method.  
	The ACTIONAID Workshops for project holders of rehabilitation programmes had evaluation issues as the main topics for discussion.  At these Workshops, key people from allied fields were chosen to sensitise the project holders to the need for documentation, monitoring, evaluation and research.  This was done in a collective, democratic set‑up with plenty of time for debate and clarifications.  Some of the project holders who had been introspecting and experimenting on these issues were also invited to share their preliminary experiences and to make recommendations for future action. 
	By 1992, a consensus had developed in the project holders regarding the need for evaluation and research in their programmes.  There was also consensus on the need for specialised technical inputs on these issues.  Around the same time, two Symposia were organised on evaluation and research  for a larger audience which included the project holders. This was an attempt to bring in experts in the field of rehabilitation, to share their experiences in  evaluation and research  and to identify critical issues for further research in the field of rehabilitation.  The topics included the experiences of research and evaluation of the Government of India, UNICEF, one of the Indian institutes on disability, the 3D projects in Jamaica and the Disability Division of ACTIONAID.  Arising from the discussions there was an agreement that research and evaluation were necessary to increase the efficacy of CBR programmes and that implementing agencies needed to be trained in operational research.  It was also agreed that research should be specific and relevant to the given sociocultural situation (Prabhu, 1993).
	The stage was thus set for spelling out ACTIONAID's strategies for documentation, evaluation and research in the supported projects (Pruthvish and Thomas, 1992). 
	Furthermore, the technical inputs requested by the projects were provided by bringing in a Programme Advisor specifically to co‑ordinate issues related to documentation, monitoring, evaluation and research in CBR projects.  Finances were also allocated for this purpose in the project budgets.  At this stage, the NGOs had moved from the stage of passive acceptance to active involvement in specific assignments related to research.  This was carried out in a collective manner, though some degree of scepticism continued to linger.  
	Throughout the stages in the first two years, the emphasis was on attitude change before initiating action. It was recognised that if evaluation was imposed on the projects without changing attitudes, they would become averse to the idea of evaluation.  Further, any future attempts to evaluate programmes and to use the results to improve their efficacy would also fail.

Phase of Initial Action
One of the first assignments undertaken jointly by the projects and ACTIONAID was the identification and needs assessment of disabled persons in all the CBR programmes.  The tools used to do this included questionnaires for 
	1) obtaining a community profile; 
	2) door-to-door survey for identification of disabled persons, and 
	3) needs assessment of identified disabled persons.  
These tools were first tested in a pilot study in one of the CBR projects.  After this, all the project holders met together and jointly field-tested the tools in another CBR project.  After modifications and translation into local languages, the tools were used by the project holders for data collection in their respective projects (Pruthvish and Thomas, 1993).  (Copies of the questionnaires are available from ACTIONAID India.)
	By early 1993, all the CBR projects had completed the identification and needs assessment surveys in their areas.  This exercise was neutral and did not expose any shortcomings, yet was useful to all projects in establishing baseline data in their respective target areas.  Besides, the project holders were actively involved in the methodology planning, data collection as well as the analysis and interpretation of findings, thereby improving their own technical competencies in this neutral area of research.  
	With this exercise, the scepticism slowly disappeared.  There was also some uniformity across projects in terms of the methodology adopted, which led to the possibility of compiling comparative data across different projects.  The possibility of publication of the survey reports also acted as a positive reinforcement.
	By this time, computerisation had been effected in most projects, improving their ability to document, store, analyse and retrieve data more easily.  Much of the information disseminated through the newsletter was also absorbed by the projects.  Slowly the need to learn more about evaluation and the need for more external expertise in this area began to come up from the projects.  Evaluation was accepted by then as necessary and important but the projects lacked the know‑how to carry it out in their programmes. 
	The need for some uniformity in the recording and information systems used by CBR programmes was also beginning to be voiced, in a tentative fashion.  However even at this stage, the NGOs were not ready to examine their failures or to let others do so, nor were they concerned about preventing others from repeating their mistakes.  It was therefore necessary to guide them towards this frame of mind, and to have them look at positive and negative aspects of programmes, in order to improve the planning and effectiveness of their own programmes and those of others.  

Phase of Examination and Acceptance of Corrections
An on‑going CBR project, located near Bangalore and supported by ACTIONAID, which was more open to evaluation was evaluated in late 1993 (Rajendra, Abraham and Thomas, 1993).  This project started in 1990 and covers 140 villages with a total population of 100,000.  In all, 825 persons with a disability receive help from the project, such as  vocational training, medical rehabilitation, education, social rehabilitation and prevention.
	A pilot methodology was evolved jointly by the project and ACTIONAID, to carry out the evaluation study on a small sample.  In this effort, scientific accuracy was sacrificed in the interests of a simple and easy to comprehend methodology.  The reliability may not have been high but the study was nevertheless valid because of the wealth of information that was generated and which helped to modify strategies for the future. 
	The major advantage of this study was that it helped to demystify the process of evaluation for the implementing NGO, convincing them that an evaluation was a relatively simple exercise which can be easily integrated into the on‑going project activities.  Besides, the results of the study helped the NGO to plan new strategies to overcome identified weaknesses and to prepare a better perspective plan for future programme implementation. 
	At this stage, the change in attitude to evaluation was from analysing neutral information to objectively analysing the positive and negative aspects of a programme.  Secondly, from mere analysis of data, there was a shift towards utilising the data to modify strategies and to plan for the future.  The summary report of the evaluation study was published in ACTIONAID Disability News for wider coverage and dissemination, so that others could also learn and be further motivated to evaluate their own programmes (Rajendra et al., 1993).

Phase of Collective Evaluation and Sharing Information
By this time, the implementing NGOs were ready to accept the need for evaluation and had largely overcome their resistance to the idea.  In addition, they were willing to objectively analyse the positive and negative outcomes of an evaluation, and to utilise the results for future planning and action.  
	The time was then right for the introduction of standardised, internationally accepted methodologies of evaluation, so that different projects could be evaluated and the results compared.  With this in mind, a Workshop on Evaluation Methodology for CBR projects was jointly organised by ACTIONAID and the UNDP Inter-regional Programme for Disabled People, Geneva.  Participants were drawn from ACTIONAID supported CBR programmes and from Government sponsored projects. 
	The Workshop attempted primarily to address the question of "How to Evaluate".  Its objectives were to finalise a methodology for evaluation and to help participants to plan for the evaluation of their programmes.  The "Operations Monitoring and Analysis of Results" (OMAR) software which has been developed by the UNDP for evaluation purposes was demonstrated at the  Workshop (Jonsson, 1994).  This was subsequently used in a modified form by two CBR projects although both found some of the computer software difficult to use.  
	The report of the pilot evaluation conducted earlier was also extensively discussed, particularly with regard to the process, the practical difficulties encountered and the outcomes.  At the conclusion of the workshop, the participants agreed that the process of evaluation had been further demystified and that they were enthusiastic about planning for evaluations of their own programmes.  

The Lessons Learnt
After five years, this is the stage we have reached.  There have been some major lessons learnt  in this entire exercise, as summarised below:

1. Any attempt at introducing monitoring and evaluation systems will be rejected if attitudinal changes have not been effected.

2. Improved availability of information to service providers can help in changing attitudes to evaluation.

3. Attitude change is brought about more easily if implementing agencies are able to see evaluation as of some direct benefit to themselves.

4. Though lack of time and money are often cited as reasons for resistance to evaluation, they may not be the major reasons.  A major cause of resistance at the project level is reluctance to expose failures.

5. Methods of monitoring and evaluation are not as simple as the    implementing agencies would like them to be, though  ACTIONAID had thought that it would be simple and easy to introduce them at the beginning of the exercise.

6. The technical upgrading of research skills in the programmes is often necessary before local personnel can take up evaluation on their own.

7. Some aspects of monitoring/evaluation are easy to introduce into programmes, while others are more difficult.  Quantitative indicators, such as prevalence rates and coverage of services are easily obtained but qualitative ones, such as changes in the quality of life and the efficacy of interventions, are more difficult. 

8. Measures of disability which are easily observable and quantifiable and are culture-free, can be easily introduced into a programme, e.g. questions relating to mobility: does the person stand, sit, walk with/without help?  Such measures can also contribute to the evolution of standardised evaluation methods.

9. However the measurement of handicaps, which is mostly culture dependent and not easily observable or quantifiable, are generally difficult to introduce into programmes, and they cannot be standardised across projects.  These include measurement of acceptance by the community and self esteem of the disabled person.  Resistance from projects will be greater if these measures are introduced in the initial stages.

10.  It may be preferable to have an independent and skilled research team to work on culture dependent and qualitative parameters of evaluation, using small samples drawn from the projects, rather than getting the projects to do this sort of work on their own.

Future Plans
In the next stage of our work, it is proposed to develop further the idea of evaluation and research, with the objective of improving programme effectiveness.  As part of this work, ACTIONAID and the International Center for the Advancement of Community Based Rehabilitation (ICACBR) of Queen's University, Canada, jointly organised an Asian Regional Symposium on Evaluation and Research in CBR.  This brought together the viewpoints, experiences and information on research and evaluation in CBR from different parts of the world.  
	The intention was to stimulate implementing agencies to move from the broad areas of evaluation of overall programmes, to take up intensive research  pertaining to specific programme components that will further improve programme effectiveness.   To achieve this, it is also proposed to bring in technical expertise,  both from within the country and from outside, to help local implementing NGOs upgrade their research skills. 
	It is also proposed to set up a CBR research team for the purpose of continually studying CBR in its different aspects and to utilise the results obtained to modify future programme strategies as well as to feed into national and international level policies and planning in rehabilitation.
	Through all these efforts, we expect the following outcomes.

	1) Implementing organisations would develop a positive attitude towards monitoring, evaluation and research, 
	2) They make monitoring, evaluation and research an integral part of their work without difficulty, 
	3) These organisations will move towards evaluation based planning and base their future programmes  on the results of their monitoring and evaluation efforts, and 
	4) They would share their experiences with others in the field and thus contribute to policies and planning at a more global level.

The entire exercise of introducing evaluation based planning in rehabilitation started in a small way, with emphasis on the whole process being carried out by the projects themselves. Special care was taken to ensure that  attitude change took place prior to action and to involve projects early in the process, rather than imposing ideas on them from outside. 
	The ACTIONAID India Disability Division helps other organisations interested in the evaluation of their programmes in India and elsewhere.  The support includes:

	 the supply of background materials and literature, 
	 arranging exposure visits to projects which have been evaluated or which are in the process of undertaking evaluations,
	 invitations to participate in workshops on evaluation and research, 
	 and in some instances, project staff can be involved in the process of evaluation of a project.

References
Actionaid Disability News. (1993). Supplement Volume 4, Number 2.

Jonsson, T. (1994). OMAR in Rehabilitation: A Guide to Operations Monitoring and Analysis of results. Geneva: UNDP Inter-regional Programme for Disabled People.

Krefting, L. (1994). Evaluation of CBR, cited in T. Jonsson (op.cit.).

Murthy, R.S. (1991). Community Based Rehabilitation: Is there a need for research? Actionaid Disability News, 2, 2.

Prabhu, G.G. (1993).  Symposium on Research and Evaluation in CBR: Summary Proceedings. Actionaid Disability News, 4, 2.

Pruthvish, S. & Thomas, M. (1992). Research in rehabilitation: Proposed strategy of the Disability Division. Actionaid Disability News, 3, 2.

Pruthvish, S. & Thomas, M. (1993). Identification and needs assessment of beneficiaries in Community Based Rehabilitation initiatives. Actionaid Disability News, 4, 1.

Rajendra, K.R., Abraham, C. &  Thomas, M. (1993). The Sourabha CBR Project: An Evaluation Study. Actionaid Disability News, 5, 1. 

Thomas, M. (1993).  Strategies for Rehabilitation in India.  Actionaid Disability News,  4, 1.

World Health Organisation (1981). Health Programme Evaluation - Guiding Principles. Geneva: W.H.O.

Maya Thomas is an executive of ACTIONAID India and holds the position of Director of the Disability Division; a post she has held since its inception in 1988.  She has a doctorate in clinical psychology and is an associate member of ICACBR, Queen's University, Canada and foreign corresponding member of the American Academy of Cerebral Palsy and Developmental Medicine.

Maliakal Thomas is a Senior Psychiatrist in Manipal Hospital, Bangalore, India.  He assists ACTIONAID as the editorial advisor for 'Disability News' and also in research and evaluation.  He is on the governing body of Community Health Cell, a unique Indian NGO which is involved in the development of community medicine. 

Contact Address
	ACTIONAID ‑ India, 
	P.B. No.5406, 
	No.3, Rest House Road, 
	Bangalore ‑ 560 001, India