Engaging with Difference

Methodology in Evaluation, Sue Stubbs, September 1996

Discussions on evaluation methodoloy are often limited to a discussion of 'method' - the tools to be used for data-collection. But methodology is much more than methods, it is about human behaviour, attitudes, principles, beliefs about knowledge, power relationships, and ultimately it depends on our own deep-seated beliefs about why we are involved in an evaluation; what is our ultimate goal? Where we want to end up will inevitably influence how we decide to get there.

Community Based Rehabilitation (CBR) projects are not just about technology they are about the whole of people's lives, and so making some sort of judgements about a project really means that people are engaged in making judgements about life; other people's and often their own. People have their own individual, cultural and traditional ways of 'giving value to' (evaluating) their lives which are often very unconscious and deep-seated. In CBR, as with most 'development' projects, this process of giving value is inevitably cross-cultural - even if foreigners are not physically present, then their philosophies and concepts undoubtedly are. The whole doner-led insistence on evaluation and evaluation reports, reflects a literacy-based Western 'scientific' approach to life which is often in direct conflict with some of the oral-based, community-orientated societies where religion, ritual and the arts are more important than science or technology.

So what are the implications of all this for evaluation methodology? I can best illustrate this by sharing with you my own experience of taking part in a small piece of research which was really an 'evaluation' of two schools in an integrated education project in Lesotho. I will describe this process in terms of a journey of exploration.

My initial attitude was a typical one of Western-style exploration; I was going to a foreign land to seek hidden treasures (results of the evaluation), to dig them up (using evaluation tools) and to bring them home and display them (in a report). It was difficult to plan for the journey - in fact 'preparation' is a more accurate term, because it was not possible to say in advance what these 'treasures' would be and I therefore did not know what tools (methods) would be needed. So I took a range of tools, but felt most hopeful about the tools which had been designed from the experience of similar journeys, and which offered a lot of flexibility. These 'tools' were the range of methods and approaches developed in 'Participator Rural Appraisal' (PRA) (see note 1).

The other important aspect of preparation was the preparation of myself for this 'journey of exploration' - I lacked knowledge and experience of the culture I was to visit, there would be language and cultural differences, and all the participants in the 'exploration' would have different agendas; some would be afraid of digging up unpleasant objects, some would have already decided what they were looking for, some may not be interested in digging at all, and others may want to dig just to cause as much disruption as possible! So I had to develop an attitude of mind which was open and flexible, and to be prepared to 'embrace failure' and to learn from mistakes and to be open to unexpected findings of hidden treasure. This is in many ways the opposite of our traditional understanding of a 'professional' attitude which means that we are confident in our knowledge and experience, and are ashamed of our mistakes and 'failure'.

If I had been undertaking this journey alone, and had kept myself distant from others, maybe my initial belief that I could dig up and bring back some sort of objective 'treasures' would have been maintained. However, one of the crucial methodological foundations was that the exploration was to be conducted in a team. This team consisted of people of different gender, culture, experience, ability and disability. Once the journey began, very soon it was the journey itself, not the idea of some treasure which became the focus.

I met my fellow travelling companions, and the whole activity of learning how to relate to them, how to 'engage with their difference' became my major exploration. I realised that I myself was my major 'tool' - my own skills (and limitations) in relating to people were key; in being honest without being offensive, in trying to understand their perspectives and expectations without totally compromising my own. As I came up against one cultural/individual difference after another, this inevitably made me look more rigorously at my own assumptions, bias, beliefs, expectations. The outer journey became an inner journey.

Because the subject of our enquiry was the lives of people (as discussed above), very soon it also became clear that this exploration was not a simple matter of deciding what we were looking for, where to look and how to unearth the treasures. Treasures began to be revealed for us by local people themselves, through a basis of human trust and in an atmosphere of support. We became vacillators in other people's exploration of their own situation, their own lives; we had no right to go and dig up their lives and to extract knowledge for ourselves. Our role was to learn, listen and help people see their own situation in a broader context, to uncover their own treasure and give value to them. Our own perceptions were not 'objectives', they just gave people a broader range of bias with which to compare their own perspectives. The 'tools' which were most effective for this task were ones which could be handed over completely to local people and which could be adapted and developed by people themselves. For example, giving a small group of teachers the task was to become ever more rigorous in my own self-criticism and in becoming aware of my own bias and assumptions. I found myself making judgements in favour of facilitating other people's inquiry into their own work, sometimes at the expense of my own understanding.

For example, when animated discussions were happening in Basotho, the local language, I often decided not to ask for a translation - I realised that my 'curiosity' and belief that I have an absolute right to knowledge could be unethical if it interrupted people's own critical enquiry into their own situations.

This leads me back to my original question; 'what is our ultimate goal?' I now believe that our goal in evaluation is to facilitate a process whereby people are supported to examine their own situation and lives in more depth, with rigour and in a systematic way. External experience can help people gain a broader perspective, but it can never be used as a blue-print. If this development of critical thinking in the 'stake-holders' of CBR, (disabled people, children, parents, community members) is not a key goal, then I suggest it is a waste of time. The role of an outsider or evaluation facilitator then becomes that of a catalyst, a supporter, a listener, a validator, a mirror, a disseminator rather than an accumulator of knowledge.

My conclusion is that it is therefore dangerous and misleading to focus all energy on 'indicators' and 'methods' rather than on the process of collaborative enquiry within which people devise their own indicators and decide on the most appropriate methods. I also believe that this approach is not less rigorous than those which try to get some 'objective' knowledge from evaluations. In fact it requires a greater self-discipline and rigour, and where Western donors and implementers require written 'objective' reports, then there is definitely a challenge to find ways to express this methodology with authority and clarity. Using personal testimony (diagrams, drawings and stories from the 'authors' of the experience) is an excellent way to promote this authority.

All of the above is true for any development project, but with disability projects, the issue of 'engaging with difference' is particularly pertinent.

How do we support a young person with Down's syndrome, or help a deaf adult to develop the skills and confidence to be able to 'give value' to their own lives and become more conscious of this? This is the challenge of evaluation methodology.

Note 1
See the following for more information on PRA: Chambers R 1992) "Rural Appraisal: Rapid, Relaxed and Participatory" Discussion Paper 311, Birghton: Institute of Development Studies.

Stubbs S (1995) "The Lesotho Integrated Education Programmes: A Case Study on Implementation" MEd Research Thesis, London: Save the Children Fund.

Theis J and Grady HM (1991) "Participatory Rural Appraisal for Community Development" London: IIED.

Sue Stubbs


Save the Children logo


Title: Engaging with Difference: Methodology in evaluation
Author: Stubbs, S
Publisher: Save the Children
Date: 1996