A STUDY OF THE METHODOLOGICAL PROBLEMS IN THE MEASUREMENT OF DEVELOPMENT

By Garry Jacobs

 May 24, 1982

• I. Introduction

In order to illustrate some of the common methodological problems in social science research, we will examine a topic which is presently engaging the minds of social scientists around the world, the problem of trying to measure the level and rate of development with particular reference to third world countries.

India has spent nearly Rs.1,00,000 crores on development programmes since Independence and is planning to spend an equal amount during the Sixth Plan. Yet until now there is no satisfactory means by which the impact of the past expenditure can be evaluated or by which future developmental achievement can be monitored.

It may not be apparent why this topic should present any particular problems to the scientist, but on closer examination it will become evident that it is fraught with methodological questions of great importance.

• II. Defining the Object of Study

The most widely accepted measure of development is the per capital gross national product (GNP) which measures the total value of all goods and services exchanged by a country in per capita terms. The utility of GNP as a measure of development remained unquestioned for decades, much like the utility of the guard post at Buckingham Palace which stood next to a bench in one of the entry halls for about half a century until one curious visitor asked why it was there and an enquiry revealed that a guard had once been posted there when the bench was painted to prevent visitors from sitting on it and he was never removed even after the bench had dried until the visitor enquired 50 years later.

When per capita GNP increases we naturally assume that the total welfare of the population has increased to some extent. But as a matter of fact there may be a sharp rise in per capita GNP without any benefit to the population or even during times of great hardship, e.g. war or epidemic disease. And when per capita GNP decreases we naturally assume that the total welfare of the population has decreased as well. But this is not always the case. For instance, the enormous strides in public health and medical care in India since Independence have resulted in a near doubling of the average life span. 80 million people living today, a population the size of U.P. would not be alive were it not for India's great developmental progress in this sphere. But since it is mainly those from the lowest income group, which is least productive, who have been saved, the result is a strong downward pull on per capita GNP and other per capita statistics. For instance, though total production of food grains increased from 55 million tons to 130 million tons between 1951 and 1980, per capita availability showed only a marginal improvement.

The conceptual difficulty we are now considering is really a problem of definition. Economists have never clearly defined what they mean by development. They have failed to make a clear distinction between growth and development. Growth is a process of expanding and multiplying activity at a particular level. The industrialized nations of Europe grew very rapidly after the Second World War by a proliferation and intensification of industry based on a pre-existent physical, social and industrial infrastructure. In contrast, development is a process of moving from lower to higher levels of activity and involves an enormous expenditure of time, energy, and capital to create an infrastructure capable of supporting activity at the new level. The progress of countries like India after the war has been mainly in the development of infrastructure facilities like roads, schools, hospitals, banks, power plants, etc. In other words the recent progress of the industrialized and developing nations differs in type as well as degree.

Now per capita GNP is an instrument originally designed in the West for measuring relative changes in the quantum of economic activity, i.e. growth. It was never intended and is not designed to measure changes in the level of activity, i.e. development. But for the last 30 years developing countries have been relying on this measure to assess their progress and in most cases have been sorely disappointed by the results it produced.

This error has arisen because scientists have failed to emphasize the theoretical and practical differences between the concepts of growth and development. The error arises from an inordinate faith in the power of a methodology and an insufficient effort at the level of thought.

• III. Objective vs. Subjective Research Topics

A recent trend in the social sciences has been to regard many social phenomenon strictly as subjective entities and to ignore or deny the possibility of making objective judgments about them. Perhaps this trend is a reflection in science of the greater social value given to individual preference and free choice. But when political values impinge on scientific method, the result is likely to be anything but scientific.

The sociologists have taken the lead in subjective research by concentrating on the individual's perceptions of a phenomenon rather than trying to construct an external objective standard. This approach is extremely attractive because it seems to eliminate the possibility of the experimenter's bias influencing the results, though it really does not. More importantly, it seems to eliminate the need for creating a clear theoretical framework in thought which an objective standard necessitates.

Of course research which studies the perceptions and values of the subject is not only valid but also very important. The error comes only when this approach is taken as an easy alternative to objective evaluation, and the subjective measure is actually confused with or mistaken for the objective one.

One of the current approaches to measurement of development, particularly among sociologists, is to regard it essentially as a subjective entity which depends on the aspirations and values of the population. Some sociologists have gone so far as to assert that any attempt to evolve objective standards for development is a dangerous imposition of the scientist's own bias on the population he is studying.

There is no question that people's conception of development may differ widely and this topic is a valid one for study by sociologists. It is quite another thing to insist that there can be no objective criterion for measure of development, whether that development is physical, social, political, economic or psychological. But to evolve objective criteria which avoid the more obvious pitfalls of subjective bias is certainly a more difficult task requiring serious thought.

In common conception development means more and better food, clothing, housing, health, education, transportation, communication, recreation, etc. It is certainly possible to measure all of these variables without any serious problem of subjectivity, so long as we take into account cultural and geographical factors. For instance, though food habits vary widely from country to country, there are certain common factors which distinguish the diet of the less developed from that of the more developed consumer, e.g. caloric value, quantity of protein, percentage of cereals, percentage of fats, and quantity of sugar. These factors can be used to construct an objective scale of development in relation to food.

• IV. Measures vs. Indicators

Another area of frequent confusion is the difference between measures and indicators of a variable. It is essential that the distinction between them is kept clear in designing research and evaluating results.

A measurement is a direct and precise process of evaluating a phenomenon by detailed study and examination according to some fixed scale of values. An indicator is an indirect and approximate means of assessment by observation or measurement of changes in some one or group of related things which are thought to truly reflect changes in the phenomenon under study. From the behaviour of an indicator we infer the status of the thing rather than directly measuring it.

This distinction may appear quite obvious, but it is often neglected in practise. Measures and indicators both involve measurements of variables. The real difference is that the variables involved in a measurement are directly and causally related to the object to be measured, whereas in an indicator the variables are correlated to the object, but not necessarily causally related.

For example, pollsters in the USA have discovered a tiny community in the mid-West which has voted for the winning candidate in every election since 1800. The voter preference of this community is highly correlated to the total voter preference of the country as a whole, but certainly the relationship is not causal. This community can be used as an indicator for voter preferences of the nation, but not as a measure.

For decades it was believed the per capital GNP is a direct measure of development and therefore its validity remained unquestioned like the painted bench at Buckingham Palace. No it is recognized that GNP is only an indicator of development, and a very poor one at that when used to assess progress within a developing nation over the span of a few decades. On the other hand GNP is a direct measure of economic activity and economic growth, though its accuracy and utility is now being seriously questioned. For instance, the environmentalists raise the objection that GNP does not properly reflect the depletion of natural resources. The more oil and minerals are mined, the higher the GNP; even if in the process a country's total resources are rapidly exhausted.

The chief criterion for a good indicator is a high correlation with the variable it seeks to monitor. Some of the best indicators may not be good measures and vice versa. For example, many nutritionists would argue that increased sugar consumption is harmful to health and should not be included as a measure of improved nutrition. But studies have shown that sugar consumption tends to increase almost directly in proportion to caloric value, protein content, and other measures of improved nutrition. Therefore sugar may be an excellent indicator, though a poor measure.

The confusion between measures and indicators goes back to the original problem of definition, of thought. Unless research is firmly founded in clear precise thought, errors of this type can enter from all sides.

• V. Assignment of Weights

One of the most difficult problems faced by the social scientist is in assigning weights to his measurements, so that they accurately reflect real life. This problem can only be solved by clear thinking and sound conception. But here too methodology, particularly statistics, is being employed as a substitute for the real thing with disastrous results.

The assignment of weights enters into every aspect of modern life. Take, for instance, evaluation of student performance in the university. Weights are assigned to each class on the basis of the number of hours of lectures and laboratory work, which may have little relation to the importance or difficulty of a particular class. The weights assigned to various questions in an examination often depend on the length of answer or the amount of information memorized, rather than the thought content of the answer.

The weighted models are always approximate and somewhat arbitrary. The only way to minimize or eliminate this arbitrary element is to constantly keep in view the reality which the model or measure seeks to represent.

Statisticians have evolved techniques for minimizing many types of statistical error, but these techniques can never eliminate the basic problems of weighting. In the recent national conference on development indicators, one scientist presented a model for measuring development by assigning weights to each variable according to its statistical variance. In this manner the error arising from comparison of variables with different variance was eliminated. Such an approach may indeed be useful, but many at the conference mistakenly concluded that by this method the problem of weighting had been solved.

Ultimately the best solution is to ensure that our models are based on real life and not on abstractions or isolated experimental environments.