Credibility and interference in official statistics: opposites at war

What we should know about the context

In this paper, a verbal presentation of which was delivered at the celebrations organized by INEGI on the occasion of the World Statistics Day, I consider the special reasons that make credibility such an important virtue. Additionally, I examine the major types of political interference and proceed to review what in the armoury of legal devices exists that could strengthen government statistical offices so as to make them less vulnerable to political attack. I find that internal efforts even when supported internationally may be insufficient. As a result, I advocate a multinational or regional initiative and argue that it is in the national interest to back such an initiative whatever the apparent short term gains of consenting to undue interference.

What triggered my proposal to deal with the matter of interference were two recent and independent episodes in which the inviolability that statistical agencies had customarily taken for granted was tampered with. But as anyone who has experience working with such agencies, it is inviolability that they require if they are to retain the virtues of objectivity and impartiality. In one case, that of Canada, the Government broke a longstanding tradition of leaving the choice of technical methods in the hands of the head of the government statistical agency (referred to below as the Chief Statistician). The rules of the quinquennial Census of Population were changed on the grounds of respect for individual privacy from compulsory response subject to sanction to voluntary against the best professional advice. In the other case, the Government of Argentina frustrated by the statisticians’ continued refusal to disclose individual information to the government officials in charge of fighting inflation, took over the production of the Consumer Price Index (CPI) in a vain attempt to improve control over the rise of internal prices.

Throughout the piece and particularly in the second of the two incidents featured in the previous paragraph, no agency outside the country found a way of expressing disappointment and much less outright censure of what was a manifest violation of the freedom of choice statistical agencies had enjoyed traditionally. There was discussion of the matter at a special session of the 42^nd session of the UN Statistical Commission.¹ The International Monetary Fund (IMF) criticized the same actions without making their retraction a condition for future talks. The United Nations Statistics Division expressed no views and continued to invite a delegate of Argentina to attend sessions of its Statistical Commission. A few individual protests were heard in the months that followed the Canadian events and sharp critical words condemned the actions of the Government of Argentina in the pages of The Economist.² No complaints was brought to the attention of the public on the grounds that citizens in Argentina and to a lesser extent in Canada were being deprived of their right to sound, objective and impartial information.

These two incidents are examples of governments taking action of an unprecedented nature and changing the relations they previously maintained with their respective statistical agencies. That change is incompatible with improvements, be they in the relations of trust between government and public servants or in the quality of the statistics that finally get to see the light of day or else on the predisposition by the statistical office to take initiatives designed to improve the quality of general information placed at the disposal of the public.

The break with the past is all the more surprising as it comes in the wake of a number of steps in the opposite direction taken by international organizations. Thus, the UN took the decision to draft, seek approval for, and finally promulgate the ten Fundamental Principles of Official Statistics;³ and the IMF egged on by the perceived consequences of the balance of payments crisis in Mexico and the perceived inability of the statistics to show glimmers of the impending crisis on the horizon, developed what would turn out to be the SDDS and the GDDS.⁴ Notwithstanding the importance of the steps taken respectively by the United Nations and the IMF, the fact remains that interventions in the way member countries manage their official statistics, even those that spill over and affect the statistical information provided by their neighbours, remain few, and weak.

The UN complemented the creation of the Fundamental Principles with a revision of the Handbook of Statistical Organization⁵ but the scope of the revision did not go beyond the law and the institutional scope of a centralized statistical office. It said anothing about how to act internally or externally in the face of outright violation of provisions embedded in national statistical legislation or in the UN’s Fundamental Principles. The question that remains is if the Fundamental Principles and the Handbook were written today, would they retain the same reticence in the face of identified threats to the fundamental integrity of statistical information? The answer is far from clear.

The question of credibility

The question of credibility is the central question of statistics because there is no practical way of finding out whether what the statistical agency publishes is correct or not.

The definition of credibility which is most effective is to maximize users’ interest in what the data mean and minimize their propensity to discuss how the data were estimated. I owe this definition to Jack Triplett (in speech rather than in writing) who applied it to Statistics Canada. In his story, someone was asked how he knew that such-and-such was the case and the reply was “…because Statistics Canada said so”. The answer was regarded as a discussion-clincher by both parties. More importantly, one of the strongest bonds that keeps the Canadian federation together consists of Federal-Provincial transfers –from the richest provinces to the poorest so as to guarantee a minimum standard to all Canadians. Political arbitrariness is prevented largely by the application of a complicated statistical formula the components of which are certified by the Chief Statistician of Canada. The fact that the system stays unchallenged is proof of the agency’s credibility in the most delicate of matters.

A very special circumstance affects official statistics. There is no practical way of verifying if statistics are right or wrong. It stands to reason that a Census cannot be replicated. Smaller household or business inquiries do not lend themselves to replication either and in any case no one other than the statistical agency is legally entitled to conduct the inquiries that lead to official statistics. At best one might track the Consumer Price Index, but even then the sheer weight of numbers would make it virtually impossible for another agency to attempt to replicate the official results. These reasons lead to the conclusion that we either believe what the official agency tells us or we do not but proper verification lies beyond anyone’s capabilities.

If we believe official statistics this is because we also believe that the statistical agency has the required complement of competent staff members who do the right things, in the right sequence, and at the right time. Observation may tell us whether the public when acting as respondent to a statistical inquiry, tends to do so truthfully whether out of civic duty or out of fear of sanction. But that is seldom more than an impression. We may also believe that the staff of a statistical agency would not fudge what figures it gives out to measure the quality of its results out of moral integrity that we assume them to have. It is therefore no more than a belief that leads us to trust what the agency says about sampling variance, non-response rate, rate of imputation and so on.

Credibility is earned over time. But it can be lost “in one afternoon” to paraphrase what was said about the British navy. Credibility can be lost for purely internal reasons or else through external intervention. The two should not be addressed in the same manner.

Because we cannot verify the statements that come out of a statistical agency, be they about facts or about quality, we can only opt for being vigilant and see if the data published are inconsistent or outright contradictory. Thus the organizing role of the system of national accounts is of unique importance to the critical user anxious to see that it lies at the heart of official economic statistics. Moreover, the national accounting framework has the unique advantage of relying on independent individual statistics to reach the same total in two or three distinct ways. No doubt, this feature is among the most important for confidence building.

Without going into details, the Census of Population provides a role which while far from identical to that of the national accounts has some similarities visa-vis social statistics. It supports household surveys by providing them with sampling weights. Statistics on employment derived from household surveys and after being weighted with factors taken from the Census can be confronted with production statistics with which they are roughly correlated; and production statistics are part of the core of the national accounts. Any flagrant contradiction of these relationships raises a question mark about the professional credibility of the statistical agency.

But losses of credibility through technical incompetence may be small and typically are local. An error detected by a blatant contradiction involving employment and output of widgets is not necessarily a wholesale condemnation of the statistical agency responsible for it. At most such an error may suggest that there is an atmosphere of neglect within the official agency that stands in the way of thorough checking and quality control before statistics meet the public eye.

But a moral error could have more devastating consequences. Suppose that there were an internal memorandum leaked to the press and showing very clearly that quality attributes such as response rates are routinely manufactured and bear no relationship to the real count. Surely such a finding could bring down the entire statistical edifice for the simple reason that if such a practice were entertained in one case, as far as the public is concerned, it could be entertained in all instances.

Whether caused through loss of moral integrity or because of careless or slipshod quality control, errors that are detected by third parties undermine the credibility of the statistical agency not only for those statistics that were found wanting but just about for everything else the agency produces. There is no merit in trying to convince the world outside the statistical agency that the error found in one part of the agency was somehow sealed off from the rest of the agency’s activities.

Normally these matters are addressed through a series of short and longer term measures. For example, if the lack of quality control is systemic, it is literally the head of the agency that is on the chopping block. Where the offense is local, the rolling of a few minor heads may be sufficient “pour encourager les autres”. Among the longer term measures a call on outside experts to vet the tightness of the procedures and additional training may be sufficient to prevent agency credibility from being further eroded. These measures are usually as important internally to avoid repetition of what caused the uproar in the first place as to reassure the users that the matter is taken seriously and is being dealt with. Perception is as important as the underlying reality.

What should we know about interference?

A more insidious loss of credibility results from perceived interference with the statistical agency’s programme, methods, announcements, and at times but more seldom, the agency’s results. The interference –perceived or real– is from government and its motives can be guessed. The numbers that fall immediately under suspicion are those for inflation (much too low); the rate of unemployment (equally low); the rate of real growth (much too high); the number of construction completions (too high); the number of bankruptcies (too low) and so on. Moreover, personal anecdotes are brought up as refutations of whatever the agency announces in fields perceived to be critical for the government and once these anecdotes become public because the press believes there is substance to them or nowadays because they enter the blogosphere and may be dressed up to sound like universal truths, the interference whatever its scale can have very negative consequences.

There is no generally agreed taxonomy to classify the various types of interference from an outside body. Usually the interfering agent is the Minister Responsible although there are instances of third parties that have sought to interfere. For example, in a crisis the Government may create a body with extraordinary powers –an intelligence agency, a Court in charge of special matters, a Board with special powers to fix prices, control or regulate production and so on. Any of these bodies is likely to collide with the confidentiality provisions of the statistical legislation and in a parliamentary system it would take Parliament or the Supreme Court to break the legal impasse. But leaving aside emergency situations there are at least three types of attempted (not always successful) interference by the Minister Responsible which can be found in just about every country with a long enough history of official statistics.

The “muzzle” is an attempt by the political arm of Government to prevent or to delay from publication a statistic that is ready to be published. The nature of the official motivation for the attempt varies widely. It could range from the perception that publication would inflict a blow on a genuine national interest to a trivial concern for the Ministry or the party in power. Either way it would present a challenge to the head of the statistical agency, one which would compel him to weigh the inevitable loss of credibility against the harm that the published number would be likely to cause. The following two examples illustrate the point. The government is committed to a policy of stimulus for employment but the latest number on the rate of unemployment shows a dramatic rise. The government is aware of the possibility and puts pressure on the statistical office to qualify the number as seasonal and dismiss its importance in any assessment of longer term trends in the labour market. It may threaten the Chief Statistician with budgetary retaliation should he fail to comply. In the second example, the Government has adhered to a UN policy of trade sanctions imposed on a rogue regime but traders are continuing to violate the agreed policy. The Government requests the statistical agency to take the Customs records and reclassify the boycotted destination to “Other countries”.

In the case of “handcuffs”, the Government tries to prevent a regular survey from taking place or else tries to prevent a question in the said survey from being asked. In this instance the Government after allowing a commitment to be made to the public, attempts to get the statistical agency to go back on the commitment without stating that it has bowed to government pressure. The following example illustrates the point. The party in power was elected on the basis of a promise to improve the lot of the rural population by providing it with better access to public services. But it has failed to act on its pre-electoral promises. The Opposition is aware of this and clamours for the inclusion of a question on access to public services in one of the statistical agency’s regular surveys. The statistical agency is ready to comply but the Government “handcuffs it” before it has a chance to alter the regular questionnaire.

Lastly, there is the physical takeover of the statistical agency of which there is a live example in the Latin American region. In this instance, Government officials occupy the physical space of the statistical agency and either force the regular employees to comply with new rules even if those rules are in defiance the national legislation as well as of the Fundamental Principles or else to be replaced by more flexible recruits ready to produce figures that are “made to measure”. In any of these instances the loss of credibility is sudden and general. Usually the targeted figures are those that measure the rate of inflation and unemployment as a percentage of the labour force. But once it is established that any one of those statistics is fabricated, all else becomes suspicious in the public’s eye. In such circumstances, if there is a genuine will to regain credibility the measures that are necessary apply to legislation, institutional design, and lines of accountability in addition to what internal measures are also adopted.

The state of the defence mechanisms

Usually, the legislation is much too weak on the following three key issues:

Let us take these points one by one and examine what is wanted as compared to the situation on the ground. Chief Statisticians in many instances are appointed at pleasure and serve so long as the Minister who appointed them is in charge. The “Minister” is that member of the executive who holds political responsibility for the statistical agency. In some cases “Minister” is the head of the executive –President or Prime Minister. Once the Minister is appointed, he (generic for he or she…) or his staff prepares a list of known sympathizers, people who hold the Minister’s confidence, and among them select one to take over the statistical establishment. There is no evidence that in the many countries in which this procedure has been adopted traditionally it produced consistently high quality management, stability and continuity. Those Governments that are more keenly aware of the dangers inherent in the traditional system replaced it by a mandate system, one in which the director’s contract is for a fixed period but can be renewed.

The required qualifications for a new Chief Statistician are a permanent issue of contention. Some Ministers believe that a moderate familiarity with quantitative methods is what matters and is bound to be sufficient but others require a more formal acquaintance with statistical techniques and would look at the academic world as the source of choice. There are few cases where the requirements include evidence of high level managerial skills combined with thorough familiarity with any one of economics, demography, sociology or marketing. Lastly, in a wide majority of countries including some members of the OECD, there is no tradition of an organized search for suitable candidates based on objective requirements and a professional judgement about personal suitability. Rather, selection is made by word of mouth references which are sometimes effective but on average less than adequate.

The existence, composition, mandate and operating rules for a Statistical Council are just as vague and inconsistent as the search for a Chief Statistician although there are many examples of Acts that include the formation, composition, and operation of a Council. There are several areas of concern which warrant examination individually. Firstly, there is the Council’s composition. The majority of Councils discussed in the legislation have a fixed membership selected by making sure that each Ministry that operates a statistical programme be represented in addition to representations from cross- Government institutions. Needless to say that the more specific the enumeration of Council members the less likely its chances of meeting regularly and discussing a jointly agreed agenda or for that matter agreeing on any agenda other than at a very general level. There is seldom any provision for enforcing Council’s regular meetings; no provision for delegation (typically when Councils meet on the few occasions they do, the original delegate has handed down the responsibility to several levels below his own and as a result the participants’ role is no better than that of a tape recorder). The Council has no specific objectives such as ensuring coordination of inquiries or use of the same version of the standard classifications and it is not the Council that discusses or deliberates over the national response to international initiatives. And yet an active Council presided over by someone with generally acknowledged prestige can create a very useful space between the Chief Statistician and the Minister above all when there are sharp divisions of opinion. It can also restrain the Minister from acting too impulsively whenever an opportunity is found to replace a Chief Statistician whose performance is judged to be sub-par. But for the Council to act in the most effective manner, its members must be selected in a way such that they can converge (not the case if each member is given narrowly defined terms of reference designed to ensure that in any division of resources each of the agencies represented receives some proportional share irrespective of needs and priorities). Meetings must be regular. The President of Council must have access to the Minister. The Council must act as a watchdog in all matters regarding the efficiency and equity of the Chief Statistician’s management and finally the Council must be morally and technically unassailable so that it escapes from becoming an object of political criticism particularly in the eyes of the press.

Before leaving the subject of the Council’s mandate, a word about coordination and integration is in order. The existence of a Council is particularly desirable where there are various centres within the Government that run statistical activities. Such initiatives are likely to be most effective where there are rules and standards and the comparison among statistics is not inhibited by barriers such as incompatible classifications, inconsistent delineations of economic units, and wide differences in the standards applied to sample survey design. The function of the Council, in addition to promoting a consistent programme free of duplications and reasonably free of glaring omissions is to ensure that all statistical activities are performed under the same conceptual umbrella and to earn the necessary respect and prestige to rule in favour of a coordinated approach and have the ruling accepted.

The last of the subjects on the list of Council functions is the creation of a space between Minister and Chief Statistician –to all intents and purposes a space between employer and employee– it being clear that their respective goals do not coincide. In fact, the professional code of conduct of the statistician may be fundamentally opposed to the behaviour expected by the politician. In the event, it is best for both and of course best in the interests of institutional credibility if their relations are mediated by a body that both perceive to be impartial, both respect, and everybody considers to be legitimate.

And finally there is the matter of what are proper relations between Minister and professional head of the statistical agency. The Council is not and cannot be a standing body. At best it may meet twice a year. It can designate a sub-committee consisting of its enthusiasts to meet somewhat more often but that is as far as it goes. This leaves the matter of day-to-day business on which the Minister must be consulted and conversely on which the Minister may require an opinion from the Chief Statistician. No outside mediator or facilitator for such meetings exists and it would be foolish to try and appoint one.

There are a number of devices that have been used with variable success in countries with a highly developed statistical system. There may be a member of the Minister’s personal staff who is appointed as liaison officer between the two hierarchies. There may be an official appointed by the Chief Statistician to liaise with his counterpart. But usually the volume of business does not justify such appointments and should problems arise involving simultaneously the two agencies, it is proper for the principals to get involved at an early stage. Another device tried in Canada but discontinued after a few years and a few Cabinet changes was for the Prime Minister to instruct his Minister Responsible to adopt an arm’s length policy vis-a-vis the statistical agency. But a serious risk implied by this scheme is for business to end up in the Prime Minister’s office whether its nature justifies it or not. And because Prime Ministers and Presidents are usually very busy, the problem would have to go away, remain unresolved, or else be solved by officials who are neither politicians nor statisticians but just happen to be around.

There is no easy solution to the problems that stem from a fundamental difference of priorities and understanding between politicians and professionals trained in a discipline that by its intrinsic nature commits them to impartiality. One can only hope that circumstances do not create a problem that both parties view as acute and fundamental but prefer radically different solutions to get rid of it. One practical idea for a standing Chief Statistician on welcoming a new Minister is to make matters very plain insofar as boundaries of responsibility are concerned. Making matters plain comprises necessarily a full briefing on what is proper for the Chief Statistician to do, what he cannot do even if asked, and what would force him to submit his resignation. Such a briefing would touch upon disclosure of individual performance or identity (no); advance submission of key releases asking the Minister for approval of the language used (no); and providing a full and detailed account of what is behind each of the key aggregates short of breaking confidentiality rules (yes). Of course, if both parties are new to their job and know each other things are easier in that it will be natural for them to work out rules of harmonious co-existence. Either way, an initial defining conversation involving the understanding that each has of his mandate is pretty much essential.

There are more fundamental ways of protecting statistical work from thoughtless interference. One of these ways which will warrant study and ongoing friendly observation is that pioneered in Mexico as part of the overhaul of its statistical legislation. It consists in giving a great deal of administrative independence to the statistical agency and to define it as autonomous in the legislation while recognizing that it is still under the purview of the Treasury for its operating budget. But at the same time the new legislation allows the agency to own what buildings, machinery and transport equipment are required to discharge its mandate and to manage its budget including the fraction of earned income which it is allowed to keep. It still remains to be tested whether in situations where there are many producers of statistics, including official statistics, the solution of becoming quasi-autonomous is compatible with playing an unchallenged role in coordinating statistics produced by others, accessing administrative registers, and influencing their design, and imposing internationally agreed standards on all producers of statistics located in the public sector.⁶

The role of international agencies

The story behind the drafting and promulgation of the United Nations’ Fundamental Principles must be told and taken into account in the design of defence mechanisms. Those countries that were in the orbit of the now extinct Soviet Union did not have a solid tradition of autonomy in the choice of scope and methods, in the preservation of confidentiality, and in keeping the rest of government at bay when publishing results of highly sensitive inquiries. And yet, they were sufficiently convinced of the merits of the Western traditions in the way official statistics were dealt with to wish them implemented in their own countries. In this respect the Fundamental Principles were a boon in that Chief Statisticians could approach their political masters and point to the Principles that were issued by the United Nations and applied to all countries irrespective of their history or traditions. Similar efforts could take place in a collective attempt to reduce attempts by Government to interfere or to reduce the probability of successful interventions. Naturally there is no fool-proof way of countering Government attempts because in the end Governments have the means to force the issue. Legislative counter-measures by their very nature are too slow and ponderous to work effectively. By the time all parties comply with the rules of evidence and with the deliberations of the legislators or judges concerned, the crisis has blown away. What remains though are the Government’s punitive budgetary measures designed to remind a daring Chief Statistician of where real power lies.

The United Nations family of agencies wields an extensive range of publications the dissemination of which is of great importance because they are the authoritative source of international comparisons. The legitimacy of these comparisons derives from the application of the same standards and the same conceptual frameworks. The inclusion of a statistic feared to be contaminated by malpractice in the comparative tables published by members of the UN family is a threat to all others. The creation of a regional or sub-regional total with contaminated statistics is to be feared particularly by those whose scrupulous application of international standards makes them subject to the greatest harm relative to their wishes and objectives. Accordingly, interested countries might adopt a proposal consisting of the following two stages:

Hopefully, the majority of situations would not require to be treated in this fashion. But the consequences of the process would go further than simple publication. For example, no country under quarantine could qualify for Paris 21⁷ review and endorsement. Presumably the IMF would require a normalization of the situation before undertaking any evaluation of the country’s trade and payments or the health of its internal economy. The World Bank or the regional development bank concerned would not entertain any loans until such time as it could review a certified set of statistics and so on. This differs from the current situation where resort is to the IMF without going through a more neutral process of advice, warning and evaluation.

Regional versus Global initiatives

There is no tradition in the affairs of the international statistical community of anything like what is proposed in the previous paragraphs and if the Statistics Commission were hit cold by a proposal along these lines the chances are that it would reject it as an unwanted example of undue and unwarranted interference in the domestic affairs of member countries. It is possible that a similar proposal might fare better if it were dealt with by regional bodies and only after having tried their persuasive powers should the proposal be tabled for review by the Secretariat of the Commission.

Either way, though, participants in the next meeting of the United Nations Statistics Commission may wish to sound out their colleagues. There are many participants who are justifiably concerned about the dangers of contagion. And there are just as many who fear the consequences of making it apparent to rogue Governments that they can act with impunity. Whereas it is entirely reasonable to act with prudence and at a measured pace it is neither reasonable nor prudent to tolerate the permanence of unacceptable institutional situations and to let them to fester the time it takes to inflict irreparable damage.

8 November 2010. London, Ontario

Bibliography

High Level Forum on Official Statistics, UN Statistical Commission 42nd

Session: Jean Louis Bodin: Fundamental Principles of Official Statistics: Threats and Responses.

Argentina’s Great Statistical Swindle: The Economist, 25 February 2012.

United Nations Statistical Commission, Papers of the Special Session, April 1994.

Statistics Division, United Nations: Handbook of Statistical Organization, Third Revision 2007.

¹ See High Level Forum on Official Statistics, UN Statistical Commission 42^nd Session and especially Jean Louis Bodin: Fundamental Principles of Official Statistics: Threats and Responses.

² See “Argentina’s Great Statistical Swindle” in The Economist, 25 February 2012.

³ See United Nations Statistical Commission, Special Session, April 1994.

⁴ The Special Data Dissemination System and the General Data Dissemination System are both IMF standards designed to ensure that statistics required for the international monitoring of national economies are not interfered with by undue Government intervention in the normal statistical process.

⁵ Statistics Division, United Nations: Handbook of Statistical Organization , Third Revision 2007.

⁶ Handbook of Statistical Organization, Third Revision: Chapter III The Statistical System.

⁷ bit.ly/4feVkkG