Challenged UIA data collection methodology
Historical Misrepresentation of a “Union of International Associations”? (Part #7)
[Parts: First | Prev | Next | Last | All] [Gallery] [Links: To-K | From-K | From-Kx | Refs ]
Profiling process: Contributors to the study are very explicit in calling into question the manner in which the UIA gathers information from the bodies it profiles. The focus of the criticism is on self-reporting, irrespective of how the information is then reviewed and amended in the light of other sources, whether or not those amendments are approved by the body at some later date. The core of the complaint lies in the failure to verify the accuracy of the information finally presented. No mention is made of how scholars verify the veracity of survey questionnaires on which they may otherwise depend -- especially the challenge of census data. Other sciences are explicit in their use of statistical confidence levels.
The study fails to accord significance to two valuable aspects of self-reporting, irrespective of any subsequent editorial enhancement of such profiling:
- it defines organizations as they wish to be perceived in terms of the image they wish to project -- their "story" -- in contrast to the desire of the social sciences to "cut through" such illusion to the reality for those disciplines to which they choose to attribute meaning
- it respects the methodology of any lawyer in presenting a client in language that can be appreciated as meaningful -- termed "own words". This approach has been fundamental to the editorial treatment of the problems and strategies that an (international) constituency perceives and may claim to address
Missing from this critique is how any relevant information is to be located in a situation in which bodies may have every reason to be less than enthusiastic about being profiled, or may only wish to be minimally profiled, or may indeed choose to misrepresent themselves in some way. These are issues common to any social science survey, including census data. Only a degree of verification is typically possible, whatever the resources available -- including information from third parties. Missing information may however be considered confidential -- including the manner in which the body is funded. The approach of the UIA was to consider every possible source available, but not to engage in verification research -- preferring instead to improve the profile over successive editions if possible. Warnings in that regard were explicitly indicated. Professional social science studies could clearly choose to estimate a percentage of errors or confidence level -- as may be done with census data.
The criticism articulated by the contributors could therefore be considered as unrealistic, if not naive. Survey data is only adequate to a degree. in the real world, it is neither as perfect as scholars might wish it to be, nor as reliable. It is somewhat ironic to note that when scholars initiate their own surveys, they are typically obliged to note a "response rate". The contributors say nothing about how survey data should be rendered as perfect as they would wish -- and validated by some unquestionable authority.
In questioning the validity of the Yearbook data, one contributor argues that:
More recently, the Yearbook has presented, for the first time, an evaluation of its data collection process: the 2014-5 edition includes a (discreet) mention that the annual questionnaire to INGOs has had a response rate of 35 to 40 per cent - as such, it hardly amounts to an annual comprehensive census. (p. 194)
This comment seemingly obscures other aspects of the Yearbook profiling process through which profiles are progressively enhanced over the years. It also avoids any mention of how scholars launching their own surveys enhance the quality of the results or derive reliable conclusions from relatively low response rates. To be clear, options by which information has been derived (separately or in combination over successive periods) include:
- empty questionnaires (as an opportunity presented to new organizations, and as a checklist of types of information sought)
- partially filled questionnaires (in light of information already received, inviting completion/correction)
- computer proofs of previous profile (inviting completion/correction)
- requests for any document containing profile information (especially if more convenient than the above)
- use of details from website of the organization
- indications in documents of other organizations (notably those with which the body collaborates, or of which it is a member)
- indications in other directories (notably those of intergovernmental organizations)
- indications from announcements of international conferences (wherever they may appear)
- indications in historical reviews (as and when they become available)
Another approach to the matter is through a form of open source profiling. The UIA has experimented to a degree with this. It suffers from the same inadequacy as being a form of self-reporting. Where a profile can be amended by any authorized informant, difficulties may well be compounded. The editorial wars of Wikipedia have been extensively documented in this regard. The fate of the Encyclopedia Britannica, with its authoritative profiles, has also been noted by comparison.
UN approval of organization criteria? The contributors endeavour to stress the manner in which the criteria for inclusion have been subject to an "arrangement with the United Nations". In 1950 the UIA retook responsibility for registering international organizations in its Yearbook of International Organizations. Its capacity in this respect was acknowledged in a special UN/ECOSOC Resolution 334B (XI) of 20 July 1950, and subsequently figured in successive Annual Reports of the UN Secretary General. The UIA has had consultative relationship with UN/ECOSOC since 1951 for that reason. On this basis the study asserts:
- This entailed constraints that have shaped the content of the Yearbook until today: the definition of criteria for inclusion in consultation with ECOSOC officials; the obligation to include in the Yearbook, regardless of other criteria, every INGO that received consultative status with the UN (a heavily politicized process during the Cold War); and the coverage of both IGOs and INGOs alongside one another. (p. 178)
- The Yearbook's content was thus defined by specific criteria, established in accordance with UN staff -- likely in the mid-1950s. (p. 190)
Such assertions are mistaken in confusing that early resolution of ECOSOC, specifically approving the publication of the Yearbook, with a non-existent formal arrangement as to what should be included. The only implication in the latter regard was a requirement to profile NGOs in consultative status with ECOSOC, and by further implication, in consultative status with other UN agencies -- on the assumption that these had been formally perceived to be of significance to the process of international organization. This has always been assiduously done -- notably for the benefit of scholars of international relations -- even benefitting from profiles provided to those agencies by the NGOs (as has been formally required by those agencies). There have been periods in which the constrained resources of those agencies had resulted in the reverse process, namely provision by the UIA of NGO profiles to them.
Is the study endeavouring to make a case that this introduced a form of bias in contrast to an unspecified approach to profiling organizations of some other kind -- a process in which the UIA was heavily engaged over decades? Is there any evidence of scholarly suggestions as to other categories of organization that could be feasibly profiled or distinguished from those actually profiled?
The assumption that Yearbook profiling was controlled in some way by the UN is usefully challenged in the light of those NGOs accepted into consultative status as a consequence of political lobbying even though (for the editors) they were essentially national organizations and merited appropriate treatment as such. This notably applied to "diaspora" organizations from particular Asian countries. This illustrates how the political bias of the UN was respected -- as of significance internationally -- but was not allowed to bias the editorial treatment of such exceptions, duly categorized otherwise.
The UIA has always been free to consider categories of international body, including intergovernmental agencies, which were not in any way a feature of this consultative status process. Of particular interest in this regard has been the profiling of intergovernmental "agencies" of the United Nations and their subunits -- for whom establishing any such profile of themselves could well be considered politically delicate and heavily constrained by bureaucratic processes. This did not prevent the UIA maintaining such profiles to the extent possible.
Of particular relevance to this argument is a conversation with regard to criteria with Alexander Szalai, at that time Director of Research of the UN Institute for Training and Research (UNITAR). He suggested to this reviewer that the UN might indeed formulate such criteria to constrain the range of bodies in the Yearbook. My response was that the UIA would be happy to give it due consideration, provided the UN would in some way formally approve such criteria. The matter went no further.
The value of the Yearbook to the UN system can however be recognized otherwise with respect to the process of approving NGOs formally for consultative status. A preliminary step in this process for the NGO was to ensure that it was appropriately profiled in the Yearbook -- where the entry might be subject to preliminary perusal by the responsible officials.
"Genuinely international organizations"? Contributors have also focused misleadingly on the potential bias in the efforts of editors to distinguish bodies to be understood as having the quality of being "genuinely international". Reference is made to "seven rules" on which other scholars are alleged to have commented critically:
These criteria were only made explicit in the 1968-9 edition: seven major characteristics decided whether non-governmental organizations were 'genuinely international' and could be included in the Yearbook. (p. 190)
It is curious that the study should focus on this reference dating from 1968, possibly written by this reviewer (What Kind of Organizations Are Included, Yearbook of International Organizations 1968- 1969, p. 11-12), when the types of organization included were constantly under review, and extended in response to recognition of entities which were of potential interest as "international organization" in some way. That checklist, discussed below, indeed predates the extensive commentary on the further challenges detected (Types of International Organization: classification categories, 1978; Types of International Organization: detailed overview, 1978).
As indeed noted by the study (p. 190), this extension was organized such as to maintain the coherence of the types having the greater claim to be called international, whilst adding other types which lent themselves to challenge in this respect -- notably by politically sensitive intergovernmental organizations and scholars of international relations. As noted by the study, however:
In almost every article I have found where Yearbook data has been used, authors have conflated the number of existing INGOs with the content of the Yearbook, and even more narrowly with the Yearbook's category of 'genuinely international bodies'. This suggests how far the categorization of the Yearbook's, and its content, have been framing the perimeter of most studies. Only a handful of the identified 172 articles which have used figures or lists from the Yearbook include any acknowledgement of the 'seven aspects' used for inclusion and exclusion as 'genuine international bodies' and the consequences of such criteria for the research parameters, questions and conclusions. While it seems likely that most scholars are aware of the Yearbook's limitations, they did not articulate their concerns: their choice to work with what they had seems to have muted the critical appraisal of their main source. (p. 190)
The appropriateness of this comment obscures the manner in which it is effectively a case of "shooting the messenger", namely blaming the UIA for the failure of scholars in their consideration of international organization. That failure recalls the distinction made in taxonomical studies of biological species, caricatured as that between the "lumpers" and the "splitters". The contribution reinforces the criticism by citing the apparent bias in the Yearbook regarding human rights and religious bodies in that the activities of national bodies were considered far less significant internationally than those identified as international. The bias is obvious in the case of the numerous "church planting" initiatives emanating from the USA. It is surely for scholars to distinguish these appropriately from those which endeavour to coordinate their strategies internationally.
The issue can be clarified by indicating the categories of organization distinguished, however extended between editions in the face of the emergence of new types of entity -- or the perception by the editors of the relevance of entities previously omitted. Examples include:
- meeting series (Type S) which may imply an evanescent, ad hoc, coordinating body for each event
- multilateral treaties (Type T) which may imply a form of international organization not requiring a conventional secretariat (other than a repository)
- religious orders (Type R) which may have a degree of dependence on the Vatican, which challenges notions of international organization
This followed from early assessment of the variety of such bodies meriting consideration (Types of International Organization: classification categories, 1978; Types of International Organization: detailed overview, 1978). Both documents, variously reproduced in the Yearbook, discuss the variety of bodies with which the editors have been confronted -- in the real world. The solution was to group organization by category, as noted in the latter (Types of organization in the Yearbook), briefly described as follows (of which a version is reproduced in the study from the 1988 Yearbook as Table 11.1 (p. 235).
| Types of organization in the Yearbook of International Organizations (abbreviated summary) | ||
| Conventional international organizations | Type A | International organizations whether governmental or non-governmental, which group together at least three other autonomous non-regional international bodies as full members. |
| Type B | Non-profit international organizations, whether governmental or non-governmental, that have a widespread, geographically-balanced membership, management and policy-control. | |
| Type C | Non-profit organizations, whether governmental or non-governmental, whose membership and preoccupations exceed that of a particular continental region, although not to a degree justifying its inclusion in Type B | |
| Type D | Non-profit organizations, whether governmental or non-governmental, whose membership or preoccupations are restricted to a particular continent or subcontinental region. | |
| Other "international organizations" | Type E | Any international non-profit bodies, whether governmental or non-governmental, which may be considered an "emanation" of a particular organization, place, person or proprietary product |
| Type F | International organizations, whether governmental or non-governmental, whose formal characteristics raise fundamental questions if they are allocated to any of the preceding types. | |
| Type G | National organizations with various forms of international activity or concern such as research, peace, development or relief. It may also include national bodies which have relations with international organizations and which are listed by them in conjunction with truly international bodies or which appear from their titles to be international themselves. | |
| Special types | Type H | International non-profit organizations, whether governmental or non-governmental, which have been dissolved, are currently inactive, or are otherwise dormant |
| Type M | Developed to include multinational enterprises, whether governmental or non-governmental (subsequently incorporated in Type F). | |
| Type N | Bodies known to hold meetings with extensive international participation. The type is also used for national bodies which have names that create the impression they should be in any of the preceding types. | |
| Type R | Religious, military and fraternal orders or congregations, together with similar bodies based on charismatic leadership or commitment to a set of religious practices | |
| Type S | Continuing conference series | |
| Type T | Multilateral treaties, conventions, pacts, protocols or covenants signed by 3 or more parties. | |
| Index-only types | Type J | Used to indicate names of apparently international organizations whose creation has recently been reported but for which no further information has been obtained |
| Type K | Used for names of units concerned with substantive matters within the selected complex international agencies | |
| Type U | Used for names of apparently international organizations whose existence has not been confirmed as well as for inactive bodies which would have appeared in Type E or F. | |
Over many editions of the Yearbook, these distinctions have been used to separate out those bodies on which greater emphasis should be placed. The statistics generated followed that pattern. Whether or not the legalistic distinction is appropriate is naturally a question -- as with the distinction of profit from non-profit (especially when profit can be explored in non-financial terms). When is a cartel a non-profit organization? Is the acquisition of merit a nonprofit objective? Scholars have been less than helpful in clarifying and reformulating such distinctions over the years. Scholars have naturally been free to "lump" categories together or to benefit from the UIA "splitting" of them into types.
With respect to any assumption regarding "seven rules", it could be that the Types A through G have been seen as establishing such distinctions. The failure to conform to the "rule" defining any type could then result in its allocation to a "less international" type -- hence the challenge to distinguishing "genuinely international organizati0ons".
An analogous distinction of types has been used in the response to information on world problems and global strategies, as profiled in the Encyclopedia of World Problems and Human Potential.
Types of organization vs "Seven rules" of inclusion: Given the importance seemingly attached by scholars to "seven rules" articulated in 1968-1969, in contrast to the set of Types articulated in 1978, clarification is appropriate. Firstly it should be noted that the label "seven rules" does not feature in the document. With respect to the seven items there, in each case their "exclusion" (as indicated), relates primarily to exclusion from Types A through D. Bodies which did not correspond to those criteria would typically have been considered for Types F through U. The seven headings in that document are:
| Aims Members Structure | Officers Finance | Relations with other organizations Activities |
Presumptions of facticity: The detection and profiling of entities in the Yearbook and other UIA publications has always been an exercise in the art of the possible under circumstances in which the collection of that information could be variously challenged. The constraints were typically clarified in cautionary explanations introducing the profiles (as noted by the study). Various contributors call into question the value of UIA data without indicating how the information could be gathered otherwise. By comparison, reference could be made to the maze of cautionary footnotes in the various statistical studies produced by United Nations agencies on the basis of data supplied by member states.
It is too readily assumed that the UIA data purported to provide completely verified answers to the questions of scholars. It would be more appropriate to assume that the data raises questions which should call for critical inquiry by researchers (which might be assumed to be one purpose of the study). Through much of the development of the Yearbook effort was made to introduce previously unforeseen categories of relevance to comprehension of international association, most notably unusual entities whose relationship to the conventional categories merits commentary. Rather than challenging the credibility of data, ideally the evolution of the Yearbook could be said to have been designed to challenge the credibility of any given research framework and the oversimplistic categories favoured.
Some contributors would appear to have sought ways to use the efforts of the UIA to gather data to discredit that process as a consequence of the dissatisfaction of some scholars. No effort has apparently been made to indicate how data gathering could be improved and verified to a higher degree. This is particularly unrealistic and naive in the emerging context (Zen of Facticity: Bull, Ox or Otherwise? Herding facts and their alternatives in a post-truth-era, 2017). Especially striking with respect to criticism of Yearbook data by scholars claiming to establish more rigorous data sets (pp. 192-193), has been the complete lack of feedback from such scholars over decades regarding errors they had detected in the data.
Presumptions regarding criteria: Where do the contributors draw attention to the criteria which ought to have been applied by the UIA and to the methods which ought to have been applied in data gathering and verification? How can any such perspective be reconciled with the categories favoured or deprecated academically at different periods ?
How might such questions be compared with widely known recognition of the manner in which biologists have carefully reported on biodiversity along the Stuart Highway in Northern Australia -- generalizing their results as applicable to the desert environment far from that lonely road? Can international relations be considered similarly myopic in its focus on "big game" -- "carnivores" (IGOs) and "herbivores" (NGOs) -- to the exclusion of the multitude of smaller entities constituting the "biomass" of global civil society as profiled to a degree by the UIA? Is IR to be explored as "roadside research"?
For example, as cited in an early discussion of the variety of international bodies and their inclusion in the Yearbook of International organizations (Types of International Organization: detailed overview, 1978), Michael Wallace and J. David Singer were quite explicit about exclusion of NGOs from their analysis:
...our interests (and, we suspect, those of most of our colleagues) are more concerned with IGOs than with non-governmental organizations... as an independent variable, one can hardly urge that the amount of NGOs is likely to be important in accounting for many of the theoretically interesting phenomena, which occurred in the system of the past century or so. (Inter-governmental organization in the global system, 1815-1964; a quantitative description. International Organization, 24, 2, Spring 1970, p. 239-287).
Responsible as Singer was for the Correlates of War project (cited in the study), and depending on the meaning of NGOs, this conclusion could be seen as referring to all revolutionary and liberation movements, religious organizations, terrorist organizations, and secret societies, as exemplified by the history of Freemasonry (Joachim Berger, European Freemasonries, 1850--1935: networks and transnational movements, European History Online, 2012).
Is it any surprise that academics have had difficulty in engaging effectively with the challenge of religious conflict, civil unrest and terrorism? What of the nebulous groups facilitated by social media, held to have been so instrumental in the so-called Arab Spring uprisings, in the Occupy Movement -- and currently in the Yellow Vests movement? Such conceptual gerrymandering presumably renders any "correlates of war" irrelevant to the Global War on Terrorism, and many other "virtual wars" (Review of the Range of Virtual Wars: strategic comparison with the global war against terrorism, 2005). Potentially even more relevant are the elusive networks which are the preoccupation of conspiracy theorists, such as cartels, crime rings, and those which enable global financial crises (Diana Crane, Invisible Colleges: diffusion of knowledge in scientific communities, 1972).
Especially challenging to the academic bias with regard to international organization has been the increasingly explicit recognition of "evil" by world leaders, notably framing the Global War on Terrorism as the requisite response to an Axis of Evil. It is unclear how "evil" is related to any correlates of war from a scholarly perspective. The recognition is somewhat consistent with recognition of "wicked problems" by the policy sciences. The challenge featured in consideration of a role for the UIA Encyclopedia (Encyclopedia of Evil Claims, Claimants, Counter-claims, and Sigils: proposed facility in support of current global strategic priorities, 2016). This could be seen as complementary to the "angelic" role with which international organization may naively claimed to be associated (Engaging with Hyperreality through Demonique and Angelique? 2016).
Further insight with respect to the potential scholastic bias suggested by Hamlet's admonishment of Horatio's "epistemic dogmatism" (as cited above), is offered by John Ryder:
Ryder's approach to pragmatic naturalism is not an unspecified appeal to methodological plurality. On the contrary, he shows that different disciplinary approaches to any subject-matter produce "virtuous circularities". According to Ryder, these circularities have the merit of allowing for a sound critique of many of the habitual, academic prejudices. Many times these academic prejudices unnaturally force students to choose between apparently opposing teams. (The Things on Heaven and Earth, Fordham University Press, 2013).
[Parts: First | Prev | Next | Last | All] [Gallery] [Links: To-K | From-K | From-Kx | Refs ]