Comptel recently released a regulatory scorecard that ranks "the relative effectiveness of the regulatory frameworks for electronic communications across countries." Quantifying regulatory governance and rules is potentially very useful in terms of providing data that allows us to test the effectiveness of particular regulations and regulatory approaches. The rankings, however, are highly problematic and uninformative for policy purposes.
The main problem is that Comptel implicitly assumes that certain regulations are good when, in fact, their effects are at best highly debatable.
Consider, for example, how the scorecard handles unbundling regulations. A country gets high marks on the scorecard for having extensive unbundling regulations. Most empirical research finds that unbundling was a failure in the U.S. (and in a recent paper I found that extensive unbundling was not good for broadband penetration across OECD countries). Yet the U.S. could improve its Comptel ranking by imposing strict unbundling regulations despite the evidence that such regulations would likely harm facilities-based competition. Achieving a higher score by imposing strict unbundling regulations is clearly not supported by the evidence.
The scorecard in general assumes that regulations are always good, and countries get higher scores for having more regulations. Some regulations have net benefits and some do not. Simply assuming that regulations that favor CLECs are good is not helpful to policy.
The scorecard has other, more subtle, flaws as well.
Consider its treatment of mobile regulation. The U.S. regulates mobile services more lightly than do European countries. The U.S., therefore, is not scored on several measures (such as "is fixed to mobile termination subject to regulation"). The scorecard corrects for this numerically in that it does not give the U.S. zeros for those categories, but does not correct it conceptually.
In particular, the scorecard does not allow for the possibility that perhaps competition benefits from the lack of regulation. The lack of mobile regulations distorts the scores and thus the rankings. For example, the U.S. could improve its scorecard ranking by regulating these areas of mobile and getting high scores according to the ECTA methodology. (Think of taking a bunch of tests. If you take 4 tests and get scores of 85%, 75%, 90%, and 80%, your average would be 82.5%. Now suppose you take a fifth test and score a perfect 100%. Your new average would be 86%. Adding mobile regulation to the list of regulations measured and doing well by Comptel's standards would similarly increase the score).
Finally, the scorecard ranks each element of regulation equally. While I do not know what the proper weighting should be, it is not at all obvious that, for example, "the portion of mobile numbers ported in 2005" is equally as important as whether the regulator has "clear rules for [a] price squeeze test."
Collecting these data can help lead to more intelligent policymaking by providing the tools for good evaluation of regulation. Indeed, with several colleagues I compiled an extensive database of regulations across countries that has since been used by a number of researchers to evaluate the effects of particular regulations.
The data have to be analyzed carefully, however. Implicitly imposing certain assumptions and creating an arbitrary ranking is not helpful to good policymaking.
As (a nerdier version of the irrepressibly funny) Stephen Colbert might say, a tip of the hat for the data collection and a wag of the finger for misuse of those data.