How Does the World Justice Project Create Its Rule of Law Index?

The rule of law isn’t just relevant to lawyers and judges—it affects people’s rights, access to justice, and safety in their neighborhoods; it affects whether governments are open and accountable; and it affects whether corruption is allowed to take root, and whether people can thrive.  

The WJP Rule of Law Index, published annually, gives insights into people’s experiences with and perceptions of rule of law in 140 countries and jurisdictions around the world.  

You can learn more about the methodology used to create the Index below.  

How does the WJP define the rule of law?  

The World Justice Project defines the rule of law as a durable system of laws, institutions, norms, and community commitment that delivers accountability, just laws, open government, and accessible justice. These general principles are further developed into eight factors and 44 subfactors in the WJP Rule of Law Index.  

The eight factors are: Constraints on Government Powers (Factor 1), Absence of Corruption (Factor 2), Open Government (Factor 3), Fundamental Rights (Factor 4), Order and Security (Factor 5), Regulatory Enforcement (Factor 6), Civil Justice (Factor 7), and Criminal Justice (Factor 8).   

How does the Index account for different definitions of rule of law and different types of governance? How does it account for cross-cultural differences in rule of law?  

The Index and our surveys have been carefully designed to be applied in countries with vastly different social, cultural, economic, and political systems. No society has ever attained—let alone sustained —a perfect realization of the rule of law. Every country faces the perpetual challenge of building and renewing the structures, institutions, and norms that can support and sustain a rule of law culture.  

In addition, the Index accounts for different definitions of rule of law and different types of governance by weighing all factors equally. No single factor is given more weight or importance than another. 

Are the WJP Rule of Law Index scores valid, statistically sound, and reliable?  

The Index has been subject to a rigorous process to ensure that the scores are valid. In developing the survey instruments, WJP paid special attention to factors such as question content, consistency, cross-country comparability, order, wording, and questionnaire length, and pre-tested and pilot tested multiple questions in several countries for two years. In developing the indicators, WJP uses a vast number of questions to tap different dimensions of a given subfactor and includes the assessments of experts in different disciplines and the public. This year, the WJP Rule of Law Index used more than 5500 question-level variables to compute scores. These are based on more than 150,000 valid responses from households and over 3,600 from legal practitioners.  

Does the WJP validate its Index scores?  

The WJP cross-checks Index scores against independent quantitative and qualitative sources such as Freedom House’s Freedom in the World report, the Global Competitiveness Index, Transparency International, Economic Freedom of the World report, World Economic Forum, the United States Human Rights Report, Freedom House’s Nation in Transit, Amnesty International country reports, and many others. The goal of this exercise is to check for consistencies and inconsistencies within WJP’s data and to provide a qualitative layer of analysis useful in interpreting scores and enhancing in-house files. In addition, the WJP collaborates with the Econometrics and Applied Statistics Unit of the European Commission’s Joint Research Centre to perform sensitivity analysis to assess the statistical reliability of the results. 

What are the limitations of the WJP Rule of Law Index?  

The WJP Rule of Law Index is the most comprehensive data set on rule of law; nonetheless, it has limitations. For example, the Index does not provide a full diagnosis or dictate concrete priorities for action. It is intended to be used in combination with other analytical tools, both quantitative and qualitative, to provide a full picture of causes and possible solutions. 

What data sources are used to calculate the WJP Rule of Law Index scores and rankings?  

The WJP Rule of Law Index scores and rankings primarily utilizes two sources of original, independent data: the Qualified Respondents’ Questionnaire (QRQ) and the General Population Poll. The overall QRQ results account for 65.32% of the scores and the overall GPP results account for 25.83% of the scores (weighted distribution). Third party sources, or TPS, account for 8.85% of the scores.  

The QRQ consists of closed-ended questions completed by in-country practitioners and academics with expertise in four subjects: civil and commercial law; constitutional law, civil liberties, and criminal law; labor law; and public health. The QRQ is administered by the WJP research team on an annual basis in every country or jurisdiction measured in our Index. These questionnaires gather timely input on a range of topics from practitioners who frequently interact with state institutions. Such topics include information on the efficacy of courts, the strength of regulatory enforcement, and the reliability of accountability mechanisms.  

The GPP is a household survey administered to the general public in each country and jurisdiction included in the Index. It is now generally administered every two to three years by leading local polling companies to a representative sample of 1,000 respondents in each country or jurisdiction. The poll has historically been conducted in the three largest cities in each country or jurisdiction, but the WJP is transitioning to nationally representative polls. The GPP questionnaire includes 127 perception-based questions and 213 experience-based questions, along with sociodemographic information on all respondents. 

In addition, the WJP incorporates a few third-party sources in the Index scores. These TPS cover topics that cannot be adequately measured through our surveys, such as the number of deaths due to terrorist incidents or civil conflict in a country or jurisdiction. These TPS account for 14 of the variables included in the Index.  

What is considered a “statistically significant” change in a factor?  

The measure of change over time is constructed in three steps. First, last year’s scores are subtracted from this year’s to obtain, for each country and each factor, the annual difference in scores. To test whether the annual changes are statistically significant, a bootstrapping procedure is used to estimate standard errors. To calculate these errors, 100 samples of respondent-level observations (of equal size to the original sample) are randomly selected with replacement for each country from the pooled set of respondents for last year and this year. These samples are used to produce a set of 100 country-level scores for each factor and each country, which are utilized to calculate the final standard errors. These errors—which measure the uncertainty associated with picking a particular sample of respondents—are then employed to conduct pairwise t-tests for each country and each factor. Finally, to illustrate the annual change, a measure of change over time is produced based on the value of the annual difference and its statistical significance (at the 10% level). 

Between the GPP, QRQ, and the third-party source data, the WJP collects and analyzes data for more than 550 question-level variables that are used to build the WJP Rule of Law Index. 

The GPP and QRQ cover sensitive topics that people may not want to discuss. How do you address this?  

Some of the questions that we ask in our surveys are sensitive in nature. Our local polling partners use a variety of interviewing techniques to ensure that respondents are comfortable answering these questions. In some cases, a respondent may not be comfortable answering a certain question. All respondents have the option of not answering or saying that they do not know the answer to a question.  

In very rare cases, it may not be possible to ask a certain question in a country or jurisdiction. If that is the case, the question may be slightly modified or removed from the questionnaire. 

 

If you want to learn more about the methodology used to create the Index, you can visit our page here. You can also visit our Variable Map, which contains all the questions that go into the Index and how they’re organized.   

Don’t miss the global launch of the 2022 WJP Rule of Law Index on Wednesday, October 26 at 10am EDT. There’s still time to register to attend our live, virtual event.