LEARNING THE METHODS OF POLICY ANALYSIS AND PLANNING Methods courses are usually taught by supplying students a tool box of analytical techniques (e.g., forecasting methods and regression analysis) that can ostensibly be applied to policy problems. There are many drawbacks to this approach. It creates the sense that all that needs to be done is to discover the proper match between problem type and specific method and then crank out a solution. This is seldom the case. It also fails to acknowledge that the complex problems facing governments are usually multifaceted and require multiple. not single, analytical approaches. Problems examined this way usually lose their context and become mathematics problems rather than policy problems.
In reaction to this approach to learning methods, some have said that since each problem is unique, what is needed is a “proper frame of mind” to do analysis. In other words, there is no standard method that can be used to approach these problems; each time an approach must be created new. Our position lies between these two. We believe that a standard procedure exists for addressing these problems. We also believe that a number of fundamental or basic methods can be used within this procedure to analyze complex problems. We have brought these basic methods together in this book for the beginning policy analyst. We believe these basic methods cannot be effectively learned without a context. Thus in this book we describe the process of policy analysis and planning and explain a number of basic methods appropriate to each step in that process. Each method is illustrated with examples. The end of each process chapter con- tains practice problems. The second part of the book contains policy cases, which will take from five to 25 hours each to resolve. For each policy case, unlike the more directed exercises provided with the process chapters, readers are on their own in delineating the problem. choosing an approach, and adopting methods of analysis. This combination of traditional learning and learning by doing was selected with several goals in mind. We hope each user of this book will:
1. Learn to recognize situations in which specific basic methods can be applied quickly and appropriately.
2. Become competent at using methods of analysis and designing approaches to policy problems. 3. Learn how to communicate the results of analysis to appropriate decision makers.
In the process of attaining these goals, we hope our readers will also:
• Learn the language of policy planning. • Learn to write with organization, clarity and precision. • Larn to use supporting documentation (maps, charts, grapha) effectively. • Learn to speak publicly, delivering critical information concisely. • Learn to develop simple models that are supported theoretically and empirically. • Learn to evaluate the distributional aspects of policies and programs. • Learn to incorporate political factors into analysis. • Learn to advocate uncomfortable positions. • Learn to work under time constraints and to allocate analytic resources. • Develop management skills and the ability to work in, and/or direct a team. • Learn how to obtain policy-relevant data, through efficient search techniques and persistence. • Practice the sifting and synthesizing of mountains of seemingly irrelevant reports and memoranda. • Practice using secondary data sources. • Develop quantitative analytic skills. • Develop skills for qualitative analysis. • Learn to design effective program implementation guidelines. • Learn to cope with’uncertainty in a policy context by being flexible and tolerating false starts and
dead ends. • Learn to read and understand legislative language. • Learn how to develop a program of researched analysis for staff. • Learn to be skeptical of their own solutions as well as those of other analysts.
Some people might argue that these skills are best learned on the job. We contend that because a standardized process has emerged and because practitioners of policy analysis and planning have developed a number of basic methods, it is more efficient to learn and practice these methods before facing them in real life.
What role do other courses play in learning to do policy analysis? For those who expect to work on state and local government problems two other streams of learning, besides methods, are important. The first is the process of policy making and planning. How do governments work, laws get passed, and administrative rules get initiated? How do professionals within organizations function in providing analysis and advice to decision makers, be they elected or appointed officials? How can the process of analysis involve various constituencies and public and private groups? Courses in planning. government, administration, and management as well as some interdisciplinary programs are directed toward these concerns. The second essential element is a knowledge of the economic, geographic. and social structures of urban and regional systems. How do cities and regions grow and decline? Which of their problems might be addressed by governments? Courses in urban and regional economics, human ecology, sociology, geography. and interdisciplinary offerings address these questions. The analytical methods portion of a planning or policy analysis curriculum usually involves courses in descriptive and inferential statistics, use of the computer on larger data sets, and courses containing a potpourri of researched analysis techniques such as survey research methods, linear programming, cost-benefit and cost-effectiveness analysis, input- output techniques, modeling, and simulation. Some curricula offer courses containing more generic methods for policy analysis and planning, such as forecasting and prediction, alternatives generation, and techniques of program monitoring and evaluation. This text should be used early in the analyst’s career. After learning basic methods of policy analysis and planning, the analyst will be ready to move on to coursework in more advanced analytical techniques and techniques of researched analysis that require more time, resources, and usually larger information bases and data sets. PRACTICAL PRINCIPLES FOR BEGINNING POLICY ANALYSTS Learning how to approach policy problems and how to choose appropriate basic methods during the policy process takes time. There is no substitute for practice. Our first bit of advice, then, is to begin thinking like analysts and planners. As you read the daily news, reflect on the problems confronted by governments and ask how you would approach those problems, what information you would need, and what basic methods might be appropriate. Over time, as you develop the habit of looking at policy problems analytically, you will gain confidence in your ability to understand such problems. The following suggestions should help as you begin to undertal~e policy analysis. 1. Leans to Focus Quickly on the Central Decision Criterion (or Criteria) of the Problem: What factor of the problem is most important to your client? On what criterion is the decision likely to be made? Will it be minimizing the cost of some service? Or might it be to spend more effectively the funds now allocated to the activity? Perhaps it will be to broaden the base of those being served by the pro.. gram. On what basis can we judge the merits of alternative policies or programs? Identifying the central “nugget” of the problem is essential. In some cases the criterion can be inferred from legislative intent; in others you might have to exhume it from a mountain of seemingly patternless reading material. When working on exercises or case studies, you will have no real client from whom to extract “the nugget.” On the other hand, the client often has no idea what the central decision criterion is. The difficulty is that public policies often have multiple and competing objectives, and the objectives are often extraordinarily ill defined. Beginning analysts must learn to focus quickly, or valuable analytical time will be wasted.
If beginning analysts are to survive, it is essential that the sea of ambiguities—extensive reading material but none with priority, reams of undifferentiated data, the blank writing pad, the motionless pencil, and the
calculator in the “off” position—be recognized and overcome. Getting started is difficult, but focussing on the central decision criterion will help identify needed information. It is better to make a bad start than to make no start, You can and should recycle: after several hours of work does your central decision criterion still ring true? There are, of course, dangers in choosing the nugget prematurely. There is a tendency to choose the one that can be defined and measured most easily (e.g., least cost) while possibly ignoring more important but less quantifiable goals and impacts and forgetting about who pays and who benefits from the policy. There is also a distinct possibility that several competing and equally valid decision criteria exist, and that early focussing will dismiss the alternatives forever. 2. Avoid the Tool-box Approach to Analyzing Policy: Some disciplines specify analytical routines in detail for many circumstances. This may encourage some people to begin work on a policy problem because it lends itself to their favorite method. Ideally the problem should dictate the methods, not vice versa, This book is intended to help avoid the toot-box appioach. Because problems are complex, beginning analysts can be very apprehensive about which method to use, We advise using the simplest appropriate method, and using common sense to design a method if one doesn’t already exist. Combine methods if you must. Use more than one whenever possible. Apprehension often forces us back to the methods with which we feel most comfortable, but try to avoid this tendency. The principal tools of the policy planner are logic, common sense, and experience with particular substantive areas. It helps to be practiced in data analysis, rational problem solving, and other specific skills. But more often than not we design our own approach or methodology to policy problems. This kind of creativity becomes easier the more policy analysis we do and the more we learn what the clients, be they real or simulated (in exercises and cases), find understandable and useful to their deliberations. If the methods applied are not transparent, the client is forced to either accept or reject the results without under- standing. Doing the exercises and completing the cases in this book, seeing others’ approaches to the same problems, and understanding the deficiencies of one’s own methods are efficient ways to obtain experience in policy analysis. 3. Learn to Deal with Uncertainty: Neophyte analysts are tempted to isolate each parameter of a policy problem and then establish their most likely future values. Having tacked down the key parameters of the problem (because the task is never-ending, many spend most of their allocated time on this phase), they believe the problem can be solved. This approach is often a waste of time. Therefore we describe basic methods of decision analysis and sensitivity testing that can aid in analyzing important parts of a policy problem even if you cannot find values for certain variables. This will be illustrated with a popular policy question: Should a city waive property taxes for x years on certain downtown properties in order to encourage their redevelopment? Our experience shows that most beginning analysts spend all their time trying to find out (for sure) whether the tax abatement will cause the development. Almost no time gets spent trying to analyze what the costs and benefits of such a program would likely be if it were instituted and did or did not cause development. Learning to live with and work with uncertainty is a must for policy analysts. Uncertainty is present in nearly every public policy problem. 4. Say It with Numbers: Much of this book deals with using numbers to understand and resolve problems. Most policy problems have an associated data base, and it is important to use these data in gaining insights about the problem. The most basic mathematical operations—addition. subtraction, multiplication and division—can yield powerful insights. Division, for example, can tell you how much a service costs per capita. When analyzing the case on solid-waste collection, you will undoubtedly want first to perform a number of simple mathematical operations. How much garbage is collected in how much time? By crews of what sizes? Using how many of what kinds of trucks? In streets or in alleys? Using what type of trash container? Compared to other cities, how do these numbers look? High, low, equivalent? What critical data are obviously missing but essential to this first-cut analysis? Of course, not all critical factors can be measured empirically. Some are intangible, but this does not necessarily make then unimportant. Even if, in the last analysis, intangibles are found to be central, the quantitative analysis will supply a good base upon which other analyses can be done.
5. Make the Analysis Simple and Transparent: Does the analysis inform your clients? Do they understand it and as a consequence make better decisions? These central questions should be asked about any policy analysis. To achieve these goals, the analysis must be simple. This doesn’t mean simple-minded, but rather, not complex, convoluted, and impossible for a bright, well.informed client to follow. Tran.sparency is another attribute of effective analysis. This means that if any models or calculating routines are used, the client should be able to see how they work, step by step, not simply be given the results of the internal machinations of a “black box.” Simplicity and transparency go hand in hand. Using a gravity model to predict retail sales provides a good example of this principle. In judging the impact of a proposed suburban shopping mall on the downtown of a medium-sized city, planners employed a typical gravity model to assess changes in shopping patterns (Figure 1-2). Those familiar with such mod- els will know that the results shown in Figure 1-2 are the product of running data through a mathematical formula, a formula whose various parameters should be clear and transparent and open for discussion, criticism, and possible testing with data by others. The mall is both generating more total business and drawing business away from elsewhere, including downtown, Unfortunately, Figure 1-2 shows only the results of analysis. To be consistent with our principle, the factors in the formula need to be defined, the data used to generate the results should be listed, and a rationale for using this model in the given context should be provided. In some reports the analyst may want to confine such detail to an appendix, but it should always be made available so that consumers of the analysis can follow the logic and check the factors being used and their measurement If it is too complicated to explain, it should not be used in analyzing policy.
Size of proposed Mall Distribution OF ANNUAL. RETAIL TRADE Downtown Mall Other
Don’t build $40M $0 $12M 10,000 square feet $35 M $10 M $10 M 20,000 square feet $25 M $30 M $ 8 hi 40,000 square feet $15 M $45 M $ 6M M – Millions of dollars Figure 1.2 . Reilly’s Law of Retail Gravitation applied to the Upland mall
6. Check the Facts: It is important to develop a healthy skepticism for widely held beliefs and established facts in matters of public policy. Such beliefs and facts have a way of becoming base-line information for anyone whobegins to study a policy problem. Yet they are not always reliable, It takes time to feel confident enough to challenge existing authorities, but uncovering erroneous or uncorroborated facts can prevent your analysis from compounding the error. A few tips for checking the facts include:
1. Analyze the sources of the facts. Is the position of the author served by the facts? 2. Never rely on a single source. Use people, reports, and the analysis of others to corroborate the facts.
Use independent sources. 3. Understand how the facts were generated. If the method wasn’t clear, discount the facts greatly. 4. Since you can’t check everything, check the facts most closely associated with the central decision
criterion. 5. Since facts often depend on definitions, check the critical ones. If the claim is that a majority of all city
families are in poverty, how are “majority,” “city,” “families,” and “poverty” being defined and measured? Can you generate the same facts by using these definitions and assumptions?
Many planning and policy analysis issues revolve around population projections. Too often the
projections of prestigious agencies are taken as facts and become bases upon which other analyses are performed. A smaller city within a metropolitan region had done this by relying on the projections of a respected regional planning agency. The regional totals were projected objectively, with the agency using the latest birth, death, and migration rates. However, these totals were allocated to communities on the
basis of the regional agency’s plan. The plan itself was at odds with what bad been happening for the previous decade. The plan was designed to encourage centrality and dense development, but the region was becoming more spread out and more decentralized. The plan was a normative statement about the agency’s desired growth pattern, but the agency had no power to implement its plan. The resulting population projection for the small city was, then, much higher than could be expected. Until a sharp analyst reviewed the fact base (including the population projections) analyses were being done under the assumption that the agency’s city-level projections were a forecast of trends. 7. Learn to Advocate the Positions of Others: There are three principal reasons that taking a position different from your own can be beneficial. This is not to suggest that analysts should be amoral. Rather, your willingness to advocate other sides of the issue can have several positive results. (1) It can raise the level of debate, bringing out the merits of both sides and displaying the problem and alternative solutions in all their complexity. This can help lead to compromises, where if left as simple arguments or arguments based on clashing values alone, the problems may remain irresolvable. (2) This approach can improve your analytical skills and your facility with unfamiliar subject material, in the process perhaps causing you to reexamine what you have considered to be established truths. (3) This approach can also strengthen the tradition of an advocacy process where a strong challenge to an established policy — even a good policy — can result in a better policy. Competitive or advocacy processes are built into some of our most important institutions: the courts, the Congress, and free enterprise. These systems rely on conflict in order to function, and their achievements would be far fewer if they had to wait for consensus or had vested a single entity with the responsibility to take a comprehensive view. Analysts should take the opportunity to learn from lawyers, whose professional training teaches them to assume either side and to play within the rules of the legal and political process. Learn to make up for a lack of substantive knowledge — in housing, health, environment, transportation. land use — by substituting an efficient learning process. Like lawyers, students of policy analysis need to be able to develop a case from any perspective and with limited prior substantive knowledge of the problem area. Policy analysts need to know how to learn efficiently about substantive problem areas because most analysts will encounter problems that shift during their lifetime, if not daily. 8. Give the Client Analysis, Not Decisions. Policy analysts and planners usually give advice to their clients; they do not make decisions for them. This has important implications for the types of analyses that are done and, even more importantly, for the methods of communicating the results of analysis. The client will make the final choice and should be able to reanalyze the policy data. This means that critical assumptions, values, and uncertainties must be reported. When the analysis is done well, the decision maker will be able to weigh the consequences of changes in assumptions, values, and uncertainties and come to an independent conclusion. In some cases the client will be seeking a recommendation, but this is typically the case only with skilled analysts who have developed a long-term relationship with their client. This point about analysis rather than decisions can be sharpened with an illustration. It was proposed to build a bridge to replace an existing ferry service over a river which separated the downtown of a major metropolitan area from its hinterland. The analysis showed that the critical variables in deciding between the new bridge and maintaining the existing ferry were the amount of time saved by commuters and how it was valued, the uncertainty of a major cost overrun on the bridge, and the assumption that traffic would remain at levels that could be adequately served by the existing ferry fleet. Good analysis would detail these factors for decision makers and assess the consequences of varying assumptions about each. Poor analysis would simply recommend action. The key is learning to present detailed information in a format that decision makers find understandable and persuasive. 9. Push the Boundaries of Analysis beyond the “Policy Envelope”. Often problems come in very circumscribed forms. Someone has already decided what the problem is and what the alternatives are. The analyst may be able to expand both the problem definition and alternative solutions. For example, if traffic congestion is the specified problem and three alternative freeway locations are the possible solutions under consideration, a good analyst might raise questions of overall traffic efficiency and equity and advocate
adding several mass-transit options to the freeway alternatives. For new and junior analysts this may not be possible, but at least the bounds should be explored. If the results are good. consider introducing them into the formal analysis. A major portion of our potential contribution is taken from us if we are handed the problem definition. Almost nothing remains if we are also given the allowable set of alternative solutions. 10. Be Aware That There Is No Such Thing as an Absolutely Correct, Rational, and Complete Analysis. Quality of analysis can be judged only in the context of time and resources available. Students working on practice problems or cases often complain that they are never given enough time to complete the analysis satisfactorily and that teachers have unrealistic goals for what can be accomplished in the time allotted. Students fail to believe that so little time would be devoted to analyzing “such an important policy” in the field. Many of the problems in this book are, however, drawn from field experiences where practicing analysts had little time and few resources. Only in practice can the twin constraints of time and’ resources be appreciated. Even if resources and time were unlimited, the analysis would seldom be absolutely correct and complete. There would still be the issues of uncertainty and competing value systems. The analyst must ask clients the level of analysis they desire: one person-hour or ten; one person-day or seven; a month or a year of how many people’s effort? Analysts must be prepared to examine a problem at any of these levels, making recommendations where appropriate about the optimal amount of effort for each level of inquiry. Time and resources should be spent in amounts that garner the maximum marginal gain in information per dollar spent. Usually two or three levels of analysis can be identified for a policy problem, and an appropriate budget and work program can then be designed for each.
The case of a proposed parking garage can illustrate this point. Should the city clear a vacant downtown parcel and build a large municipally owned parking Structure? At the first level we may have less than ten hours in which to analyze the idea. The work program would probably concentrate on verifying the total cost to the city of building and operating the structure and estimating the revenues it might generate by using fee structures and utilization rates from several adjacent lots. Given more time, say 40 hours, the analysis might be broadened to include an examination of the opportunity costs of building the garage, that is, the benefits lost by not selling the land to a private developer who would build a 20-story tax-generating office building. A sensitivity analysis might also be performed on the garage’s proposed pricing schedule and expected utilization rate to see if the decision to build holds under pessimistic scenarios. With a work program that might stretch to six months and include several staff members, it would be possible to inventory the city’s private and public parking facilities for pricing practices and utilization rates by location and relate this information to an overall plan for downtown development. Major secondary effects such as the impact on mass-transit ridership and retail shopping could also be explored. If still more time were available, it might also be possible to enter into more detailed negotiations with potential buyers of the downtown site and get written commitments to various types of development. The overall impacts of those proposed developments could then be compared to the parking-garage alternatives.
This last level looks very much like what we have described as researched analysis. However, drawing the line between basic and researched analysis is very difficult. Most policy problems can be approached on various levels, given different time and resource constraints, and useful advice can be given to decision makers at all levels of analysis. The analysis can get more comprehensive and detailed as resources and time increase, but even when detailed analysis is done, questions remain. In the parking-garage example, these might include the uncertainty over utilization rates and the effect on mass transit and retail activity, as well as uncertainty about possible alternatives to the garage and their impacts on employment and property taxes.
We hope that beginning analysts will grow comfortable with the idea that they can provide clients a product that aids decision making, no matter how limited time and resources are. It is a matter of designing the work program to maximize information and analysis within given constraints. Getting good at this takes time and comes with experience. Practicing on sample problems and case examples is an ideal way to begin.