CANADIAN COUNCIL OF PUBLIC ACCOUNTS COMMITTEES
Report of Proceedings
(Hansard)
Tuesday, September 10, 1996
Morning Session
The meeting began at 9:08 a.m.
C. James: May I have your attention, please. Welcome again this morning to the second and final day of the sessions of the Canadian Council of Public Accounts Committees. We have this morning two international speakers, and this afternoon we have our joint session in the chamber as well, and we have council business.
Under council business, I would ask everybody to turn their minds briefly to issues they feel the council may wish to address over the course of the next year, locations where they think the conference may be held, projects they think would be useful for the council to undertake over the course of the next year -- a committee or subcommittee, whatever it happens to be -- and to have a look at the constitution for the council, which is included in your program as well. You may have certain ideas about amending that, adding to it, deleting something or just leaving it alone -- whatever you prefer. We can deal with those briefly under the council business this afternoon.
This morning in your program you will note that under Managing Public Sector Performance (continued), we have two sessions: Developments in the United Kingdom; and The Public Accounts -- the American Perspective. First, under "Developments in the United Kingdom," you will note that Mr. Lew Hughes, assistant auditor general from the office of the comptroller and auditor general in the United Kingdom, is scheduled to be here. Unfortunately, his wife is rather ill and he is unable to be here today. In his place from that office is Martin Sinclair, to my left, who will present the topic "Developments in the United Kingdom." I will turn the mike over to Martin in a moment and he can proceed with that.
Before doing so, last night a pair of glasses in a little case was found. If anybody has lost their eyeglasses, if they could make themselves known to Mary Newell in my office, I'm sure that with the proper identification she'll turn them over to you.
Secondly, if those persons who are involved in the joint session this afternoon could meet with Mr. Fred Gingell and myself just outside the chamber at coffee time this morning, we will go down to his office and review the proceedings for the joint session this afternoon. I will make that announcement also for the auditors who are involved. Now, without further ado, I will turn the mike over to Mr. Sinclair.
M. Sinclair: I'm here in kind of sorry circumstances in that I was a sort of short-notice replacement for Lew Hughes. But it's a great privilege for me to be here in British Columbia and, indeed, to have the opportunity to address you in this rather magnificent assembly chamber. Can I also, right at the start, pass on to you the good wishes of the chairman of the U.K. Public Accounts Committee, Robert Sheldon. He asked that his best wishes for a successful conference should be conveyed to you, and I do that. My boss, the comptroller and auditor general in the U.K., Sir John Bourn, also adds his good wishes.
Perhaps I should take a moment to explain my role in the national audit office to put in context what interest I have in the subject of looking for public sector performance developments. My role is that of a financial audit director in the national audit office. I am responsible for the provision of central advice to the office on accounting and audit methodology, and I'm the focal point supporting the senior managers of the office in advising on the developments that are currently taking place in the U.K. and their relations with the U.K. Treasury, the ministry of finance. So that's the position that I come from.
I spoke yesterday to the legislative auditors and listened to their debates. The echoes and similarities were indeed the most striking part of the whole day. Many of the subjects that they were discussing were readily familiar to us in the U.K. Indeed, we've learned a lot from mutual exchanges between Canada and ourselves.
If I may just digress for a moment, over the years, the U.K. national audit office has had a great many interchanges with the office of the auditor general, both professional exchanges, exchanges of staff -- we've worked closely together on audits of United Nations agencies, so we feel we know one another; sometimes you have to say perhaps it's a dangerous thing that you don't actually know the whole system; but we feel we understand each other -- and also some very specific links with the auditor general in British Columbia and, again, staff secondments in both directions.
As I indicated, the conference organizers asked Lew to speak about public sector performance developments in the U.K. I think that in the U.K. we've gone through ten years of very radical change on a whole number of fronts to reorganize our public sector, to place the emphasis on performance and on delivery. As in Canada, the political imperative, of course, is to get more for at least the same cost and preferably for less cost, and to squeeze out whatever efficiency gains we can. In that sense, I think we're part of a much wider international move. Many countries are now beginning to be interested in performance in the public sector.
[9:15]
That has largely been independent of political affiliations. It hasn't really mattered whether it's been a right-wing government or a left-wing government. This has been a central direction in public sector developments. Indeed, in the U.K. we have the situation where, having gone through 17 years of a Conservative government which has pressed very firmly for gains in public sector performance, we now have the opposition Labour Party saying: "Not only do we embrace the reforms that the Conservatives have proposed, but we actually want them to go further and faster. If we were elected, we would push it through much more directly."
Can I just define the sort of areas that I would like to talk about today? I thought I would concentrate on just three areas out of the sort of menu of things that are going on in the U.K.: firstly, the transfer of many functions of government to executive agencies; secondly, the move to commercial-style accruals accounting in the public sector and, linked to that, the move to more accruals-based budgeting as well; and thirdly, the introduction of performance measures arrangements for all functions of government.
In doing so, I won't touch on a number of other areas that you may be aware of or have heard of in the U.K. But I will be quite happy to answer questions later if you've got particular interests. I'm thinking of things like the privatization program, the competing-for-quality initiative that we have and something called the private finance initiative, which is an effort to bring private sector resources into the provision of public sector services. Nor will I speak specifically about the Citizen's Charter, though there may be some elements that I touch upon.
Can I first talk, then, about the transfer of functions to executive agencies? I suppose, reduced to the core essentials, the work-for-government department is in two distinct parts. There's the policy advice and policy development section, that part which directly supports the minister and the development of government policy. Then there are the bits of a department that are about executive action, actually implementing the policies and making those a reality on the ground.
What the U.K. government has done in the last five years is essentially to separate those two separate functions, albeit within a departmental umbrella. What has happened has been that the executive function of implementing policy has been put into what we call executive agencies. These are arm's-length organizations which have sole responsibility for implementing programs and are not involved in the policy-making element of the department. So, in effect, departments have shrunk back to a core role of information-gathering, policy-making and monitoring the way that the executive agencies perform. This is a process that has gone on across all government departments. It ranges from bodies like the Ministry of Defence which, when the process is through, might have up to 70 executive agencies under its umbrella, through to small departments where virtually the whole activity has been put into an agency.
I might illustrate with one example of a department that I know rather better than most, which is our Department of Social Security -- the organization that makes welfare payments in the U.K. It's been split into five separate agencies: the benefits agency, who are the people who assess entitlement and make the payments; the contributions agency, which is the body that manages the national insurance fund and collects the national insurance contributions and maintains our national insurance record; the war pensions agency, which does a similar job to the benefits agency in that it looks after war pensioners but has an additional welfare service, so that was thought to be a discrete package; the child support agency, which is about, in summary, tracking down errant fathers and getting them to pay support for their children; and the information technology services agency, which essentially provides the computing power to the rest.
Each of those agencies has been set up as a separate entity with its separate management team, with an appointed chief executive who is responsible directly to the minister for the delivery of services. The advantages of this arrangement can be set out four ways. I will readily admit that the first is a very arguable one: the ability to draw a very clear distinction between policy development and implementation which, it is suggested, requires different sorts of skills, so that the policy development can be left to the traditional civil servant -- the Whitehall mandarin of caricature -- while the delivery of the program can be left to business managers, often brought in from the private sector to inject that more commercial sort of approach to the activity to deliver the program.
I think you will very readily understand the difficulties in drawing these distinctions between policy development and implementation. Indeed, the recent history of some of the agencies that we have illustrates exactly those problems, where decisions taken by chief executives which they thought were within their area of autonomy -- which they thought were about administration -- turned out to be highly political. Of course, instead of the minister standing back and maintaining the arm's-length relationship which the system is supposed to create, the minister has stepped in and there have been lots of problems between the chief executives and the ministers.
Just to digress, the prison services agency has a particularly difficult history. The first chief executive got sacked following the escape of two IRA prisoners from a prison. Yet he had met his target about tightening up prison security. The number of escapes was down 50 percent, so in terms of the objectives that had been set for that chief executive, he had met that one. He had just been paid his performance bonus, yet that one single escape cost him his job. That was an example where a commercial man -- who had been chief executive of a television company -- was brought in to run the prisons. His successor was a civil servant because he would be much more manageable. He would be much more understanding of the political environment. About six weeks ago, again acting on his administrative authority, he decided that the method of assessing what a prison sentence actually meant had been wrong for 30 years. He changed it overnight, and 500 people were put in the streets without any pre-release training. Again, there was a massive political storm. The minister said this was wrong, and he overturned the chief executive's decision. Curiously, because of the difficulty of finding a prison services chief executive, for a much greater gaffe, he wasn't sacked; he has been kept at his post. So the distinctions are very difficult to draw.
I think you have touched on one of the other principal advantages of the creation of the agencies. The freedom to manage those agencies in a much more flexible way than heretofore under civil service rules has enabled private sector people to be brought into the management of the agencies. It has enabled them to be less concerned with the details of rules on staffing, pay rates and such like, provided they meet the targets. Provided that they live within the quantum of resources that have been allocated to that agency, they have been given a great deal of additional freedom to manage their activities.
We've got division between policy and implementation; arm's-length agreements between the minister and the chief executives of those agencies based on performance contract; performance targets for perhaps five or six different targets for the main areas they're responsible for; performance pay for chief executives based on meeting those targets; and a more detached relationship and greater flexibility to manage the service in the way that's best for the efficiency of that service, rather than ways that are necessarily dictated by the politician or the minister getting involved in the detailed decisions. Those I've indicated have quite a lot of problems there.
So what are the implications for the PAC in these moves? First and foremost, the proliferation of agencies means that there are a great many more people likely to appear before PACs as witnesses, because what accounting officers -- permanent undersecretaries, in our context, or deputy ministers, in the Canadian system -- have done is delegate some of their accounting officer responsibilities to the agency chief executives. The PAC, instead of calling the deputy minister or the accounting officer, may now have to call an agency chief executive to account to them, or it may have to call both a deputy minister, or the accounting officer of the department, and the accounting officer of the next-steps agency. There are quite a lot of handling problems. Who is actually accountable for the service that the PAC is investigating? Who is accountable for answering a report which we at the audit office have produced? Another factor that was surprising was that where witnesses have a commercial private sector background, they are quite unused to dealing with the accountability requirements that appearing before a Public Accounts Committee demands. This has led to a number of very fraught sessions where chief executives have not answered fully or have been evasive, in the eyes of the Public Accounts Committee, and it is quite complicated.
As I touched on, there's the issue of whether the departmental accounting officer should appear with the chief executive. There's a danger here that some areas of responsibility don't get covered. They fall between the cracks, and it's not easy for the Public Accounts Committee to know exactly to whom they should address the question. Should it go to the departmental minister or departmental deputy minister who is responsible for the policy and for monitoring the performance of the agency, or should the questions be addressed to the agency itself?
There are also a great many concerns resulting from bringing in commercial sector people to run agencies, because they bring in commercial sector practices as well, and then they quite properly want to take risks to cut corners. But at what point does that cutting of corners cut across public sector propriety, or proper accountability for the taxpayers' money entrusted to them? We have had quite a number of instances where people brought in to run agencies have conducted themselves in ways that have not been consistent with public sector standards: the usual range of things like letting houses to friends, misuse of vehicles and misuse of travel and subsistence, etc. There is a very fine borderline there between acceptable risk-taking under the flexibility extended to chief executives and impropriety and maladministration.
About three years or so ago, the PAC got so concerned about the number of cases coming before it -- often very small in their own terms, not involving a great deal of money, but enough of a drip, drip, drip of cases involving chief executives of agencies -- that they issued a report on the proper conduct of public business. This has come to be seen as a bit of a landmark, as a sort of shot across the bow of the new system, to say that we entirely support greater management flexibility; we entirely support a move to greater efficiency and better effectiveness; but just watch out -- you've got to pay attention to the proper accountability arrangements that we expect in the public sector.
So where are we now? I think it's important to emphasize that most people would accept that the policy of creating next-steps agencies has been a great success. We have seen very discernible improvements in management: greater flexibility, greater responsiveness and much more acceptance of the citizen as a customer who is entitled to proper treatment as a customer of public sector agencies. In any number of sectors you will see vast improvements in the turnaround time of improving services. For example, it used to take several weeks to get a passport; you'll now get it virtually by return of post. You've many other examples like that. So there is big success, but some warnings about public accountability and the need to do these things carefully.
[9:30]
The second area I was going to talk about was the move to commercial-style accounting and budgeting, which goes under the phrase "resource accounting and budgeting" in the U.K. Essentially, I think everyone would agree that it's absolutely fundamental that all expenditures should be properly accounted for, should be used to the best effect, and that we should know what assets are being devoted to providing public services. But predominantly in the U.K. we still budget an account on a cash basis -- on a receipts and payments basis -- with absolutely no accounting for assets and liabilities. This started to change about ten years ago, when what we call non-departmental public bodies -- the sort of fringe of commissions and other bodies -- were put onto accruals accounting. As they have been created, the executive agencies have also been put onto accruals accounting. But at the moment the core departments themselves remain on a cash basis. Last year the government took the decision that all departmental accounts will go onto an accruals basis before the turn of the century, and this will involve an extensive process of actually finding out what assets the government departments hold, valuing them and establishing a proper basis of determining the cost of delivering services.
So the overall aim of this process is to improve performance in the public sector by improving management and stewardship -- in particular, to recognize the full cost of holding and using assets -- to provide a much better focus for fixing outputs and expenditures against the aims that are set for each individual program. Again, that's something we didn't have universally across the government sector. This is going to result in reporting total resources, not just the cash that is used in achieving the aims.
This is a huge challenge for departments. Essentially, at the moment they don't have the necessary accounting systems. They continue to think in cash terms only, because in the reality of it, the way you got on in a government department was by getting a good public expenditure -- around settlement with the treasury by winning cash to fund your services -- not by thinking about what the total cost of providing those services and making the necessary trade-offs between the assets that you had used.
That's going to involve a very significant change in attitude amongst the civil servants who are running departments. They will now have to think longer-term. They will now have to think about the cost of assets, not just securing the cash to provide the services. In doing so, they're going to have to think -- almost for the first time in some cases -- about defining and specifying the aims and objectives of the programs they're delivering, and they're going to have to develop appropriate performance criteria to set against those programs. Again, there are major implications here for parliament and for the PAC.
It is strange to say, but despite this being in effect the first major reform of accounting and budgeting in the U.K. since Gladstone and for 150 years, it's been quite difficult to engage the Public Accounts Committee in a discussion about this process, which we found quite difficult to understand. The lead in the U.K. has actually been taken by the treasury committee, who are the people more interested in the budgeting side of it, as opposed to the PAC, who are interested in the financial reporting and accountability side. But the PAC is going to have to learn to cope with an entirely different presentation of financial information to them, and they're going to have to think not just in cash terms but of the total cost of providing services, and of the total cost of providing and holding assets.
We face the prospect that when these accounts first come on stream around the year 2000, we're going to have many qualifications of those accounts, because the departments have not really been able to get on top of the requirements as quickly as we might have liked. So the PAC is going to have to be selective in the way that it tackles those issues and in the things that it picks up. I think we're in for a very interesting time. When we start to fully appreciate the real cost of providing services, we're going to throw up a multitude of issues about questions of value for money, things that we've taken on trust for years. We're actually going to know how much it costs to provide things for the first time ever. So I think that's going to stimulate a great amount of debate, both in the Public Accounts Committee and in Parliament more generally about the direction the government is going in, and about the directions and choices that we have to make in spending our resources.
The third area I'm going to talk about is to look in a bit more detail at the introduction of performance measurement. Really, as you'll understand, it's linked to the previous developments I've been discussing. I've indicated that this is a sea change. There has always been a sort of limited range of performance information available as part of government departments' annual reports, but in the U.K. that has been separate from their financial reporting, and the two sides have not been linked together. The movement to agency status for many parts of departments coupled with commercial-style accounting has greatly increased the volume of published data that we have available. As I indicated, each chief executive has an agreement with the Secretary of State, which is composed of half a dozen performance indicators. Those are now published in his annual report, so you can look at trends over time and see whether the targets have been achieved or not.
At the moment, there are still big questions about the validity of the targets, because they are not subject to any independent validation. We don't know whether the targets themselves are soft targets, testing targets. As yet, we have no validation of the performance outcome against those targets. Indeed, it's an interesting element of the resource accounting proposals that the government does not intend to ask the national audit office to validate these performance measures, although it's obviously something that we are pressing very strongly to have brought into the resource accounting package.
As I said, implicit in the move of departments to resource accounting and to reporting expenditure against objectives is a much clearer articulation of those aims and objectives, and the setting of targets and analysis of costs in meeting those targets, into a proper framework. We still have quite a considerable way to go in the next 12 to 18 months before we see exactly the shape of those frameworks, but the most striking thing at the moment is the political commitment from both government and opposition to get this done. We're not talking about something that will drift on for years up to the turn of the century. The political will is there to make it happen and to give it some reality.
We in the audit office, of course, are very much in support of these developments. We would like to be in a position to offer Parliament the service of validating the performance measures -- certainly the measurement of the outcome performance against the targets that have been set by the government and by ministers. We are in the process of developing our methodology and, of course, are beginning to flush out many problems in doing this. In a number of agencies, for example, we found that the targets set for the chief executive by the Secretary of State don't bear any relationship to the published Citizen's Charter targets which the customers -- the public -- are told they can expect to receive. Often we find that the Secretary of State's targets are somewhat less than the expectations expressed to the public.
I think there is a whole range of important issues for the PAC. If performance measurement is going to be the basis for decisions on resource allocation, the PAC is going to have to see that that is done properly. The PAC is going to have to become skilled in interpreting performance measures, in examining them and becoming familiar with using them -- and using them to question chief executives. I think the PAC will want to see that performance measurement data validated, but we are some way off that yet.
Overall, what conclusions can we draw? The U.K. is following a general worldwide trend. We very much back the move to greater focus on performance against aims and objectives, of proper resource accounting and full costing of achieving those programs as an aid to better decision-making. We've got a range of initiatives in place. Some of the changes for functions being retained in-house are linked and are fundamental to bringing about performance improvements. In the U.K., the establishment of the executive agencies has been core. Commercial accounting will bring about the focus of performance measurement that we are trying to achieve.
We are at a stage where both auditors and Public Accounts Committees have to work with departments and agencies. We have to be supportive through this changing phase. There are bound to be teething problems as departments and government learn. Some agencies are going to fall on their faces, because that's how lessons are learned. At least, that tends to be how lessons are learned in the U.K. -- it's the failures and the cockups which provide the best lessons, so there needs to be some tolerance there.
In both the audit office and in the PAC, we have a very important role -- a vital role, really -- in providing a constructive but questioning attitude to the changes and the new information. It really is quite an exciting period. There is the prospect of much more effective management of government resources, something we should all be striving for.
C. James: Thank you very much, Martin. Are there any questions or comments?
C. Santos: If the performance behaviours included in the executive contract with the Secretary of State are not validated, in the sense that they do not really measure or do not precisely measure what they are supposed to be measuring, how do you know whether they are doing good or not?
M. Sinclair: Good question. We have been doing a number of reports which have looked at the setting of the performance targets on chief executives by the Secretary of State. We've been asking exactly the sorts of questions that you've been posing. Do they cover the full range of activity? Are they meaningful measures? Are they items that can be measured? Can they be validated? Are the measurement systems robust? Can we rely on those figures?
Of course, what we're finding is that the pattern is inconsistent. Some you can; some you can't. We find inconsistencies between those targets and what the agencies say to their customers. By pointing these out and bringing this to the attention of the Public Accounts Committee, we're hoping to exert pressure on government to be more robust in setting those targets.
Of course, the natural inclination of the chief executive, because . . . . This has become almost a negotiation in some instances between the minister and the chief executive to reach targets which the minister thinks are sufficiently tough but the chief executive feels he's got a chance of achieving. It's quite clear that, depending on the nature of those negotiations, sometimes the targets have been too soft. It certainly doesn't seem to have been the pattern that they've been too tough, because almost universally chief executives have received their performance bonus or at least some element of their performance bonus.
It is interesting that in one study where we looked at this in a very great amount of detail, on the meteorological office in the U.K., the report led to the chief executive having to repay his performance bonus. When we validated the measures, we found that they had been done in such a way as to take him over the relevant targets, when in fact on four out of the ten measures his actual performance was at least one or two percentage points below his reported out-turn. That resulted in him having to repay the bonus.
[9:45]
J. Williams: I enjoyed your presentation, Martin. I've got a couple of questions on the executive agencies which have been created at arm's length to the government. I'm thinking about the unions and the union agreements. What has been the position regarding union agreements? Do they follow the movement of the people to the executive agency? Does the collective agreement remain in place? Or is the executive director or chief executive officer able to start anew and move off in his own direction? Or are they still very much a public service part of the collective union?
Another question I have is on accountability. Early in your presentation, I think, the phrase you used was something like "competing in quality" or along that line. We are focusing a lot on objectives and performance reviews. But are you trying to build some methodology of competition into government, so that there's an ongoing challenge to ensure that the executive agencies are delivering service that the customer needs, on an ongoing basis? Could you elaborate on that?
M. Sinclair: Okay. If I start by looking at your question on union agreements, yes, what has happened is that department staff have been transferred into those agencies. The union agreements have followed them, for existing staff. But the chief executives have been given a great deal more freedom to adjust the terms and conditions for new people coming into the agency. They have also been given greater flexibility and, in many cases, some money to make compulsory redundancies and so free up some resource within the agency. Many new people coming into these agencies are now coming in on short-term or casual contracts. I think that almost universally the agencies have worked to reduce their core staff so that they're considerably smaller in terms of the core staff than they were when they were departments.
So the union agreements are in place, but unions are now finding that they have to deal with each individual agency on many matters. Probably excluding pay, on almost anything else they will have to deal with the chief executive. Even on some pay matters, most executive agencies now operate a performance bonus system, and the union will negotiate with the chief executive on the way that that performance bonus is distributed, even if they're still negotiating with the department centrally on the mainstream pay ranges.
J. Williams: Is there an actual move to try to break the union into constituent employers? Or do you envisage that the unions will remain monolithic, representing the entire public service?
M. Sinclair: At the moment the unions are still monolithic, but that block is very rapidly breaking down. The union is having to readjust to an environment where they have to deal with several hundred chief executives on many of the matters on which it used to just deal with the office of public service. The office of public service is very much pushing and delegating responsibilities for most staff-related matters to departments, and then on into the executive agencies.
J. Williams: The second part of my question?
M. Sinclair: You asked about competing for quality. Perhaps I should put some of these reforms into context in the U.K. Essentially, the government strategy is now fairly well embraced by the opposition, though not completely. It has been, first of all: does government need to do it to undertake this activity? If not, let's privatize it. So almost all of what you would call Crown corporations are now in the private sector -- privatized. I think there are only two left: the Post Office and the BBC.
Within government, there are a range of services that we must continue to provide. But what can we do to bring the private sector in, so that many activities are being contracted out? Information services is a good one. Many government departments are now outsourcing their computing activity, so we're encouraging competition for provision of those services, hopefully getting them to be more efficient.
Many of the next-step agencies themselves are now candidates for privatization, because the view is that they can be made more efficient -- that many administrative functions can be made more efficient by having the private sector bid for those contracts and come in and take them over. So it wouldn't be . . . . I think very soon we may well have the administration of passports, for example, done by a private sector contractor -- a contract let under rigorous competition -- and more and more of government is going to end up in the private sector. Agencies and departments will become more like contracting agencies: purchasers of services. They will have a choice of whether to purchase that service from the public sector or the private sector, but there will effectively be a competition between them to see who is the most efficient provider.
I think your question probably went a bit further than that as to whether we actually test the demand for the services that government is providing. I think we really haven't found a good way of doing that, though every agency is now subject to what's called a fundamental review every five years, where it must justify its continued existence and the continued provision of the service that it's set up to provide. That has become a very rigorous process, though I think there has only been one agency that has been created to date that was actually then closed down because its remit was no longer necessary.
M. Abdurahman: This is all being done in the name of accountability. The question that comes to my mind is: where is the ministerial accountability in this process? You used the prison service as an example, where the chief executive officer had appeared to have met his performance goals, and yet he was the one who was canned and the minister came out looking rosy. I'm really concerned: where are we going with this accountability if the bottom line is that ministers are no longer being held accountable? That's what I could see when I visited Britain, quite frankly.
M. Sinclair: This is a concern shared by many people in Britain. The days when ministers resigned because of administrative failings are gone. Ministers have been quite clever in saying: "I've set up an agency. I've delegated the authority to manage that service to the chief executive. If you have a problem with the way that the service is being administered, you must question the chief executive about it. I'm not responsible for that activity, or, if I am responsible, it's only in the most general way of setting the policy, determining the performance targets and measuring performance against them. I'm not going to be held accountable for the specific detail of the administration."
They seem to have been able to sustain that approach, because the recent history is that that's exactly how it has worked. Chief executives have taken the responsibility and have been accountable for administrative errors and failings, whereas ministers have been held accountable only for policy, even to the extent that in the House of Commons, if members ask questions, it used to be that you got an automatic response from the minister. But ministers now simply write back and say: "Thank you. This is a matter for the chief executive of Agency Blah. That chief executive will write to you in due course." That's what has happened.
Initially, when the system was first set up, there was no requirement even to publish the chief executive's reply to that inquiry, but we now have a system where at least the chief executive's reply is published in Hansard. It took about six months of fighting to get even to that level of accountability. One the one hand, the system at the macro level has been strengthened, and accountability is getting much greater, but I think there is quite a genuine concern, with some truth in it, that at the micro level, accountability is actually diminishing.
F. Gingell: Martin, you spoke of the speed at which passports are now sent out. I guess that one of the important things about the issuance of passports is that you issue them only to people who are entitled to them, and that important function may be lessened by a desire to get them out as quickly as you can. How is that being balanced? Is there concern about the propriety of the passports that are being issued?
M. Sinclair: That's not an issue that has come up, but in establishing the performance measures for the passport agency, several of the criteria will be about the quality of the work done, so that should always be built into the performance measures established for the service. The trade-off between quality and speed is not one that has come into focus, because these improvements in efficiency have always been required at least the same or, if not, at better quality. So I don't think that's a particular difficulty we have faced.
C. James: Any further questions or comments for Martin? If not, thank you very much, Martin, for your presentation this morning.
We'll now move on to the next business session, which is The Public Accounts: the American Perspective, from Mr. Harry Hatry, director of public management programs of the Urban Institute in Washington, D.C.
H. Hatry: I'm going to be using some overheads, but please do not turn down the lights. I don't want to lose the audience.
It's a pleasure, and I certainly am very appreciative of being invited to Canada once again to speak to you. I've had the advantage of visiting, I think, at least six of your provinces, and they are all beautiful. It's a very delightful and hospitable country. I'm very pleased to be here.
Let me first mention that I'm with the Urban Institute in Washington, D.C. It's a private, non-profit organization. It receives funds in the way of contracts and grants from many different governments, as well as from foundations. Also, we have an endowment. We are non-partisan, just as the Public Accounts Committees are supposedly non-partisan. It's sometimes difficult to remain that way, but we certainly try to do that. We're a policy analysis research organization. I'm going to be drawing on experience from many years of working with many different individual executive branch agencies at local, state and federal levels. I will occasionally draw also from working with some other countries.
[10:00]
I have to tell you a story of Johnny, who was sleeping in his bed and his mother came up to him and shook him and said: "Johnny, you have to get up." Johnny mumbled and said: "I don't want to get up." And his mother said: "You have to get up." "But I don't want to get up." "You have to get up." "I don't want to get up. Why do I have to get up?" His mother said: "Well, you're 45, and you're the Prime Minister." The elected officials take quite a beating in the public. They do in the United States; I'm sure they do here in Canada. Wrestling with all these issues that we're talking about today is very difficult.
I'm going to be focussing on non-financial reporting issues: getting value for money. Some people have used the terms "managing for results" and "budgeting for results," and I think the new term that you should be concerned with is "legislating for results," which I think is a good idea. Certainly as Craig Holt said last night very well, the bottom line of all of this is improved services, using taxpayers' money -- improved performance. I'm going to be talking about a lot of measurement-reporting. It's not for its own sake, but it's very important for that.
I'm going to give you a tour through these topics. These are very highly interrelated issues, and I'm going to be talking about each one of them in turn. It's sometimes very difficult to know when you're talking about one versus the other, because they're very overlapping. For example, all of these except for Nos. 8 and 9 need performance indicators and measurements in order to make any sense out of them. You can't do the first six items unless you have some sort of decent measurement and certain decent data. So they're all very much related. However, what I'm going to try to do is . . . . Plus there is a lot of difference in definitions, which I'm not going to try to get into. What I will try to do is describe some of the practices, give you some sources so you can go for further information on what's happening, as far as I know, in the world today, as well as give you some pros and cons of each of these tools. These are just tools that are available to governments of any sort.
I'll point out quickly that in the United States, during a period when downsizing is a major issue -- Canada is certainly facing it. . . . Most countries throughout the world are concerned with downsizing. We have downsizing; we have re-engineering; we have reinvention. We have all these things, and it does get overwhelming with some of the public officials. Certainly in the federal government, particularly, where all these things have been dumped on the federal government agencies, it can, if you're not careful, overwhelm some of the management.
I want to go to strategic planning first. Let me ask you: how many of your provinces require strategic planning by agencies that you know of? Do any of your provinces require strategic plans? There are some people nodding their heads. I think most of you have business plans. It's a little difficult to distinguish these things sometimes.
In strategic planning, I'll mention a few things that are happening. First of all, in the government and state government sector -- and Craig waved a version of this at you -- there is the Oregon Benchmarks project, which is one form of strategic planning. Very similar to that -- in fact, they really evolved from the Minnesota ones -- are the Minnesota Milestones. We have Oregon Benchmarks; we have Minnesota Milestones; and thirdly, we have a more recent one from June '96, the Florida Benchmarks report. So you have three states that are very much into one form of what can be called strategic planning.
By the way, these are available. If any of you want to look at these during the rest of the day, they'll be available to you to look at. I think I've suggested to Craig that perhaps he might want to xerox the cover pages and give them to all of you, so that if you're interested, you can go and access them and look at them in more detail.
They all have certain characteristics. They all involve major statewide efforts to bring in citizens. All of them have used large numbers of citizen meetings and groups throughout the states. There has been travelling throughout the states to have group meetings and to identify the indicators that people are very concerned about. So every one of them has performance indicators in them that represent some statewide concern, such as low-weight births, educational elements and health characteristics. They all are outcome-focused, so that they do focus on outcomes. They are not just state indicators for state agencies; they're not agency indicators as such. They're statewide indicators. It's a very important point in these three states' style of approach to keep in mind.
I want to contrast that with some body like the state of Texas, which has done some strategic planning but which focuses on strategic planning for their agencies. The same thing is being done by the federal government in the United States. Each federal agency is supposed to provide a strategic plan. I'll be discussing this in just a minute. So in the case of Texas and the federal government, they are agency-oriented plans prepared by the agencies; in the case of Oregon, Minnesota and Florida, they are statewide and involve everyone.
In fact, when I was asked to evaluate the Oregon Benchmarks project two years ago, we found a great deal of trouble. The state agencies had trouble relating their work to the statewide Oregon Benchmarks indicators -- it's that alignment problem that Craig Holt mentioned yesterday. There is a problem in covering. . . . They were struggling in the state agencies to relate what they were doing to these statewide indicators. It's very attractive, because it does get the citizens involved very heavily and is statewide. But one thing you have to be careful about, which Oregon and others have found, is that the strategic plans do not cover everything that the states do. There's a danger of leaving out some major state activities in a state plan, in either demoralizing them or otherwise ignoring them. For example, in Oregon, if you look at their 250-some-odd indicators, they have not included an indicator of road rideability -- road bumpings, I like to say. That's a major activity of that state. The money they put into road maintenance is a very big piece of business. So if you're doing strategic planning of that sort, you have to be careful, to be sure that you somehow take account of major programs the state is doing and how to handle those.
The pluses of strategic planning. It's a great idea. I'm not sure I'm fully supportive of the implementation. In all these tools it's one thing to talk about them; they're usually great ideas. Implementation is murder, and it's very important, obviously, and the use of the stuff properly, and strategic plans particularly are having this problem. They are very good at getting managers together to do strategic thinking; they get away from the office for a bit. Usually, certainly in the federal agencies in the United States, the agencies have taken their people on retreats -- perhaps assistant secretaries who have never before gotten together in sort of a joint effort -- and they've tried to work out strategies, and that's a great idea. So that's a big advantage. They do try to put on indicators and targets for the future.
Almost everyone has been very weak in looking at strategies in the real business sense, which is analysing different ways of doing things. Usually the top level officials who do these sort of focus in very quickly on one approach, and they really document that. So they haven't really spent enough time looking at the future population projections -- the environmental changes that are likely in the future. So those are important issues to be concerned about.
As elected officials, sustainability of strategic plans is another issue that I'm sure you are very sensitive to. The strategic plans usually tend to be three to five years out. The benchmarks in Oregon and Florida go out to 2010. The federal government requires five-year strategic plans, but the trouble is that we have elections in between, and major people are switched. Sometimes different parties -- even the same party -- may change a major person. For example, the federal Food and Drug Administration has very recently been ordered to regulate tobacco products and their advertising. That's a major new theme which didn't get into their strategic plan. So there is a question; you have to be careful. These strategic plans change very quickly. Ideally, the basic mission statements, objectives and indicators should be fairly stable; how you get at those that normally will change.
Performance measurement. I think we can say that Christopher Columbus and his crew were probably the first executive branch agency. When they started, they didn't know where they were going. When they arrived, they didn't know where they were. When they got back, they didn't know where they had been. And they did it all on government money. Performance measurement is really neat. It can tell you a little bit about where you are going and, subsequently, where you've been. It really is a very important piece of business; it's a basic underlying element needed for most management. We have the analogy from the business sector: there's a bottom line, profits and market share. Another popular analogy is in the sports field, where you've got to know the score to know whether you're winning or losing, and therefore to know whether you need to make changes or not. So performance measurement is very critical.
There are two forms of performance measurement that you all deal with and that public officials deal with. First, there are the special studies that are done. The legislative audits that you all review are a way, if they are performance audits, to get at information on performance on just a few programs each year that they've done -- because you don't have the resources to cover the hundreds of programs in a government or even in one agency. So that's their limitation. It can only be done periodically on a very few programs.
It's the other form that's gotten extremely popular and is the current in-thing throughout the world. It's regular performance measurement, which means you regularly, at least annually, measure a set of outcomes and report on indicators for each of those indicators. The first problem everyone faces is what indicators should be measured. Craig went through this, and most of yesterday we had some discussions. We're not here talking about the more traditional type of measurement, which is . . . . I assume that most of your agencies keep track internally, in that when you do budgeting, you review outputs that they produce, like number of miles of roads maintained, repaired or constructed; number of meals served in a prison, in an institution for the handicapped or in hospitals, or number of applications processed. These are all good output indicators that managers need, and they do help when budgeting. But, of course, they do not get at outcomes, at what the results and the effects are.
[10:15]
In the evening last night -- you weren't there -- in downtown Victoria there was this drunk under the street light, looking around. A policeman came up to him and said: "What are you looking for? Can I help you?" And he said: "Well, I'm looking for my keys. I lost my car keys." The policeman said: "Where did you last see the car keys?" The man said, "They're over there; I saw them last over there," and he pointed to this dark alley. The policeman said: "Well, why aren't you looking over there? Why are you looking here?" The drunk said: "It's the only place I can see." That's the same thing that we've done traditionally in measurement. We measure what we can easily see, which are the output indicators. We've been fairly reluctant to try to push this out to much more of the outcome type of information. I'll be getting into that a little more.
Let me try to give you a definition of outcome indicators. They're kind of difficult to define, but let me take a shot at it anyway. The distinction that I'm drawing here is between something that the agency itself does and something that happens to the customer.
Fred talked earlier about response times. Response times, by the way, have been the major outcome indicator that agencies throughout the world are reporting on. When they perceive from outputs, the first thing they do is measure response times. Those are good. Response times are important in almost every service imaginable. I don't know of any service where, other things being equal, it's not better to do something quicker than slower. Response times are important to customers, and how you respond to response times. But as Fred pointed out, response times are falling, but they do not tell you what happened if they responded. They may still have goofed or spoiled and done a bad job on the response itself, even though you responded quickly. In the field, we've begun to use the term "intermediate outcome" for things like response times, because they're important to customers. But they do not tell you the end outcome. So we sometimes distinguish between intermediate and end outcomes.
One thing that's extremely important to deal with is expectations about performance measurement. There's a tremendous limitation that you should all recognize in the performance measurements you will get if you move into outcome measurement, such as British Columbia is trying to do. I don't care what it is, it has this limitation: it does not tell you why something happened. It only tells you an outcome. It's like the score of a baseball, football or soccer game. It tells you whether you're winning or losing. It doesn't tell you why. We don't have the basic tools to tell you fully why. Now, to do that, you need to question the whys. You need to go further, as to why the outcomes occurred, and you usually need special studies to determine the whys.
I want to make this clear. Whenever I deal with the media on questions of measurement I have to spend several minutes with them going over this point, because they assume. . . . It's a very important point. This accountability issue is very, very peculiar on performance. Very seldom is an outcome affected only by a single public agency. I don't know of any end outcome that's affected by only that agency. There are always external factors such as economic conditions and economic types of indicators that affect outcomes, so I think we need a new paradigm of accountability. I'll be getting into this a little bit later when we talk about performance partnerships. It's a very important point that I want to bring to bear on you.
There are some things that are going on today that you might want to look into. Minnesota has done some very good work. For several years the Minnesota Trade Office has been doing feedback surveys of its business customers. Usually we talk about customer surveys as being households or individuals, but in the case of economic development and helping businesses, they are in the export trade promotion and in small business development. They regularly take surveys of the clients of the trade office to see how well they are doing. In the early days, in one of the first studies they did, they found out that they were doing very well in Minneapolis-St. Paul -- the metropolitan areas -- but the returns from the outlying areas of the state of Minnesota were much worse. They therefore shifted more resources attention to the outlying areas in order to help some of those locations. Minnesota has also just released a new handbook by its Human Services, which is very good: Focus on Client Outcomes, which I do recommend to you.
The federal government in the United States has passed something called the Government Performance Results Act. It was passed in August of 1993 unanimously by the Senate and the House. As of today, we still don't know how many of the Representatives and Senators really know what's in it. There's a movement now, that I'm part of, to try to give them some more education into what's in it and how they can use this material. This calls for approximately . . . . Starting in August 1993, it requires full implementation by the year 2000, so it's about a seven-year time period. Yesterday you heard about six-year time periods. In this case, we're talking about from the time they start to when the first performance reports are prepared.
Let me show you the time schedule which I think may be of interest to you for the federal Government Performance Results Act. Even though it started in '93, things got along very slowly. They had some pilots going on. Another process is to provide for pilot projects. By September '97, which is coming up next year, they are required to have a strategic plan for each major agency and program and also a first annual performance plan. In the performance plan, they include a set of performance indicators showing outputs and outcomes. Both outputs and outcomes are required by each major federal program by September 1997, plus targets for the fiscal year. This is actually for the fiscal year '99. They have a very-long-term planning schedule, so for each of the indicators there will be a target set for the forthcoming fiscal year. Then in January 1998, OMB submits one, puts them all together and submits it to the President and Congress. Since the '99 fiscal year ends September 30, 1999, they're giving them six months afterwards, until March 2000, to submit the first annual performance report. It's a very tough time schedule.
There is a great deal of activity, and for those of you who are interested in particular services, you may want to look at the federal agencies' efforts in both strategic plans and the indicators that they've been developing. As you can guess, it's enormously mixed. There is enormous activity going on in this whole area, as they're now building up to this period. Some are doing very well and some are having very big difficulties, as you can imagine. These outcome measures are really something different.
There's also effort at the state and local government levels. Have any of you heard of the U.S. Governmental Accounting Standards Board? There's something called the Governmental Accounting Standards Board, which is an independent, private non-profit organization that sets accounting and reporting standards for state and local governments in the United States, and most states adopt their standards. When Elmer Stotts, the head of GAO for 14 years, who was very well known and recognized, retired, he became director of that GASB. At that time he moved into what's called service, efforts and accomplishments reporting.
There has been a series of reports on a number of different topics -- about ten service areas -- using accountants to identify indicators, and you may want to look at that series. I did not bring that with me, but that should be readily available to you through the Governmental Accounting Standards Board. They've been recommending experimentation in reporting annually. They're concerned about reporting to the public. What they were seeing was that the financial statements are terribly drab. I don't know if any of you ever read financial statements -- well, you probably do: they come out two years later, after the end of the fiscal year. They're dull as hell. So they have been pushing for the other side of the coin: what are you getting for the money? That's been the position.
One city -- Portland, Oregon -- has come out for about three years with something called their annual report on service efforts and accomplishments. It's a very well done piece of business, and I do recommend that you look at it. They do cover city services. The city auditor actually does this work in preparing this from the agencies and does require information to come from the agencies. One thing that's very interesting is that a major part of their work annually is to mail a survey instrument. Here is a copy sent out in '92 -- it's a little old, but it's the same one they were using -- to about 10,000 randomly selected households. They have about two or three mailings, and they get back a return rate of about 50 percent, which is just amazing. They asked them a number of questions to evaluate each of these services in some detail. If anyone wants to look at that questionnaire, you're welcome to do that. They report that information regularly in the annual report.
Something else that's intriguing in the United States is that we have a very big, major sector of non-profit agencies. I assume you also have human service delivery agencies to some extent. You have the United Way of Canada, I believe. Is that correct? The United Way of America is now working with United Ways, and several are now saying that their donors are telling them that they want to know what they're getting for the donated money. People are charitably providing funds through the United Way campaigns annually, which the local United Way will then disseminate to what could be hundreds of little charities like the Boy Scouts, the Red Cross, the church societies, welfare groups and private day care centres, to help them. That is a major effort, and some of the best work in measurement is coming out of places like the United Ways of Minnesota, Milwaukee and New Orleans.
The United Way of America has just produced a volume called Measuring Program Outcomes: A Practical Approach, and I am proud to say that I was part of this. This is being issued through the United Ways to the thousands of service agencies in the United States, to encourage the agencies themselves to begin to measure the outcomes of their own work. The idea is to get each of the service delivery agencies to start measuring what happens to the clients after they receive services. The local United Ways then request that information be fed back. They can use it, but they're very cautious about how to use it in allocating funds. This is a new piece of business, but they're very cautious about how to do it. I think most of the United Ways will take the position locally that they're going to try to encourage, but not require, reporting, at least initially. I'm going to leave a copy of this with Craig, if anyone wants to look at it.
We want to break at 10:30, is that correct, Craig?
C. James: Yes, whenever there's a logical break in the presentation.
H. Hatry: I'm almost finished with the performance measurement material. Before we break, let me finish with one other piece of business. I'll make a commercial pitch.
We're into new technology. We talked about new technology yesterday. The Urban Institute, along with three other public sector organizations. . . . The American Society for Public Administration, the International City-County Management Association and Public Technology Inc., which are all professional interest groups for local governments, have issued a CD-ROM on performance measurement. It has a tutorial which you can give to any program manager to help them design their own system. It also has a "now it's your turn," where you actually put in the keyboard your own mission statements for a program, your own indicators and some other bells and whistles. If you're interested, copies of that can be obtained from the International City Management Association in Washington, D.C.
Let's stop at this point. Are there any questions or comments on performance measurement or strategic planning that anyone wants to raise?
[10:30]
C. James: If not, Harry, if it's all right, we'll break between now and 11. Members, feel free to wander back up to the Ned DeBeck Lounge. If those who are presenting in the joint session this afternoon could remain behind, I'll lead them down to Mr. Gingell's office for a review of the presentations for this afternoon. We'll start again at 11:00 and continue the presentation, followed by questions and comments on Harry Hatry's presentation.
The meeting recessed from 10:32 a.m. to 11:10 a.m.
N. Reimer: For those of you who haven't met me, my name is Neil Reimer. I'm a committee Clerk here in the province of British Columbia. I am alone at the table for the moment. Mr. Gingell will be back shortly, but perhaps I could simply turn it over again to Harry Hatry to continue his presentation.
H. Hatry: I have just one last piece of business for your information. Another very good source of information on performance measurements is the state of Texas appropriations from the House bill. They actually put performance indicators into that bill. This copy was too heavy for me to bring up from Washington, so Craig was nice enough to loan me this. You may want to look at that.
This is a fun topic -- customer service standards. Probably the best-known customer service standards are the United Kingdom's Citizen's Charters. This is a great idea if it's properly done. The President of the United States in 1993 issued an executive order called Setting Customer Service Standards. It's a two-page, very brief. . . . It requires each federal agency to survey its customers regularly, to post service standards and measure results against them. You may want to look at this and use it as a starting point for your own version.
This piece of business was issued fairly recently: Putting Customers First, '95: Standards for Serving the American People. It is the first year's set of customer service standards for United States agencies. It's got hundreds of service standards that the agencies have identified.
A Voice: Whose names are on the front of that?
H. Hatry: Bill Clinton and Al Gore are the people on the cover. It's part of the national performance review, so it's a popular document.
Interjection.
H. Hatry: Putting Customers First, '95. I'll have a comment a little bit later on that.
The United Kingdom also. . . . Something that Martin Sinclair didn't mention is the local water commission. They have done a remarkable job of comparative performance measurements on citizen-oriented indicators. This is an old piece. I think the indicators were initially mostly output types of things, but they've been improving in recent years.
The country of Colombia, which I visit on the world. . . . Do any of you speak Spanish? Trato hecho!
Interjection.
H. Hatry: What does it mean? They tell me it translates into a wonderful term: "It's a deal." They've started to do customer service standards. The government of Thailand has some customer service standards. The country of Canada -- you know where that is -- in 1994 started a service standard project, which I understand from the Canadian auditor general's office has probably tapered off a bit.
[11:15]
Let me tell you a couple of problems that people have had with these. It's a wonderful idea if it's properly implemented. The first thing you find is that many of these standards are very vague. It doesn't do much good to. . . . Let me read you one. I just sort of opened this at random. This is a Clinton-Gore piece, one from the State Department. Business Facilitation Services to Help Businesses. It says: "Our service standards: we will provide these services in a prompt, responsive and courteous manner." The same thing occurred yesterday. That's what it says. To me that's not a service standard; it's a general statement of what you'd like in general to do. So without putting some numbers on it, it doesn't mean much.
The second thing is that to make progress on standards, you need to measure and report them. It's just obvious. If you're going to have service standards, you need to measure what actually happened and to later report them to the public, to the customers. That is something that just makes sense to me. Many of these organizations have not done very much of it. The U.K. has begun to do it; I think the National Performance Review in the United States plans to do that. For the moment, they've just identified standards.
The thing we ran across in a couple of these countries -- and in Colombia we had debates with the woman in charge of this -- is that if you're going to have a customer service standard, it should be part of your regular performance measurement system. If it's important enough to be called a customer service standard, it should be important enough to have in your system of performance measurement, and it should be reported regularly. The information should be fed back not just to the customers, but I strongly urge you to encourage your agencies to provide the data back to the agency staff -- everyone, including the secretaries and clerical workers. Have them post. . . . I love the idea of posting these performance indicators on everybody's door and perhaps drawing a red line that shows where they are in that quarter, so everybody has these customer-oriented standards.
Finally, there's one issue which is fairly dramatic. Most of these standards -- "it's a deal" -- only go one way. I'm promoting the idea that if you're going to do this, do it both ways. What do I mean by that? I mean that it's not just a question of what the government does for you but also that, for most services, the citizen or business has a role to play. Let me take a simple one: solid waste collection. The local government has the role of collecting the garbage, doing it quietly, on time, without banging up things. What is a citizen supposed to do? Just yell it out. Does the citizen have a role in collection of garbage? They have to put it out in the right place in the right container; if it's recycled, they have to. . . . I believe that we should be moving towards the notion of two-way standards. That we should have both what the government agency will do and what the citizen will do. I think we should report on both.
Let me just finish with this on customer service standards. I want to make a point on customer surveys, because one of the major tools that I think should be in customer service standards is some variation of customer satisfaction. It's all right to talk about response times and what have you, but you also want to see if the customers were satisfied with the service they received. So I do recommend that these be linked with service standards, that there be service standards on the indicator percentage of citizens who gave an excellent, good, fair or poor rating to the service. To my knowledge, no one has done this yet, and I would recommend it for the future.
There is an exception. I made a misstatement: what country? Does anybody recognize this? She is not here, but this is Alberta's Measuring Up, first annual report, June 1995. The Health department did a phone survey in 1995 of 4,000 Albertans on health and the health system in Alberta, and they rated them excellent, good, fair or poor.
Benchmarking. What does the word "benchmarking" mean to you? There's a real problem with definitions in this business, and benchmarking is one of the terrible terms. Would somebody just indicate what benchmarking means to Ontario? How do you use the term? What do you think it means?
G. Pouliot: My colleagues are quite busy. I'm used to questions in question period but not question-and-answer. "Benchmark" would mean basic, the common denominator or what you use as a base to go by.
H. Hatry: Does anybody else want to. . .?
J. Leefe: In the Canadian geodetic survey, a benchmark is a position which gives you exactly where you are geographically at that point.
H. Hatry: Fine. In actual practice, we have people using the term differently. That's the point I want to make. You heard the term "Oregon Benchmarks" from Craig yesterday; I'm sure you've heard of the Oregon Benchmarks by now. They use the term to mean both the indicators of performance as well as the targets, the numerical number that they want to aim at to get. That's one of confusions that occurs: do they represent the indicators or the targets?
Then you have the business sector. What does the business sector define as benchmarks or benchmarking? The business sector has basically used it as "the best in the business." In the United States, at least, when they have used the term benchmarking, they have used it a little differently than either of the other two phrases. They have used it to mean that it's the business that is doing best or has the best practice. The state of Virginia did a recent study -- and I'll also leave it here for you to look at today -- which calls it "best-practice benchmarking." This means you look for best practices. That's another way of doing it. The dictionary definition does usually use the term something like a yardstick, so it really is the target that is the purest definition.
What I want to show you now are some of the options for benchmarking. I'm assuming here that we do mean that you want to compare. The key with benchmarks is what you want to compare your own performance to, to find out whether you're doing good or bad or better or worse. These are six types of benchmarks. The one that you'll all use is previous performance -- I mean, that's the standard one -- and it's fair to compare this year's performance to last year's. Last year's is a benchmark.
Similar units. You have an organization, and in each of the provinces, you do have multiple offices in some of your programs. You have multiple parks; you have multiple social service and mental health offices, many hospitals, etc. If you have those, you can compare different units. If you wanted to have benchmarks in excellence, what would you do? You would pick the best performing unit -- their performance on an indicator -- and use that as your benchmark. If you want to be more conservative, you might use the average overall for all the units as their benchmark. If you really think things are bad in your province, you might use the low end, so it should achieve at least the low-end targets. I'm hoping that in Canada you would go with the excellence version.
The third one is different client groups -- performance for different. . . . Say, males versus females: does a mental health program help males versus females? Does it help if there are different ethnic groups? In the United States, obviously, as you know, we have blacks, whites, Hispanics -- major groups. Normally in those cases we use the performance -- let's say educational performance -- of whites, and say we want to use that as the benchmark and try to bring the blacks and Hispanics up to that level of performance. That's a benchmark.
Performance of other jurisdictions -- this is very difficult. I'll mention a couple of cases of that in a moment. But the performance of other jurisdictions means that. . . . In this case, you have several provinces and territories. In fact, I would almost make a recommendation to you all -- possibly sponsored by the Public Accounts Committees -- that you have a task force of representatives to try to work out indicators that are common to the provinces.
There are two or three benchmarking-comparison interjurisdictional projects that I know of. Germany has one for local governments. The U.K. is doing them. Their Audit Commission has probably the most advanced one, which I mentioned earlier -- the Audit Commission's local authority performance indicators -- where they have been doing this and getting similar indicators with similar data collection procedures.
In the United States we have something called the local government large-city and county consortium, which I'm providing technical assistance to, which is about 40 of the largest cities. We have one city in Canada that's a member; Calgary is a member of the group. They're the Canadian representative among the 40-some-odd cities and counties. They have gotten together in a number of task forces: fire, police and a number of neighbourhood services and internal services. They identified a series of indicators and tried to define those indicators. I recommend that if you can pull this off at all in Canada, you try to do that similarly for provinces. If you do, let me point out that the real problem here is that there are major differences in definitions. If you don't do that. . . . Everybody is going to define them somewhat differently.
Teenage pregnancy -- I looked at the benchmarks in Oregon, Minnesota and Florida. They all use different age ranges. One uses 10 to 17, another 15 to 17, another 17 to 19. With exactly the same performance, you would expect different rates. I mean, 10 to 17 includes the ten-, 11-, 12-, 13- and 14-year-olds, where the rates should be much lower, so the aggregate will be much lower than the rate in the other jurisdictions that have 15 to 17 or 17 to 19.
You have to be very careful about the data collection procedures. Of course, every province will say: "We're different from every other one." That's true, you are different. But a lot of these indicators, you know, are good benchmarks. I suggest you try to combine them and try to see if you can't get some common indicators.
We very seldom have federal and state standards. I think you probably do have, in the area of environmental protection, air and water standards. I assume you have, like we do in the United States, so that you do have some targets or some standards against which to compare. There may be some others like that which you may have.
Finally, the item that we talked about, preset targets. That's the piece we've all talked about. Most systems do try, as the U.S. and England does, to talk about setting targets or having the agencies set targets, and then later compare actuals against the targets.
The first and last are very obvious ones that should be tried, but the others I do recommend that you consider, as well.
Let me hit performance-based budgeting. I know very little about this. It's a term that's gotten into the picture. The problem with performance-based budgeting is that we don't quite know what it means -- anybody. It somehow means that you consider outcomes, not just outputs, when you do budgeting. It is fairly easy, and it is standard practice, to relate output to dollars: the number of reports prepared, the number of roads repaired, clients seen, inmates in institutions. It's not so difficult, if you know the population that's served, to estimate the outputs you expect and the costs that are associated. That's what a lot of budgeting analysis is based on.
What we don't know today, in most cases, is the relationship between outputs and outcomes -- the end results. An obvious example is how much money it will take to increase the satisfaction level of citizens from 70 percent to 80 percent. We really don't know that, so it's much more difficult. One thing that is clear and that I think everybody is accepting is the fact that when you do budgeting and budget and fiscal analysis, you should consider, and you should be asking the agencies, what you expect to get in terms of outcomes. When they do put on targets, in a way they are doing that. So they are implying a relationship between resources and outcomes when they put targets on an outcome indicator.
[11:30]
I don't have much more to say about that. At the moment, we have a little bit of work going on in the United States to try to look at how you use the performance data in budgeting, but at the moment it is not a very clear-cut picture. Traditionally, budget analysts don't think in terms of outcomes, and it's very difficult to get them to do so.
Performance partnerships is probably my favourite topic, because I think it's the thing that is emerging. This is the new piece of business I want you to consider, and I want to help you make a point with me, if you would. In the benchmarks for Oregon, Minnesota and Florida, the priority benchmark they use is teenage pregnancy. In the United States this is a major social problem. I didn't see them in Alberta. Does that mean Alberta is okay on teenage pregnancy? Is that a problem?
A Voice: They don't have sex. [Laughter.]
H. Hatry: I want to ask you all: what agency in Alberta -- I'll start with Alberta -- is responsible for teenage pregnancy? Notice the way I put that.
M. Abdurahman: It's men. [Laughter.]
H. Hatry: I'm going to put that down. I think that's a good answer. What else? The women have nothing to do with it? What agency?
A Voice: Social Services.
H. Hatry: Social Services. What else? Any others?
A Voice: Health.
A Voice: Education.
H. Hatry: Anything else? Who else? Those are the only people who affect teenage pregnancy?
Some Voices: Economic Development.
H. Hatry: Economic Development. How is that? That's a new one; I've never heard that one before. Explain that to us, would you?
Interjections.
H. Hatry: Single-parent employment, okay. Well, I'll put it down. Any other agencies?
Interjections.
H. Hatry: All right. Are any other levels of government concerned with this?
A Voice: Justice.
H. Hatry: Is that part of the provincial government?
A Voice: Yes.
H. Hatry: Does the federal government have anything to do with this? How about local governments? Who else?
A Voice: Private agencies.
H. Hatry: Private agencies. Such as?
A Voice: Churches.
H. Hatry: Very good. I assume these are non-profit. Churches certainly have a lot to do with it. Who else?
A Voice: Volunteers, like food banks.
A Voice: Condom manufacturers.
H. Hatry: Volunteers. Condom manufacturers. I'm not going to put that down. I draw the line somewhere. But you also mentioned. . . . Let's call them boys and girls rather than. . . . We'll put men and women. It's obvious, if you start thinking about it and you really want to affect something, that you've got a whole set of people who affect teenage pregnancy. So why blame the social services, health or education agencies of the province? The schools, I assume, are in here, too; they play a major role with this, as well.
The state of Oregon, for example, has done a very good job of this because of the statewide indicators they've used that I mentioned earlier. The state of Oregon began to have a special task force, and they recognize this difference. It's a combination of state, local, church and non-profit activities, and they are forming a task force. When I talk about performance partnerships -- let me go back to that term again. . . . What we're dealing with is that you can have performance partnership between levels of government. In the United States, there are a number of efforts to have partnerships between the federal government and the states. The Oregon option that I think Craig mentioned last night is one of the main ones, one of the first ones that have come in. Agreements can be between different sectors; they can be within a sector between departments.
In the federal government in the United States we have a program called School to Work, which is basically trying to help youngsters leaving school to have the training and ability to find good employment. The Department of Labour and Department of Education have been meeting together. What are the components of a partnership? They meet together and jointly establish a mission and objectives. They should jointly identify the performance indicators and work out how to measure and track those and report them. They should also jointly decide who is responsible for what. If they're going to have a partnership, what does one group do and what does the other do?
Let me show you something from Multnomah County, Oregon, which is not too far from where we are now. You can't read this, but I just want to give you the flavour of what they did, because I think this was sort of a first. They also recognized. . . . These are agencies across the top -- children and families, health, aging, juvenile justice, etc. -- and down the line are their indicators -- in fact, teenage pregnancy, ten to 17, is up there first. They checked off the agencies that are responsible. They've recognized what that means is that these three agencies -- in this case juvenile justice, health, and children and families -- share a certain responsibility for this indicator. It's a very unique approach and philosophy. It's simple common sense, but it's there, and that's what they've done. They've taken all their indicators and said who has a significant role in that and displayed them.
I just picked up a partnership agreement between the state of Delaware's Department of Natural Resources and Environmental Control and the United States Environmental Protection Agency. Here is a 30- or 40-page document which starts off a type of performance agreement. The problem is that there are some dangers with these; that's very clear.
First, the upper-level governments tend to mandate everything. It's hard for an upper-level government to enter into a true partnership. If you're going to do these as partnerships, you're going to have to do them together. There are difficulties in developing the indicators and the measurements, but it's extremely important to try. I think that in the long run, particularly if you begin to include the customers in the partnerships, you really will be doing what's right. The whole notion of accountability is that the provinces have only a partial responsibility for these items. You share responsibility among everyone else.
Let me go to a cost for a minute. I'll just make a couple of quick points. Measurement is not free: you don't get something for nothing. Most agencies today try to use their existing data systems for performance measurement for all these things, and that's very limiting. Very few agencies in the past have had information on what happens to clients if they complete the program. For example, in a stop-smoking program, you can track at the end of the program whether the trainee or the client has stopped, but that doesn't tell you whether they will stop smoking. What you need to do is follow them up, let's say, a year later with a questionnaire, which is fairly straightforward to do. Without that extra step, you really won't know whether it has been successful. Most programs do involve information that government agencies, anywhere in the world, do not currently collect. So I want to caution you that that's the downside of it.
To get into performance measurement of outcomes, the agencies will need to do some modifications of their data collection procedures and add some new things. Even response times is somewhat new. Most agencies have not collected response times, because you have to track the time when the person came in and the time when the service was completed. You have to compare the two, come up with a time and add them all up. That's simple conceptually, and with modern data processing equipment this is all very feasible. But it does take effort, and the development effort in the first place can be particularly costly, because it takes a lot of time and effort to establish a good performance measurement system. It takes time and effort to develop a good customer survey. When you do customer surveys, you should also use focus groups to get the customers into them, so you know what questions to put in, what to ask about. So it does cost money.
The other thing that you all need to be very concerned about is training and technical assistance for the program managers of the agencies. The upper-level managers need some sort of exposure and training and assistance. But I also include the legislature and yourselves and your staff, because there needs to be some effort made -- and this costs money. I think outcomes are common sense. Really, when we talk about government and what it is supposed to do, it's common sense that you want to get results, and that's what government is for: to produce good health, good environment, good roads and good education. But it's amazing how difficult it is to get managers to think in terms of these outcomes and translate them into something that they can measure in a practical way. So you do need some training and technical assistance.
Oregon -- the example used by Craig -- used a very unique approach to save themselves money: they trained the first level. Normally you'll need some outside consultants -- and you've got many good ones in Canada -- who can do some training initially, and then you can use those resources to help train others in the province. These things are transferable across agencies. If you find some good managers or staff who are good in performance measurement in one agency, they can be helpful to other agencies as well. So I think you should definitely try to get that, and remember that a good performance measurement system is going to require initially some cost and effort and resources.
Let me go to the last topic. I want to just give you some other recommendations which I've put together. Actually, I've used them in the U.S. Congress, but I think they apply just as well to the province. I'll list eight. Some of these are fairly obvious -- at least, I hope they're obvious.
First, Public Accounts Committees should probably try to encourage other committees to do this, just as in the United States at the federal government level and in any state government. If the legislature does not ask the executive branch for outcome or good performance information, it's not likely to go terribly far. It's very important.
Second, they should definitely encourage service agencies themselves to collect and use the information to improve services. The key issue is improving services. That's why you're collecting all this information. If you know that some units are doing well and some units are not doing well, that should help that agency know where they need to pay some further attention.
The third item is that the legislature should request explanatory information on significant findings. That is, if you find an indicator where they haven't performed as well as you think they should -- as well as their targets -- press them for explanatory information of why that's the case. Use performance-based legislation. I know this is not for Public Accounts Committees, but you're also legislators in your own right, separate from your committee membership -- to the extent that the legislator should think in terms of preparing future legislation based partly on outcomes. It should focus on what you want to accomplish and make sure it does that. That's just common sense.
[11:45]
At the provincial and state levels the provinces should encourage the local governments -- the cities and municipalities -- to track performance and quality. In fact, a lot of the data that I think you'll find. . . . I know that in the United States much of the data the states need comes from the local governments, and much of the data the federal government needs comes from states and local governments. So you should encourage. . . . It really should be a joint effort.
The sixth item is the education point of view. I mention again that I think effort is needed to educate legislatures and their staffs on the limitations as well as the usefulness.
The seventh is the issue of quality. There's always the problem: once you're doing measurement, what's the quality of that data and how is it lanced? The legislatures should provide some way for encouraging some sort of quality control on the data. One thing I'll recommend this afternoon is that the legislative auditors probably be given that as one of their functions. It's not simply to do their own studies, but also to audit how accurately agencies are collecting data on independent outcomes.
The last item is something that's probably been debated here in Canada among the provinces, I gather: and I think that each province should consider an annual compendium co-report to the state's citizens. If you're really serious about all this, I think at some point -- I'm not saying tomorrow -- you should do that. The Alberta measuring-up is similar to that. It's a type of version. . . . I do not mean that you should preclude or exclude each agency's own annual report, but even those annual reports should focus more on outcomes, not just what seems to have. . . . When I look at annual reports from state and local agencies, I don't see much information. I see dollars and tasks and activities, but you never see what you're getting, and I think that's the switch we should do.
Let me just close with a little story. In Vancouver one of your billionaires was having a meeting. He threw a party for his associates at one of the big hotels. After dinner he asked them to come out to the swimming pool. In that swimming pool was an enormous crocodile that they imported from Australia. Not New South Wales, but they imported this from Australia -- Queensland, I think. The billionaire said to his guests: "I will give $1 million to anyone who will jump in that pool and swim to the other side." No one budged. Five minutes went by; no one budged. Finally they started going back into the dining room, and there was a splash. They all turned around. There was this one guy in the pool swimming like crazy for the other side. When they pulled him out he was tattered and a little bruised and bleeding, but he was in pretty good shape. The billionaire ran up to him with his cheque, signed his name and said: "That was the bravest thing I've ever seen. Here's your $1 million. Is there anything else I can do for you?" And he said: "Tell me who pushed me."
I don't want any blame for having pushed you into this. These types of efforts we talked about earlier are a bit like that swimming pool filled with alligators. There are just lots of pitfalls and dangers in there. There is no question about it. If you succeed in getting to the other side with good information, I think it will be worth it. Good luck to you all on that.
N. Reimer: I'm inquisitive with respect to the last transparency you had up. What happened to No. 7?
H. Hatry: That was an error. You caught that. There was no No. 7. You're not missing anything; it just wasn't picked up in the transparency. Is there anything else anyone wants to raise or comment on? Anything you hate about this stuff?
P. Mella: On the jurisdictions that have reports to citizens, I can see the function that it would serve in the long term, but I can see in the short term how the public would perceive it. People are very skeptical of things that come out from government trying to tell them what government is doing on their behalf, and it may be perceived as a propaganda initiative or whatever. How do you get people to the stage of wanting to take a personal interest in what their tax money is used for from where a lot of people are right now? How do you make the switch to having them become part of the process rather than looking on from the outside and saying that government is doing this to us instead of for us?
H. Hatry: That's a very difficult question, but first I would try to get the credibility of the reports. The credibility of any report was one of your issues, I think. To some extent, a legislative audit review will help. That is, if it has been basically looked at and examined by some legislative audit operation, I think it would help. In performance measurement, you can get away with manipulating the data one year, but it's kind of tough because, in the next year you're going to have more troubles. So it is difficult. There really is a problem with trust in government; today, we really do have that. If you have citizens who are completely distrustful, I don't think there is anything that can overcome it.
But the idea here is to get them in it gradually by having dialogue on those items and giving them confidence. I think that some of those indicators should be indicators of citizen performance as well. In the performance partnerships, the citizens play a role in government, and one of your messages as legislators should be that government can do just so much for you. You've got a role in all these services. With teenage pregnancy, I mean, my goodness: the males and females, the boys and girls, have a role in that. It's not just the governments that are doing it.
G. Pouliot: Thank you. I did appreciate your presentation immensely and certainly benefited from it, and therefore I thank you. I would appreciate your response to the following brief comments.
Polling and tracking by governments are done on a continuous basis. The positive side is that polling has many facets. Polling and tracking are conducted to build a database, to know where a society is -- where an electorate is, where the citizens are -- at any given time on any subject matter. Polling and tracking are also done to better serve democracy, or, as you near an election, to give you the ability to walk across the harbour when the writ is issued. They serve, for the politicians, two equally noble goals: that of getting elected or -- re-elected -- and also to have on the other side of the ledger a constant database, a better pulse to feel, that can exemplify when we need the tools to respond to the needs of people, the mandate.
Governments on the one hand will be very forthwith, very expedient in giving you their information -- sometimes for ulterior motives, but we shall close our eyes to these peccadillos or bagatelles, for we have closed our eyes so often. Yet on other matters you must go through the freedom-of-information process in order to get what is really run-of-the-mill information.
What I'm trying to say, and would welcome your comment on, is that in this art of the possible you have two. . . . They need not be opposed -- they certainly need not all the time -- nor always integrate. You have the building of the database -- genuine, bona fide information, a flow going back and forth, done by seeking what people want, where people are at. "Factoring in everything, where are we at on teenage pregnancy and, once we have the database, what do we do?" And on the other hand, you have the very political animals that we are -- not my colleagues. This is a very expedient, direct-action kind of honourable profession, the old trade that we exercise.
So how would you in your wisdom blend, merge or mesh the two approaches?
H. Hatry: I have to tell you a little story this reminds me of. This is about John Smith, who went into his doctor's office, and he had a series of tests. The doctor sat him down and said: "John, I have some very bad news for you. I've looked at all the 55 legislative audits on this health condition you have, at the randomized control experiments, at all the data. You only have one week to live." John just gasped. He sat back and said: "But, doctor, it would take me six months just to pay you, to pay the bill." The doctor looked at him and said: "Okay, you have six months to live."
The political nature does often differ. When I talk about surveys, of course, I'm talking here about surveys of experiences of people with specific services, retrospectively. Polling is used for multiple purposes. For the moment I'm talking about systematic polling or surveys. They should normally be done professionally. They have to be, to be credible. Basically, let's call them technical data that is the data for working out. . .as opposed to the political issues. Political judgments are always going to be there, and they're going to dominate -- no question.
The issue that you're raising is whether this new paradigm or new set of information on outcomes will enter enough into the discussion and debate to alter beneficially the decisions and choices of legislators in the executive branch. Knowing those for sure, we take on faith that if you have better information on what you've accomplished in the past, it will enter into the political judgments. So if you have these annual reports, it will be a little more difficult to base judgments strictly on politics if you have some data that goes contrary.
What will likely happen is there will be manipulation of the data. That's the worry we all have: that legislators will. . . . There's always the point of whether the bottle is half empty or half full. It's really often very difficult to tell whether a performance is good enough. I don't know if that answered your question, but it's a tough one.
C. Santos: One statistician said that what you cannot measure, you do not really know. The only problem is that all measures involve value preferences, and the formulator of the measure is, of course, built into the measure. Are there universal criteria or values that are at least value-free or value-neutral so that you can construct a measure that is not expanding or retracting like a ruler made of rubber?
H. Hatry: In other words, a stable. . . .
C. Santos: A fixed, universal measure in measuring performance -- can we do that or not?
H. Hatry: Performance information is always limited. It can never tell you what you should be doing. As elected officials, you have the major role and will always have that in deciding who wins and who loses. I don't care how good the analysts, evaluators or auditors are. All they can do is give you data and information, and you're going to have to make choices as to which groups are going to benefit and to what extent. Eventually, it's all value judgments. All that we're talking about here with all these procedures is some way to try to make that information better, and that's all it can do.
It's probably lunch time.
N. Reimer: Are there any further questions?
That will end the morning session, and I'd invite everyone down to the parliamentary dining room for lunch.
The meeting recessed at 12:01 p.m.
Copyright © 1999: Queen's Printer, Victoria, British Columbia, Canada