JOINT MEETING OF THE
CANADIAN COUNCIL OF PUBLIC ACCOUNTS COMMITTEES
AND THE CONFERENCE OF LEGISLATIVE AUDITORS
Report of Proceedings
(Hansard)
Tuesday, September 10, 1996
The meeting resumed at 1:07 p.m.
C. James: We're now at the juncture of the joint conferences where we have our usual joint session. As you know, we're holding the joint session in the chamber today. The proceedings of the Canadian Council of Public Accounts Committees are being recorded on videotape. I've been letting delegates know that a copy of the videotape and the transcript of proceedings will be sent to each jurisdiction. Also, a copy of the videotape of the entire proceedings of the CCPAC, including the joint session, will be sent to each auditor general's or provincial auditor's office across the country. If you require more copies of anything, you could let me know later on. It will probably take two to three weeks, once everything is edited and recorded and so on, before that occurs. We also have photographs that were taken yesterday of the Public Accounts group. At the break, you could approach the table and pick up a copy for yourself. Each delegate and anybody that's in the photograph is more than welcome to take a copy for themselves; I'll just set them at the head of the table here.
The joint session today is Inputs and Outcomes: the Art of Measuring Performance. The moderator is Mr. Fred Gingell, MLA, British Columbia, and also the president of the council this year. He is Chair of the Public Accounts Committee in British Columbia as well. The panellists are the Chair of the Public Accounts Committee from the House of Commons; Mr. Ray Chappell, member of the Public Accounts Committee in New South Wales; and Mr. Harry Hatry, director of public management programs at the Urban Institute in Washington, D.C. If they could approach the table, I'll remove myself, and Mr. Gingell will act as moderator for this session.
Where there are questions or comments, if you could please identify yourself and wait just a moment until the cameras are on you so that it can be recorded properly, we would very much appreciate that. Over to Mr. Gingell.
F. Gingell: Thank you, Craig, and welcome everyone to this joint session -- Inputs and Outcomes: the Art of Measuring Performance. I think we all recognize that in whatever organizations, whatever management changes we make, the key in the end is to have useful, relevant, meaningful information that can be attested to by independent third parties -- a critical part of the program and the move in all jurisdictions around the world to performance-based management systems.
To deal with this this afternoon, we are going to commence with three presenters. Each will make a presentation of somewhere around 15 minutes, and I'm sure you will all have lots of questions for them. After the presentations, the intention is to have as lively a discussion as we can on the issue of the art of measuring performance. First of all, we have Mr. Michel Guimond, Chair of the Public Accounts Committee of the House of Commons in Ottawa.
[1:15]
M. Guimond: Thank you.
F. Gingell: Before you start, for those who are using the translation earphones, if it goes off, just press the start button again.
M. Guimond: Bonjour à vous tous et à vous toutes. Avant toutes choses, je voudrais vous dire que je suis très heureux d'être parmi vous cet après-midi pour participer à cette session conjointe avec, en plus, des collègues membres des différents parlements, des collègues de la Conférence des vérificateurs législatifs. J'imagine que d'être en présence de tant de comptables et de fellows C.A. en même temps pourrait rendre n'importe qui nerveux sauf moi, parce que dans une vie antérieure récente j'ai travaillé deux ans au bureau de Caron, Bélanger, Ernst and Young à Québec, où j'étais un des rares non-comptables du groupe. Ayant une information en ressources humaines, j'ai travaillé avec des gens du milieu comptable.
Je constate aussi que je suis le seul représentant du gouvernement fédéral à cette table; d'autant plus que le programme parle d'un groupe d'experts, avec un collègue député australien, une sommité américaine, de m'avoir affublé de titre d'expert je ne sais pas si ça devrait me rassurer mais. . . . Je vais essayer de faire de mon mieux, parce que je dois vous informer que je suis un néophyte dans le domaine, puisque j'étais le président du Comité des comptes publiques à la Chambre des Communes il y a six mois seulement.
Par contre, à titre de député, je suis bien conscient de l'importance de mesurer le rendement dans la fonction publique. Je ne peux m'empêcher de dire que mes électeurs et mes collègues élus savent aussi que nos électeurs ne manquent pas de mesurer notre propre rendement en tant que députés lors des élections. C'est ce qui est beau en démocratie, même on a un collègue ici qui nous a dit hier qu'il y avait déjà eu quatre ministères, puis maintenant il fait partie d'un troisième parti d'opposition. Donc, c'est ce qui est beau en démocratie; c'est qu'on occupe certaines places dans les banquettes, puis on se retrouve près des rideaux, ou un peu plus loin.
Il y a beaucoup à dire sur le sujet de la mesure du rendement. Je voudrais vous entretenir plus spécifiquement de l'aspect information de ce concept. Il est reconnu par tous que les contraintes budgétaires étant ce qu'elles sont, les gouvernments sont amenés à faire des choix difficiles dans l'avenir. Il devra faire connaître, le gouvernment, ses priorités à la population. Seule une information de qualité lui permettra de prendre des décisions éclairées qui seront comprises plus facilement par la population. Le besoin d'information au sujet des programmes est donc plus criant que jamais.
Cependant, l'information traditionnelle ne suffit plus. De plus en plus, on exige des gouvernments qu'ils obtiennent de meilleurs résultats. Les contribuables, que ce soit au Canada ou ailleurs, veulent en avoir plus pour leur argent. Et plus pour leur argent, c'est les taxes et les impôts qu'ils payent. Et ils exigent que leur gouvernement lui démontre que c'est bien le cas. Combien de fois on a entendu les gens nous dire: "on est tané de payer, puis on veut en n'avoir pour notre argent." Le gouvernment, lui, demande aux gestionnaires de compiler l'information sur le rendement de leurs opérations, et de s'en servir. Pour ce faire, on devra développer de nouveaux outils, comme des indicateurs de rendement et des normes de service.
Il faut être bien conscient que cela constitue une tâche énorme pour les gestionnaires. Cependant, ils doivent relever ce défi puisqu'on se tournera vers eux, pour nous prouver que tel et tel programme est encore rentable.
Les parlemenaires auront aussi besoin de ce nouveau type d'information. Il est donc légitime pour nous de se demander: "est-ce que cette information est disponible?" Si oui, comment est-elle présentée? Sinon, que fait-on actuellement pour rendre cette information disponible? La réponse à ces questions constitue en fait la première partie de mon exposé. Je voudrais d'abord vous donner un aperçu des efforts qui sont faits actuellement au gouvernment fédéral afin d'améliorer l'information destinée aux parlementaires, surtout en ce qui concerne le rendement. J'exposerai ensuite les défis qui, à mon avis, devront être relevés d'une part et d'autres afin que ces efforts se traduisent par des résultats concrets.
In the second part of my talk, I will look at what the Public Accounts Committee might call its favourite topic: program evaluation. The committee believes that evaluation constitutes the ideal means of measuring the performance of government activity. At the request of our committee, the office of the auditor general took a look last May at the implementation by the Treasury Board secretariat of the recommendations by the committee in a report published in 1994. I will speak about the conclusions reached by the auditor general and his team.
Regardons maintenant l'information destinée au Parlement. Au début de 1995, le projet d'amélioration des rapports au Parlement a été lancé. Ce projet visait à trouver des moyens d'améliorer l'efficacité de l'information et de son mode de diffusion. On voulait améliorer la diffusion des résultats stratégiques et du rendement, ainsi que des données financières. Pour ce faire, on voulait inciter les gestionnaires à mettre l'accent sur les résultats. Cela devrait mener à une capacité de réédition de comptes accrus.
A peu près tous les groupes ayants un intérêt pour le sujet ont été consultés. Un groupe de travail parlementaire, présidé par M. Ronald Duhamel, député libéral de St. Boniface, au Manitoba, et formé uniquement de représentants libéraux. C'est ce qui est beau en démocratie: le gouvernment forme des groupes de travail. Nous, dans l'opposition, leur disons: "Vous vous privez de nos lumières. Vous vous privez de ce que l'opposition pourra vous dire." Mais le gouvernment a été élu pour gouverner, donc le gouvernement gouverne. Le gouvernment forme les groupes de travail, et l'opposition n'est pas représentée; de toute façon, je ne suis pas ici pour vous faire un débat là-dessus, mais c'est un constatation, j'imagine que vous en vivez dans chacune de vos législatures.
En 1995, le groupe parlementaire et le secrétariat du Conseil du trésor demandaient au Sous-Comité de l'étude de crédit d'adopter une motion en vue d'autoriser six ministères à déposer, à titre d'essai, une version révisée du budget des dépenses qu'on appelle la "Partie III" à Ottawa pour l'année financière 1996-1997. On se proposait ensuite d'évaluer l'impact de ces changements. Ces gens voulaient savoir si l'information contenue dans cette Partie III était meilleure, et si elle incitait les comités à s'y intéresser davantage.
Pour avoir consulté la Partie III du ministère des Transports -- parce que, comme je l'ai mentionné hier, dans un passé récent j'étais critique au niveau des Transports, et on avait eu l'occasion d'étudier cette Partie III au Comité des Transports -- je crois personnellement qu'ils sont sur la bonne voie, et ceci dit en faisant abstraction de toutes partisanerie politique. Dans l'ensemble, les commentaires recueillis ont été favorables. Cependant, on a signalé qu'il restait du travail à faire, en particulier pour ce qui est de l'information sur les résultats.
Cela n'est pas tout à fait surprenant. Il ne s'agit pas seulement de changer la présentation des rapports. Il faut modifier la culture organisationnelle au sein du gouvernement pour l'axer, non seulement sur les inputs, mais aussi sur les outputs ou les résultats.
Par exemple, jusqu'à tout récemment, l'Agence canadienne de développement international -- ce qu'on appelle l'ACDI -- n'était pas capable de mesurer les résultats de ses interventions. Elle ne pouvait pas dire, avec ses 2 000 000 $ qu'on a dépensé: "Nous avions comme objectif de faire diminuer le taux de mortalité infantile de 30 % dans tel et tel village d'Afrique. Les résultats montrent qu'ils ont diminué de 50 %, donc le projet est un succès." Par contre, l'agence pouvait savoir si les 2 000 000 $ avaient effectivement été dépensés.
Je vous ai donné cet exemple pour montrer que les ministères devront désormais se doter d'objectif précis s'ils veulent mesurer le rendement et en faire rapport.
Les prochaines étapes du projet d'amélioration des rapports au Parlement sont intéressantes. Le secrétariat du Conseil du trésor veut déposer à l'automne, à titre d'expérience, des rapports sur le rendement. Ces rapports décriront dans quelle mesure on a respecté les objectifs décrits dans la Partie III de l'année précédente. Il faut préciser que la Partie III contient déjà une section sur le rendement. La nouveauté, c'est que le document serait présenté six mois plus tôt, et qu'il contiendrait davantage d'information sur le rendement.
En présentant ces rapports à l'automne, on veut donner la possibilité au Comité de la Chambre des communes de faire des recommendations dans le cadre des consultations pré-budgétaires, en ayant de l'information sur le rendement passé des programmes.
The secretariat thinks that it will take three to four years before the new process can produce performance reports. Is this realistic? When the United States passed legislation on performance reports, all departments were given five years to adapt to the new system and to prepare satisfactory performance reports. It will be interesting to know how it went in Australia when they are very advanced with respect to information. I'm sure that Mr. Chappell may give us some more details about it.
What are we to think about this initiative to provide information emphasizing performance and results? A skeptic will say: "I will have to see it to believe it." What I will say is that the ball is perhaps in the parliamentarians' court. Allow me to explain. The project is concerned with the information provided to parliament and not the use that parliament makes of it. The best performance reports will serve no purpose if we do not use them. In the evaluation mentioned earlier, public servants apparently expressed doubts about using resources to prepare a separate document if it is not ultimately going to be used by parliamentarians. These concerns seem to echo those expressed by public servants in other public administrations.
According to Professor Thomas of the University of Manitoba, a study of these countries revealed that what people most disliked was that parliament hesitated or was unable to make extensive and effective use of the improved information provided to it. Personally, this gives me pause.
Je crois donc que nous, parlementaires, devrions faire un effort pour participer davantage au processus d'amélioration de l'information. Il serait facile de dire, lors de la publication des rapports sur le rendement, à l'automne, que le produit ne répond pas à nos attentes et de ne pas s'en servir.
Il faudrait plutôt examiner clairement ce à quoi on s'attend en terme d'indicateur, d'évaluation et de résultats. En faisant chacun un bout de chemin, nous ne pouvons qu'en sortir gagnants. Pour cela, il faut laisser la chance au coureur. Il faut, également, même si cela risque d'être difficile, ne pas chercher un coupable à tout prix lorsque les choses ne vont pas comme prévues.
Je ne veux pas dire par là que les fonctionnaires ne devraient pas être responsables de leurs actions. Par contre, je crois que ceux-ci, les fonctionnaires, ont parfois le droit de dire: "Nous avons mis ce project en place en vue d'obtenir tel ou tel résultat. Nous l'avons ensuite évalué, et il ne fonctionne pas comme prévu, pour telle et telle raison. Nous allons y mettre fin." Les fonctionnaires peuvent être réticents à fournir de l'information lorsqu'ils se croient mal parés. Cependant, ils révèleront plus facilement leurs erreurs si on ne leur tombe pas dessus à la première occasion.
Les députés qui sont dans la salle se demandent probablement dans quel camp je suis. Ils se disent qu'on ne devrait pas nous attribuer tous les torts, et que les fonctionnaires devraient rendre compte de leurs gestions. Ils ont parfaitement raison. Vous avez parfaitement raison, chers collègues.
Je me souviens que, lors de l'arrivée du Bloc québecois à Ottawa, notre parti voulait examiner les dépenses des ministères en détail pour y repérer des cas de gaspillages et demander aux gestionnaires de nous rendre des comptes. Lorsqu'est venu le temps d'examiner le budget des dépenses dans nos comités parlementaires respectifs, nous avons présenté plusieurs motions à cet effet, croyant que c'était là la tribune idéale pour débattre de ces questions.
Cette même année, des représentants du Reform Party ont proposé que soit réduit le budget de certains programmes. En tant que député, et particulièrement dans l'opposition -- l'opposition officielle de surcroît -- nous avons vite réalisé qu'il était quasi-illusoire d'influencer les plans de dépenses des ministères, et encore moins de les modifier. En effet, les pouvoirs traditionnels des comités, en droit parlementaire britannique, se limitent simplement à approuver, réduire ou rejeter le budget des dépenses qui leur est présenté. Il ne peut pas réaffecter l'argent, ni augmenter un crédit donné.
En plus, même si les comités parlementaires peuvent, en théorie, recommander des réductions de budget, une recommandation dans ce sens pourrait être interprétée par le gouvernment comme une motion de non-confiance. Cette question de la confiance ne relève pas de la procédure parlementaire, mais plutôt d'une convention politique dans le système de droit parlementaire britannique. Même si cela peut varier d'une législature à l'autre, les gouvernements présents et passés ont toujours considéré le rejet d'un article du budget comme remettant en cause leur confiance. Bref, le processus actuel est une source de frustration. Il n'est pas surprenant que l'étude de la Partie III ne soit pas une priorité.
In this connection, one might ask if a change in the process will encourage parliamentarians to make greater use of the information contained in part 3 of the estimates. If it would, a means should be found to promote interest in the accountability process. This is not an easy task. It is nonetheless the challenge taken up by the subcommittee on the business of supply, of which my colleague John Williams is a member.
[1:30]
This subcommittee has invited a number of parliamentarians, public servants and expert witnesses to come and offer their suggestions on reforming the estimates, as well as on ways of improving the processes and mechanisms used by the House and its committees to examine the budget and reach decisions. Opinions vary widely, ranging all the way from a greater effort on the part of parliamentarians, to a change in the existing committee structure. The subcommittee is expected to table its report in the fall.
Un des points intéressants qui est ressorti à quelques reprises lors de ces audiences est l'étude du budget de dépenses, même en version améliorée, qui ne permet de se pencher que sur 30 % des dépenses gouvernementales. En effet, 70 % des dépenses sont adoptées par voix législatives. Une des solutions souvent avancées pour examiner ces dépenses est une évaluation cyclique des programmes.
Deuxième et dernière partie de mon exposé: l'évaluation des programmes. Pour que les gestionnaires décident de mettre fin, ou de modifier un programme, il faut qu'ils aient l'information pertinente sur laquelle se baser. Dans certains cas, les indicateurs de performance, seuls, ne peuvent aider à prendre ces décisions. Je voudrais illustrer cela par un exemple: En 1984, le ministère des Transports du Canada a évalué le programme de subvention aux transports des marchandises dans la région atlantique. En se basant sur cette évaluation, et sur d'autres considérations, la sous-ministre adjoint des Transports a dit au comité qu'il était évident pour le gouvernment que ce programme ne répondait plus aux objectifs qui avait donné lieu à sa création, et donc qu'il n'était probablement plus nécessaire.
Par contre, j'estime que si l'évaluation avait porté sur l'efficacité opérationelle seulement, les résultats auraient été tout autres. Par exemple, supposant que l'on est à chercher à savoir si les clients étaient satisfaits, ou encore si le taux d'erreurs dans le paiement des subventions était bas; dans ce cas, le ministère aurait conclu que, sous cet angle, le programme est bien géré. Mais bien géré ne veut pas automatiquement dire efficace.
En donnant cet exemple, je veux souligner qu'il est primordial que les ministères évaluent leurs programmes du point de vue de leur pertinance, de même que de la réalisation des objectifs et de la rentabilité.
Le Comité des comptes publiques manifeste depuis longtemps son intérêt à l'égard de l'évaluation des programmes, apparamment depuis au delà de dix ans. Notre comité a réaffirmé sa confiance dans les avantages auxquels elle pouvait donné lieu dans un rapport que nous avons publié en novembre 1994. Afin de s'assurer que les engagements du gouvernment soient tenus, et que des progrès soient accomplis, le comité a demandé que le Bureau du vérificateur général fasse un suivi exhaustif au printemps 1996.
Voici quelques constatations que nous faisait le Vérificateur général du Canada, M. Denis Desautels. Je vous cite certains commentaires qui apparaissent dans son rapport: "Les politiques du Conseil du trésor concernant l'évaluation de programmes ont changé. Elles mentionnent d'autres sources d'information sur l'efficacité comme les vérifications des examens."
Un autre commentaire: "Même si, dans certains cas, les ministères ont pu évaluer les résultats des programmes clés, la plupart des services d'évaluation ont continué à mettre l'accent sur de petites composantes des programmes, et sur les besoins opérationels des gestionnaires."
Autre commentaire: "On fait des progrès pour ce qui est de l'évaluation des programmes de plus de 1 000 000 000 $. Les évaluations accordent encore peu d'importance à la question de savoir si les programmes sont toujours nécessaires, ou s'ils existaient des solutions de rechange plus rentables."
Ces conclusions semblent peut-être décevantes à prime abord. Cependant, on sent que certaines choses bougent, et notre comité n'est pas étranger à ces changements. Ça c'est dit en toute modestie, puisqu'en mon cas, très très très petit comme responsable des changements, je pourrais davantage souligner que M. Williams a fait partie de ce Comité des comptes publiques depuis l'élection de la Trente-cinquième Législature en 1993.
Le rapport du président du Conseil du trésor, un rapport annuel sur le rendement de l'évaluation, qui a été publié pour la première fois en novembre 1995, en est un exemple. Le Vérificateur général a jugé ce rapport comme un pas dans la bonne direction. Par contre, le rapport ne fournit toujours pas certains renseignements demandés par notre comité, comme de savoir quelles économies les études ont permis de réaliser. Je peux vous dire que le Comité des comptes publiques continuera à suivre de près cette question et mettra les efforts nécessaires pour faire avancer les choses.
En terminant, je veux dire que si le nouveau processus de gestion est prometteur, sa mise en oeuvre dépendra des changements dans l'attitude du gouvernement et des parlementaires. Elle dépendra également de la capacité des gestionnaires à nous fournir les nouveaux outils pour nous permettre d'accéder à une meilleure information. A l'art de pouvoir mesurer le rendement, il doit aussi exister l'art de vouloir s'en servir. Merci de votre attention.
F. Gingell: Thank you very much, Michel, for that most enlightening discussion on what's happening in Ottawa.
Next we have Mr. Ray Chappell, who is a Member of Parliament in New South Wales and Chairman of the New South Wales Public Accounts Committee.
R. Chappell: Thank you, Fred, and thank you, ladies and gentlemen. It's a great opportunity and a privilege to be here to participate in the conference and to be able to tell you a little bit about the Australian experience in this matter over the next few minutes.
I've only recently come back onto the Public Accounts Committee -- at my own request, I might say -- to grapple with some of the issues that we're talking about here through this conference. I spent a little bit of time on the Public Accounts Committee before going into the ministry, and graduating to the shadow ministry when we lost government early last year. But by my own choice, I came back onto the Public Accounts Committee. I think the work we have to do is very significant.
I want to start off by quoting Mr. Arnold Love, who is a past president of the Canadian Evaluation Society, when he was in Australia for the annual conference of the Australasian Evaluation Society -- New Zealand is included in that -- in 1994. He referred, first of all, to the great similarities between Canada and Australia, particularly the physical, political and economic environments in which we work. But he made a point of the key difference in Australia as compared to Canada that there is a greater centralism in the federal government in Australia. It has greater taxing powers relative to the states and has more direct responsibilities in a range of issues than has the Canadian federal government. In other words, it controls the chequebook. Therefore, to a large extent, it controls some of the things that we do at the state government level, and that makes a considerable difference when compared to Canada.
In the late 1980s the federal government started a drive towards program evaluation. It did this by way of complementing and supporting the budget allocation process. The idea was to focus more on results and outcomes by way of encouraging measurement of performance, establishment of service standards and new ways of reporting to Parliament. At the time, Mr. Love reported that the moves that had already been put in place in Australia put us at the forefront of performance management and, he said, at that stage, perhaps some five to ten years ahead of the United States. I think that might have been a bit of an overstatement, but certainly Australia was recognized as being right up there where it was all happening.
Because the reform process in Australia was being driven essentially by the federal Department of Finance, it tended to focus on all those quantifiable measures: dollars and numbers and inputs in relation to the outputs of programs. With the catch-cry of efficiency, economy and effectiveness, as you would expect, the primary emphasis was going on efficiency and economy. In New South Wales at the time, under a new government, we had another catch-cry, which was: "Let the managers manage." So as you can appreciate, the sentiment was more towards economic management in the classical sense, rather than dealing with measurement of outcomes and results in the hands of customers. Questions of effectiveness and appropriateness of policy were still some way off into the future.
Although all Australian states have a lot in common, especially in their commitment to reform and to the concept of accountability, it should be noted that there were a great number of differences, too, in the size and scope of the governments and the commitment, and therefore the pace at which the reforms were being introduced. In New South Wales, until recently, improved accountability was interpreted as better measurement and reporting of existing programs and activities. This led to increased demands on service delivery departments for reports of different kinds, and the demands were coming from different agencies. So not only the department's minister and the parliament were calling for reports, but the treasurer, the auditor general, the Premier's department, the director of equal employment opportunity, and so forth, were all calling for various reports. They were all calling for them for different reasons and according to different parameters, and therefore a whole lot of reports were being written; but a lot of the information wasn't necessarily being well coordinated.
The question that was also being asked at the same time was how much energy and how much expense was being incurred in preparing a whole lot of reports which weren't necessarily well interlinked and all pointed in the same direction. Obviously, if within the state that multiplicity of disconnected reports was being called for, we had to overlay on top of that the federal government also calling for a whole lot of reports. So there was a whole industry being established of reporting and accountability and so on that was costing a lot of effort but not necessarily producing quality outcomes for that effort. So to some extent the whole efficiency of the efficiency gains that we were seeking had to be called into question. A whole lot of effort was being diverted away from the core business of the various agencies of government.
It became clear that we needed to get a better grip on that. So the Public Accounts Committee became very active in that and made some fairly significant contribution, I think -- but I'll refer to that in a moment. At the federal government level we have an organization known as COAG -- the Council of Australian Governments -- and they meet regularly to discuss efficiency gains, new processes, reporting standards and so forth. They set out to develop a basket of performance indicators common to all states for specific activity areas. That's been a fairly arduous process because, as I've said, there are a lot of similarities, but there are also a lot of differences.
One thing that was important was not to so diminish the lowest common denominator of reporting which would satisfy all as to invalidate, in a sense, the value of the material you're producing and reporting upon. It was necessary to get a best fit which was still going to be of value to each of the levels of government and understandable between each of them, in a way that the federal government could in its own way report upon the relative performances of state governments when it was considering allocation of resources.
In New South Wales, we found it was of great value to make better use of existing documents -- corporate and strategic plans, business plans, performance agreements for senior executives and all that raft of reports, standards and so forth -- that were already in place. We had to ask the question whether we could make better use of those reports, and I think that was a worthwhile thing to do. The targets and undertakings in those various reports and agreements with senior executives and so forth were between, in that particular case, the senior executive, the responsible minister and the central government agency -- such as the Premier's department and Treasury. These could all be the basis for testing efficiency improvements over time, not necessarily just on an annual reporting basis, but perhaps quarterly and so on.
So there was a movement toward using existing reporting methodologies to gain some efficiencies. We thought that if this could be done properly, the one single report of the department could be used for a number of reasons. There would be consistency in it if we could just get the fit right between the various agencies within the government and between the various governments, so that at the federal level we had a very clear picture.
[1:45]
Those were the sorts of movements that were going on at that stage. I want to comment a little on the Public Accounts Committee's role in all that. In March 1996 we produced a report called Annual Reporting in the New South Wales Public Sector. Interestingly, if you'll refer back to the welcome we had at Government House the other night, the subtitle on this is The Truth, the Whole Truth and Nothing but the Truth, because I think that was the phrase used on that occasion. The Public Accounts Committee in New South Wales is of the view that there's a great deal of very useful performance information being produced in all the public sector agencies at any one time. The problem was really how to access that information in a meaningful and consistent way, and how to make sure that the information really became publicly available. It was obvious to us that a whole lot of the reporting that was going on was between the various government agencies and central agencies such as the Premier's department, Treasury and so on. It often wasn't getting through to parliament and, very often, it wasn't getting through to the community at large, certainly not in any way that was useful to the community. Internal reporting is all well and good, but if our responsibility is ultimately to the community out there, we need to make sure that the performance standards, the performance target-setting, the measurements, reporting and so forth are all available right through the whole process and not just kept as internal documents.
Over the last several years, a project known as program performance statements has been well developed, and that teased out a lot of the information that government needed. In fact, I have some figures here that by 1994 -- this is at the New South Wales state government level -- some 32 agencies were included in this project of program performance statements. They included some 210 programs, 800 outcomes and 1,350 outputs being measured. At this stage, only the regulatory and service delivery programs are being included in that project. All of them went to central agencies but not necessarily to Parliament and certainly not necessarily to the community.
However, the 1995-96 budget papers include some information from those program performance statements. This is quite a significant development, because it not only means that the performance information is now being made available to the public, but it makes the performance information subject to parliamentary scrutiny through the estimates committee process. Unfortunately, the degree of information provided still varies greatly between agencies. Most of the information takes the form of output measures; however, some outcome measures are now being included in a small number of agencies -- and, hopefully, a rapidly growing number of agencies.
The program performance statements project does not cover government trading enterprises, the corporatized bodies. The performance of GTEs is subject to detailed monitoring in a separate initiative under the auspices of the Treasury's GTE-monitoring unit. The Treasury set up a central unit to monitor the performance of those government agencies which had been corporatized. The financial performance of GTEs is measured against a range of performance targets, such as rate of return on assets and equity, which have been agreed between the chief executive of the GTE, Portfolio minister and the Treasurer. A summary of efficiency, service and financial information for each of the 20 largest GTEs is published each year by the Treasurer.
In 1991, the Special Premiers Conference, which is a coming together of all the state Premiers under the leadership of the Prime Minister, established a steering committee on national performance monitoring. The role of the steering committee is to establish a consistent set of performance indicators for GTEs and to publish these each year. Key financial ratios are published, together with a range of non-financial information relating to service quality, efficiency, pricing, safety and so on. Performance information about ten of the New South Wales GTEs was first published by the steering committee in April 1995.
I want to make a comment about auditing of performance indicators, particularly referencing the government of Western Australia, which has perhaps broken a lot of new ground in this way. In the Western Australia model, the auditor general has responsibility for providing an opinion on the relevance and appropriateness of performance indicators contained in annual reports. The Western Australia Treasurer's instruction 9.04 requires that all public sector agencies include in their annual reports key indicators of efficiency and effectiveness for each program. The auditor general is required to audit these performance indicators and form an opinion on whether the indicators are relevant and appropriate, having regard to their purpose and whether they fairly represent the indicated performance. By the end of 1994, the Western Australia auditor general was able to report that the quality of performance indicators being reported had improved to the extent that from 1995-96 unqualified or qualified audit opinions could be issued. Therefore Western Australia's auditor general is very actively involved, and I believe a similar situation obtains in New Zealand, where the auditor general has a similar responsibility.
Our auditor general in New South Wales, Tony Harris, submitted to our inquiry, which led to this report, that improving the reporting of performance indicators is "the single most important remaining means to improve accountability in the New South Wales government," and that the auditing of performance indicators would help to moderate the resistance to such reporting by agencies of government. That was very strongly recommended by our auditor general. It wasn't taken up by our government, and indeed, it was not actually supported by way of a recommendation from the Public Accounts Committee. The state treasury in New South Wales submitted that it would not be appropriate for the auditor general to be given a role in examining the relevance or appropriateness of performance indicators, as this was a matter to be determined between the government, ministers and the CEOs of agencies, and it was not part of the audit mandate.
There is still this tension going on in New South Wales in this matter, particularly when it comes down on this divide as to when you begin to get into policy issues rather than auditing in the traditionally accepted sense. So that question remains unsettled in New South Wales. We have a committee there formed by the present government, which has a role in looking at efficiency and effectiveness issues. It is intended at this stage that this committee will be the one that will provide some comment in terms of efficiencies, rather than having the auditor general do it.
I thought I might just finish by giving you some of the recommendations we had. We thought this matter of reviewing the annual reporting process and annual reports across all the agencies of the New South Wales state government was a way of trying to get a common focus on the appropriate supply of information to the parliament and to the community at large, and if the reporting was in a form and at a level that was appropriate to all of our needs, then this would be a better way of going about it. So I'll just finish off with those few recommendations.
The Public Accounts Committee reaffirmed in very strong terms that the parliament is the centre of accountability of the public sector. It's through its accountability to the parliament that the public sector is ultimately accountable to the people. It's a motherhood statement, I know, but nonetheless, I think we need occasionally to restate those things. The Public Accounts Committee therefore recommended that the performance information contained in the program performance statements and provided to the GTE-monitoring unit must be regarded as the minimum level of performance information to be included in annual reports to the parliament.
We recommended the establishment of ministerial customer councils, involving members of the public and representatives of key interest groups, for each New South Wales government agency. The role of these ministerial customer councils in respect of annual reports would include advising the minister on the performance indicators which should be set for each agency. The committee recommended that the New South Wales auditor general should be given a role slightly different from that in Western Australia, where the auditor general has that role of auditing performance indicators, mainly because we have that other body in existence.
We thought those matters of annual reporting were a significant way in which we could help to keep the matters clarified and keep the accountability processes advancing. That is a contribution our Public Accounts Committee in New South Wales has made. In each of the other states, I believe, there have been general movements in the same direction.
So, Fred, I'll leave it at that and be happy to answer questions later on. Once again, may I say what a privilege it has been to be here this weekend and to be part of your conference. Again, because several have asked, I remind those people about the Australian Council of Public Accounts Committees biennial conference in Sydney, a delightful spot, on February 10 and 11 next year. Anyone who would like to come, see us down at the back. We'd love to have you.
F. Gingell: Thank you very much, Ray. That was a most interesting view of what's been happening down under. It seems as though, as well as being down under, you may be well out there in front. I think all members of this body and conference found your remarks of great interest.
Next we have Mr. Harry Hatry, the director of public management programs with the Urban Institute in Washington, D.C., to bring us up to date with the state of the science in the United States.
H. Hatry: I will try to make some comments, suggestions and thoughts on measurement methods, drawing from a variety of experiences at the federal, state and local government levels of the United States. Fifteen minutes is not very much time to do much on measurement methods; it could take 15 years to do something like that. So I will be very selective, and I'm going to use the overheads.
First, I'd like to address the question of the type of auditing approaches that legislative auditors and Public Accounts Committees should consider. You're already fairly familiar with these. There are three basic types of roles that legislative auditors can take. One is to do your own evaluations. Most performance audits in the United States and the legislative audits here in Canada in the provinces are probably situations in which the legislative auditors do their own evaluations of program outcomes.
The second procedure, if you are fortunate enough, is that the agencies do evaluations themselves -- as I think Quebec indicated they are requesting. In the United States, at the federal government level, many federal agencies do sponsor evaluations. By the way, some of these program evaluations are mandated by the legislature, and they are very in-depth and often very costly. If agencies do undertake in-depth program evaluations, the legislative auditors certainly should look at those carefully, use those in their own deliberations and present a development of information.
The third, which is the new kid on the block, is to review agencies' outcome-monitoring procedures. I think this is something that is emerging as a major new role for legislative auditors throughout the world. As countries are beginning more and more to develop performance measurement systems with a special new focus on outcome information, this means that the legislators are going to need some sort of assurance that the information is halfway decent and will want to have their legislative auditors look at it. That could be a big task in the future for legislative auditors throughout the country.
The second point I would like to talk about is the issue of selecting the outcome indicators in the first place. When you have been in this business a long time, as I have been, they seem obvious sometimes and nothing more than common sense. But it is very clear that it's very difficult for program managers and even legislative staffs, in order to change their mode of thinking a bit, to become more oriented toward outcomes.
There are two types of procedures I'll mention here briefly that I would suggest auditors consider using. One is the use of focus groups. Focus groups is a jargon term for small groups, maybe eight to 12 people, usually -- a customer focus group for a service is what I'm going to be focusing on here. You may want to hold several of these throughout the province on the particular topic on which you call the group. You invite in a dozen citizens who are served by the service and basically ask them two questions: what do you like about the service, and what don't you like about the service? You listen to them for two hours in a comfortable setting, and if you listen well, you will find out a number of characteristics about the service that should be included in performance measurement systems. That's just one of many sources of information.
[2:00]
Another device that is becoming quite popular is something called a logic model. It's a fancy term. This label says this is an outcome sequence chart, which aims at helping organize one's thinking. This is useful for anyone to do. You pick a program and try to think through the various activities and steps. In this case, this is an example of a program which was initiated in Massachusetts to reduce pollution in the waters of Massachusetts from the hazardous wastes of businesses. They had initiated a program of technical assistance, and this is a logic model that was developed to indicate what they did. The technical assistance personnel from the state government basically visited the businesses and made a number of suggestions about how to reduce the use, the handling of hazardous wastes and about new processes in manufacturing.
This is basically an activity count. The number of people seen would be an activity count.
The next step you would expect to see is that the business firms would do something about it. They would have listened to what the state people said and would have made some changes. This is what we call a first-level outcome. The term that is fairly useful to use, one that is becoming used in the United States and elsewhere, is that this is an intermediate outcome. In other words, there is something that the customer -- in this case, the business firms -- has done but you still don't know what it leads to. If the businesses do change their processes and materials-handling methods, you would hope the next step would be that their pollutant discharges would be reduced, whether it's air pollution from the smokestacks, water pollution through pipes or whatever is put into the soils. A good indicator of that would be the estimated amount of pollutants discharged. That can be labelled an intermediate outcome, because you still don't know what's happened. It just indicates that the emissions probably decreased after they received the assistance of the state agency.
From that you then probably want what we might call an end-outcome. You would expect that the water quality would improve, if indeed your assistance has been significant and the discharges have been reduced significantly. Ultimately and later on, you would basically expect reduced fish and human disease, more use of recreational areas and all the good things that come out of the use of the waters. This outcome sequence is just a nice device for organizing your thinking. They are usually straightforward. People understand them. I do recommend that you consider using them.
On this issue of intermediate outcomes, just for illustration, I want to show some more intermediate outcomes which are very popular and, I think, of concern certainly to elected officials. These are immediate things, the first things, that customers are usually concerned about. These are various elements of the quality characteristics with which a service is delivered: timeliness, accuracy, pleasantness, convenience, accessibility of location, hours of operation and whether you can find somebody to talk to. This one has been added fairly recently. A lot of the programs I've talked to have pointed out that a lot of people, a lot of the customers, are not aware of the programs that they're supposed to receive, and therefore estimates of the awareness of citizens about the program services are very important, if it's only a bus schedule or whatever. The condition of the facilities that the customers use. . . . They come into a hospital or an institution of some sort. What sort of condition is that facility in? Overall customer satisfaction is the last item.
This as a checklist of things that you might consider important to customers. I do want to hasten to point out that these do not tell you that the nature of whatever the province then did led to anything good. It only tells you various quality characteristics that we can call intermediate outcomes; these are not end outcomes.
In the main thrust of the rest of my presentation, I talk a little bit about information collection procedures. Here are a number of procedures, some of which are very familiar and some of which are possibly not as familiar. These are basically a menu of the types of data collection procedures that you can use. The first, and the one that's used most often, almost invariably is the one that's used today throughout the world in all three levels of government. This is the use of data from agency records. If you keep track of timeliness of your services, you can keep track of response times by measuring the time the request came in from a customer until the time that they received the response. Recidivism rates in mental health services and social services can tell you something about whether the clients came back for service after they presumably were cured, or whatever the problem was.
The second item of trained observer ratings is not one that's obvious to many of you, but it's an interesting one, and it's becoming a little more popular these days. Here are just four examples of applications of trained observer procedures. They've been used extensively in health and safety inspections, some of the things that you're probably already doing in hospitals and long term care institutions in your provinces. You certainly want ratings of the facility's safety. In many institutions that provide service, you can do trained observer ratings of the quality of service. In one project, I went into the state of Delaware and sampled the food in all of the state's prison facilities. That was a rather distasteful job. I had one of my best sandwiches there, a peanut butter sandwich, which was put out by the women's institution, by the way, but the rest of the food was pretty bad. They were thinking of contracting out the food, and that was one reason for the session.
On the condition of institutionalized clients, ratings of the condition of disabled clients use some sort of activities-of-daily-living scales whereby a trained observer sees how well the individual does in grooming themselves -- their sanitary condition -- feeding themselves and dressing themselves. So there really are a number of cases where trained observer procedures are likely to be useful as long as you have systematic rating scales. One thing you have to be sure of with these -- I don't have time to go into this very much -- is that you have to have a fairly explicit, systematic rating scale so different inspectors will give the same rating to the same condition at different points in time. There is some very interesting work that's been done in photographic rating scales.
Items three and four are two forms of surveys. Personally, I think that surveys are probably one of the major emerging procedures that are inevitable, if you really mean to measure and get feedback on clients, and I don't mean only satisfaction levels. People don't realize that you can get lots of factual data from client surveys, and I'm sure you do that nationally in the federal government here, as well as in the United States. You can get information on employment -- and unemployment comes from that -- and on various health characteristics of the population, such as whether they stopped smoking. If you have a stop-smoking program, you have feedback to find out whether indeed a year later they indicate that they have stopped smoking. So these are very important devices.
I've divided them into two items: client surveys and household surveys. The household survey is intended, as defined here, to be a survey of the general random sample of households in your population. Most of the polls that you're aware of are household surveys: polls of voters ahead of time, and most of the polls that Harris and Gallup and all the polling firms and universities do. However, for individual service programs, the client survey is probably going to be much more useful. Much of the work of your audits on individual programs should be client surveys. These are surveys of the clients of a particular service. Therefore they should be easier and less expensive to do than a household survey, because in general you should know who the clients are. You probably know their addresses and their telephone numbers, and you can reach them fairly carefully. Also, the response rates from client surveys tend to be much higher than the response rates from household surveys, because they're familiar with what you're asking about and they have a self-interest in that topic. So I would recommend that you think in terms of client surveys.
Both of these surveys are very credible. As elected officials, you should be very attracted to surveys of clients -- of course, as long as you think that they'll come out okay. But basically, if you're trying to do an independent evaluation of those services, you can hold yourself at distance from that data. So I certainly recommend client surveys and household surveys as a major tool.
A role-playing test is a little different. Now, auditors particularly should consider this on occasion. In the United States, these are primarily done by the Departments of Labour and Housing, and Urban Development on programs such as equal employment opportunity and equal housing opportunities. In the case of those types of role-playing, they're fairly elaborate activities. What will happen is that the auditors will actually select pairs of people to role-play. They'll be actors -- one will be a minority person and the other will be a white Caucasian person. They will both be trained a little in how to process an application either for a house or a job. Then they will go to answer at random a selection of advertisements for jobs, employment or for homes, and see what happens. They will keep close records of how far they progressed and what resulted. That's becoming fairly popular.
The other version of this, the much less expensive version which is applied to some services, is one in which the role players would perhaps call an agency by telephone or contact them by mail to get some service. An obvious case would be tourism services, which has been done, in which you simply write for information, see how long it takes to get that information and whether it's helpful or not. So it's any service that sort of lends itself to somebody calling.
There is a problem with this, and there have been occasionally some questions raised as to whether this is correct, that a government employee or government contract employee will be acting -- falsifying information by claiming they're something that they're not. So there's a bit of a problem there. For the most part, in the United States that has not been a major problem to date; but it could be.
Item six, physical measurement equipment, is basically things like roughometers to measure vertical displacement for road roughness, or air and water pollution equipment. It's not often used, but when it's used it's important.
The last one, systematic use of expert judgment, is sort of a last-ditch stand if you've nothing else to do -- if you're doing some development or research programs, you might have experts write the system.
Let me do just one more thing for you, because this is extraordinarily important for you to think about when you look at data and analyze information, whether you're an auditor or an elected official. We're the victims of aggregationitis in the world: everyone likes to see data aggregated.
Here are two units. This is derived data, which shows that each of them have 500 clients during this year. Unit one had 300 successes and unit two had 235 -- total health. By that I mean that their condition actually improved substantially. So unit one has an overall rate of 60 percent, versus 47 percent. There's a problem with this: it's awfully highly aggregated. Much of the information that's going to be useful to both sides, to both of your groups, is this aggregated data broken out by type of client -- male, female, age group, geographical location, which unit.
But there's another one called client difficulty. I don't know of any service where the incoming workload is homogenous as to its difficulty. For almost every service, the workload that comes in has different degrees of difficulty in bringing outcomes -- even roads: the amount of traffic, particularly truck traffic, the amount and nature of the soil conditions and weather conditions can greatly affect the ability to keep those roads in good shape. The point I'm making here is to look below; look beneath the aggregate data.
[2:15]
If you break this up into difficult and non-difficult cases, you will see what I'm talking about. Here the 500 patients are broken up. Notice that unit two has a higher success rate for difficult cases. It also has a higher rate for non-difficult cases. Is that possible? Did I goof with the numbers? No, the numbers are correct. What happened here? The mix of the clients is different in the two units. Unit two has a much higher percentage of difficult cases. Therefore in aggregate they look worse, whereas when broken out they look better. So I urge you not to take the first data you see at full faith but to look at it more closely.
I'll close at this point and as a last comment just make the comment that one thing auditors need to be careful about is excessive precision in the data. I think one of the big problems we have in measurement today is excessive precision on surveys and what have you. My message here is that it's better to be roughly right than precisely ignorant.
G. Morfitt: Ladies and gentlemen, I'm Fred Gingell before he became an MLA and his hair turned white and he started wearing bow ties. I'm filling in for Fred for a few minutes.
Thank you very much, Harry, for that presentation. Fred has had to go out. It seems there's some business about a balanced budget not being balanced. No, it couldn't be, not in this jurisdiction. However, we'll find out more about that perhaps later in the coffee break. Fred should be back shortly, I hope.
We've heard from three knowledgable presenters this afternoon. I was interested in Ray Chappell's mention of what he called COAG. You've got to be terribly careful about acronyms. COAG to me sounds a lot like coagulation, and I'm sure that was the exact opposite of what that group was trying to portray. Of course, in British Columbia here we had the good old BS fund, meaning the budget stabilization fund. But it got called the BS fund, and it didn't last very long after it was given that.
We've got time for questions and information-sharing, and I'd welcome questions from all of you. Or if you've got some observations on some of the comments that were made, we'd be pleased to have them shared.
G. Pouliot: M. le président.
G. Morfitt: Just give your name and where you're from, perhaps, for all of us.
G. Pouliot: Mr. Speaker, I will readily acquiesce. . . . My name is Gilles Pouliot, and I am the representative for the riding of Lake Nipigon in northwestern Ontario. With your kindness, I'd like to direct my question to M. Michel Guimond, and I will, with the courtesy being extended and all the appreciation and the good deeds out there, Mr. Speaker, have a supplementary. I will prelude the supplementary by way of comment, if it's okay with you.
M. Guimond, en vous remerciant de votre participation, moi, je me suis demandé, comme politicien dans le contexte de notre système parlementaire si, avant d'en arriver à une meilleure évaluation des résultats, et pour y arriver en respectant l'échéancier, n'est-il pas, au préalable, préférable d'évoluer à l'intérieure d'un système uniform, tant qu'en dedans des jurisdictions, et à travers et au delà des provinces, dans un climat d'entente avec le gouvernement central.
Laissé grosso modo, en votre exemple vous expliquez en bref ce qui se produit. Chez vous, aux gens du Québec, vous n'avez pas présentement de Comité des comptes publiques comme tel. Par ailleurs, au Manitoba, les gens nous ont informé que le ministre des Finances siégeait au sein du Comité des comptes publiques. Je sais que chez nous, en Ontario, les ministères n'ont pas le même système de comptabilité. La compétition entre les ministères est présente; elle est palpable; elle est concrète. Ne serait-il pas préférable, avant qu'on se prononce sur les bienfaits d'un échéancier pour y arriver, qu'on évolue à travers le même jeu de cartes, qu'on joue tous la même partie.
Et j'aurais la supplémentaire, M. le président, tout de suite après.
M. Guimond: C'est bien entendu que les compositions des comités de comptes publiques varient selon chacune des législatures. Je ne peux pas répondre, moi, pour les délégués du Québec, si le système actuel est fonctionnel, opérationnel. Je pense que M. Baril hier, dans ses commentaires, mentionnait qu'il était dans l'intention de ce gouvernement provincial d'avoir un comité permanent des comptes publiques. Puis, moi, je peux vous dire, ayant côtoyé M. Bouchard depuis deux ans et demi avant son arrivée, je pense que ça s'insère, en tout cas dans ses valeurs à lui, que ça serait important d'avoir un comité spécifique. Je ne veux pas répondre pour lui, mais que ça s'insère d'avoir un comité spécifique, qui se pencherait sur la question.
Je pourrais donner l'exemple de notre Comité des comptes publiques à Ottawa. Il est présidé par l'opposition officielle, mais les pouvoirs qu'on a s'arrêtent là, parce qu'il y a deux députés du Bloc, deux députés du Reform, puis il y a sept députés libéraux. Mon collègue M. Paradis. . . . Il y a des règles de procédures -- on ne peut pas parler des absents en Chambre, mais je veux dire. . . . Mon collègue M. Paradis avait un avion à prendre pour trois heures en raison d'autres engagements, mais il sera en mesure de corroborer ce que je dis là, c'est que le système parlementaire britannique fait en sorte qu'un gouvernement. . . . Comme je disais tout à l'heure: le gouvernement gouverne. Puis le gouvernement est majoritaire, et puis c'est correct comme ça. Et il est majoritaire au sein du comité. Mais tout est dans la question d'une notion de non-confiance. Si on pouvait avoir un vrai Comité des comptes publiques, présidé et majoritairement avec des membres de l'opposition, mais que le gouvernement ne pouvait pas être défait sur une motion de non-confiance qui viendrait de ce comité-là, peut-être qu'on serait plus en confiance d'avoir un Comité des comptes publiques comme chien de garde des dépenses gouvernementales, composé majoritairement de représentants de l'opposition. Mais c'est vrai qu'il y a encore du chemin à faire, encore des législatures. Mais je pense que chacun évolue à son rhythme dans ce domain. Il faut respecter. . . . Apparamment ils nous disent que le Canada, c'est l'unité dans la diversité. Permettez-moi d'en douter, là. Mais apparamment c'était une des devises lors du centenaire de la Confédération en 1967.
G. Pouliot: Mr. Speaker, we have the opportunity, one more time, by way of the human dimension, for the pleasure of renewing old friendships, and you meet new friends and you share information in a climate of comfort. In this case, it's enchanting indeed, given the surroundings, so it's conducive to good order.
In our search for a better way, and in keeping up with the quickly changing times, we're constantly being reminded that although we're heading in the same direction -- we have, or we profess to have, the same interests; we share the same shortcomings, pitfalls and, of course, benefits. . . . Yet, on the one side, we're guardians of our own parish, and we cherish those capacities always with sincerity, sometimes with passion. When we see a search not for autonomy -- that's established. . . . But to deliver the goods, you must be given the tools to do so.
I've heard, though I know it was said matter-of-factly, candidly. . .to conduct the affairs of the state like a business; this should be one of the lightning rods, the guiding line. As a politician, I can tell you, sir, and you've guessed it, it makes me a little nervous. It doesn't mean I'm right. I've been wrong so often. But at times I have had some difficulty with the thin line, the nuance, between accountability like a business and political responsibility when it comes to supply in our great constitutional monarchy. Supply is meant to inform the people -- but I only have one channel at home and have to watch the parliamentary channel -- how great the people of Lake Nipigon, Hudson Bay, Lake Superior, etc. . . . Then supply gets passed. I guess there's a motion, and it gets passed, and then we go to estimates. Well, estimates are meant to let you carry the guilt because it becomes you; you're the government. My job is to define for you, with the aid of Webster's and Oxford the true definition of what this system has become. Then we will pass estimates, too. So any interference to satisfy the needs of the system makes some politicos quite nervous. Value for money is not the conscience of government. It is accountability of government. If the government of the day -- and I will conclude now, sir -- wishes to spend whatever is in the purse, and if they see fit to give it to Miss Jones who is never going to show a profit, because she's marginalized -- we're all on the waiting list, but she's there already -- if they wish to do that, value for money is indeed in here. But accountability. . . . Once Miss Jones was the recipient, the system went a little awry. More so, it's not the conscience of the people. Politicians can do that, and heaven knows the turnover is big time all the time, and you'll be up again.
A better system of accountability, that's what I'm taking back to Ontario, and we work well -- we come as close as possible to work without too much partisanship. But given our frailty sometimes, we feel provoked rather easily. So we do our best, and we want to go beyond. In fact, we've indicated that we will go to the transfer agencies when the money flows from the province to the MUSH sector -- school boards, hospitals -- that they too should be accountable. So I've learned a lot, and I've done so because of your good office, and I thank you, Chair.
[2:30]
G. Morfitt: Thank you for those remarks. Any response to those remarks or additional comments that people would like to make?
J. Chagnon: J'aimerais faire une mise au point sur les propos que M. Pouliot a mentionné tout à l'heure concernant le manque de Comité des comptes publiques au Québec. En fait, au Québec, nous avons huit commissions parlementaires qui touchent à tous les aspects de l'administration, sans spécialiser. Un peu ce que le député -- et modérateur, en votre absence, M. le président -- M. Fred Gingell nous expliquait hier que la Colombie-Britannique cherchait à faire, cherchait à spécialiser ses commissions parlementaires, ou à développer un principe des commissions parlementaires spécialisées sous des sujets tels la santé, l'éducation, les transports, la justice -- des choses comme ça.
Nous avons déjà au Québec ce processus-là, et les crédits des ministères et les engagements financiers de chacun des ministères sont étudiés dans chacune de ces commissions-là. Nous souhaitons remettre en place, comme il y a déjà eu dans le passé, une Commission des comptes publiques. C'est ce qu'a indiqué le président de la Commission du budget de l'administration, mon collègue d'Arthabaska, parce que nous croyons qu'on a, avec le temps, avec huit commissions parlementaires, rendu diffus l'opération de l'étude des crédits, mais particulièrement des engagements financiers des différents ministères. Rendu diffus dans huit commissions parlementaires, et nous pensons que si on avait une commission qui ne faisait que ce travail, nous serions encore plus efficace. Je ne voudrais pas qu'aucun collègue ici croit que l'opération de la vérification des comptes publiques ne se fait pas au Québec, ou ne se fait pas parce qu'on n'a pas de commission des comptes publiques.
Je voulais juste clarifier ce point-là, et aussi poser une question, maintenant que la clarification a été faite, M. le Président, particulièrement à M. Chappell. Un peu plus tôt, M. Hatry a mentionné qu'on utilisait, soit des focus groups, soit des sondages, pour déterminer jusqu'à quel point le public peut estimer être satisfait d'un service, ou de services. Je sais que nous utilisons parfois ce processus-là dans certains ministères. Mais en Australie, chez le Nouveau Pays de Galles, on l'utilise aussi; est-ce que la commission, est-ce que les parlementaires utilisent ce modèle de fonctionnement pour vérifier le niveau de satisfaction des contribuables? Est-ce que c'est comme l'a suggéré le spécialiste M. Hatry, un moyen que vous utilisez chez vous?
Et je voudrais, M. le Président, en terminant, au nom de la délégation du Québec, vous remercier et remercier M. Gingell de l'hospitalité que nous avons eu ici depuis trois jours avec vous en Colombie-Britannique. Nous avons apprécié cette chaude hospitalité et l'acceuil que vous nous avez servi tout au long de ces trois jours, et la qualité de l'organisation que vous avez su monter. Vous pouvez transmettre ces salutations et ces remerciements à M. Gingell, et à tous les gens qui ont contribué à faire de cette réunion le succès que nous y voyons.
R. Chappell: I guess there are two different levels on which we could answer the question of whether, in New South Wales, we use customer focus groups and so forth. It would be true of many of our Members of Parliament that they use databases, which they maintain in their constituency offices, of different interest groups to whom they can direct a survey on a particular policy issue and ask: "Are you happy with this?" or "What's your view?" Some people would do that more effectively and more often than others, because some people wouldn't do it at all, so it depends on the individual.
More importantly, at the government level, a previous Premier who introduced a lot of management reforms was intent on developing some customer councils, where information was to be gained by way of bringing people in on a regular basis so that they would be informed of the process and the particular policy area, so that they would be able to contribute to the goals or the outcomes that were being set and the ways in which those outcomes could be measured. That didn't quite happen, but some of the departments are now using some of their key groups. In environmental policy issues, for instance, some of the departments have key people who regularly come from the interest groups to discuss with them the outcomes that might be achievable over the next program period, be it one year, three years or whatever. There's a fair amount of interaction at that level for some agencies, but not all. I think we've got a long way to go, but there is certainly a recognition that, at the end of the day, we need to find ways of ensuring that we are satisfying our clients.
I think that will come as a further development beyond what we are presently embarked on, which is a better information reporting mechanism in the ways I've described before. If we get the reporting better and if we get the general public and Members of Parliament used to going through the information and actually seeing what is going on, I'm sure there will be a more ready group of people, both parliamentarians and the general community, who will want to participate in the process. No one is ever going to design the system right, once and up front. It all does evolve.
To come back to a question that was posed before: is five years a reasonable period? I think it very much depends on the program. If anything, we should set the time frames shorter rather than longer to press the point and to get those changes flowing. If I can give just one example, New South Wales decided some years ago to implement full accrual accounting. We realized at the time that there was going to be a massive job of work. We didn't realize how massive, so we embarked on it. We set it in train, and now it is substantially completed. If we had sat around until we designed the perfect way of introducing accrual accounting, we would still be trying to find the perfect way. With all those things, somewhere along the line you have to decide, even though it may not yet be perfect. Take the first step and work with it.
Interjection.
H. Hatry: It was about New South Wales, so it was appropriate.
The one point I would make is that you are probably already using citizen survey information in your deliberations, if not for the customer satisfaction elements, which are new, but because of the data on other things such as unemployment, employment and health conditions. Many of these things are not really that new. I'm sure your government, as are all here, is doing some survey work or making use of survey work relevant to your own province.
M. Abdurahman: Firstly, I'd also like to join my colleagues from Quebec in saying a sincere thanks for this most informative conference. I'm very pleased to be here.
My question is: in New South Wales, in the health care system or the justice system, do you have outcome measurements in your programs -- for example, so you could look at the outcomes of cardiology? Or if you're looking at someone who has a pedophile sentence, in a treatment program, would you know what the results of that treatment program are?
R. Chappell: The answer would have to be yes, but I'd have to qualify that by saying the quality of the information and its timeliness and so forth could be better yet. That has to do. . . . With health in particular, I think you would find fairly ready access to information which was very accurate but maybe two years old before you could access it.
On the other hand, we in the Public Accounts Committee have just been doing a study of, if you like, customer satisfaction in the justice system. We've found it very difficult to access information. We in fact subtitled the report with the sentiment that the only person who was given no consideration whatsoever in the whole court process was the person it was being inflicted upon. The judges, the staff -- the whole process -- the advocates, the lawyers, everyone was being taken notice of except the poor client. So in terms of client satisfaction in that department, we came up with a very clear and simple answer: no satisfaction at all. We believe and have recommended that there clearly needs to be a refocusing of attention on the needs of those people who are being processed through the courts. I think the courts themselves, the Department of Justice, have clearly recognized that need to become much more customer-focused than they have been.
So it varies from department to department. Some are very good, and others are on a steep curve.
M. Abdurahman: Supplementary, please. In the budget process, is there allowance now for outcome measurements?
R. Chappell: Yes, they've been introduced. They need yet to be more fully developed, but outcomes measurements are now beginning to appear in budget papers. That will enable us as Members of Parliament to track those over time and to include much more rigorous debate on those issues in the estimates committee stage. So -- not only in annual reports as tabled but also in estimates committees -- because these things will be included in the budgetary process, we will have much better access to information and much more opportunity to improve the performance of government.
G. Morfitt: Perhaps I could raise one issue. I don't know if it's been discussed by the MLAs and MPs here or not, but it has been said that getting in the way of open and accountable government -- talking about results and the lack thereof -- are ministerial accountability and the fact that if managers are given the right to manage and some room to make mistakes, ministerial accountability immediately gets in the way. As soon as there is a mistake and the minister is held accountable by the press or the public, the clamps are put back on, and you don't get that openness again. I don't know if that's been discussed at all, but that's a subject of interest to the auditors.
Anyway, are there other, either that subject. . .? Ian, you were going to comment.
I. Thackeray: My question would be addressed primarily to Mr. Chappell, but I'd be very interested if the other panelists had a reaction on that. I was wondering if, in New South Wales or elsewhere, you've succeeded yet in establishing a link between the availability of good performance information or indicators and the subsequent resource allocation or budgetary processes. If you haven't succeeded in doing that yet, and I think it's quite rare that it has been done well, do you have plans or do you think it's possible?
R. Chappell: I certainly think it's possible, and we're moving in that direction. Some agencies would perhaps already claim they are well along the path of having better information than they have ever had before, with better outcomes and measurements, and that they are therefore able to better direct their resources at budget time to internal programs in the agencies. As in all cases, I think the answers you would be given would be perhaps a little optimistic, but there is a recognition that if government is going to be about serving the needs of the community in the best and most efficient way, those factors must be taken account of in terms of development of programs in the first place and then in terms of the allocation of resources to them at budgetary time. I wouldn't say that we are anywhere near the way that would satisfy any of us yet, but we are certainly pointed in that direction.
H. Hatry: There is little question that the use of performance information is currently the key weakness today throughout the world at all three levels of government. Managers have not seen the information and legislators don't often have it -- or, when they do, they don't have much experience and don't know how to use it.
At the lower levels, there are two examples. One was in Minnesota, where the trade office did at least alter its resources between the metropolitan cities of Minneapolis-St. Paul versus the outlying areas. They did alter their resource allocations. Another classic example of how you can ultimately use the information is that of New York City and its solid waste collection, which has used trained-observer approaches to regulate it all, using a photographic rating scale to rate the cleanliness of its streets throughout the whole city. It has started to do that on a monthly basis, and it actually allocates its collection crews, based on its findings from previous data, on the basis of where the streets are clean -- well, cleaner. New York City is not clean, but. . . . That's a classic example, and a much more detailed example than I'm sure you meant. It is spotty at the moment.
[2:45]
J. Leefe: I would like to spend a moment reflecting on the conundrum you rolled out across the floor just a moment ago respecting ministerial accountability, on the one hand, and allowing the managers to manage, on the other. Indeed, I think it is a conundrum. I have listened over the past few days in this conference and listened in others, and we are inclined to try to grapple with the general accountability of government using measurements that were essentially designed to fit the private sector rather than the public sector.
This may be an oversimplification, but it will do the job here. It strikes me that, with a private sector company, the shareholders are quite content provided that their investment is well-husbanded, their stock value is increasing and they are being paid reasonable dividends. I don't think that this kind of measurement necessarily suits itself well to measuring governmental outcomes, because government essentially provides a very different kind of return on the taxpayers' investment than is the case with an investment in Bell Telephone or something of that nature.
The great challenge to us as legislators and to us more particularly as members of Public Accounts Committees, and also to auditors general, is to try to define the balance which allows managers to manage but at the same time ensures -- absolutely ensures -- the integrity of the parliamentary system, whereby ministers are fully accountable for what happens or what doesn't happen in their departments. The bottom line for me and probably for most of my colleagues in Nova Scotia -- perhaps all of them, and I hope all of them -- and for many colleagues I've met across the country, is essentially this: if we either advertently or inadvertently lose sight of the fact that we, who have been elected and entrusted with the greatest public responsibility of all, move away from the accountability of our office, be it as a member or in the case of management as a minister, then the entire democratic system will suffer. Whatever credibility it may have now will decline as a consequence of that. That's the greatest challenge that faces all of us. I wish I had an answer to it; I'm sure all of us do. Perhaps some of you do have one. If you do, please share it with me.
J. Williams: I found the presentations in the last two days to be very interesting, enlightening and informative. I think most jurisdictions in the western world are grappling with how to come to terms with doing more for less and all the various buzzwords that are used today, to try and ensure that the taxpayer gets his value for money and that the services are delivered as best we can. But one of the things I didn't hear as we've tried to grapple with how governments can deliver these services is the comparison between the government delivering services and the private sector delivering the services which it delivers in a capitalistic society. I'm certainly an unabashed capitalist and a believer in the capitalistic, competitive system, where competition is by far the best model to ensure survival and better than survival -- to ensure that if you provide the best service, best quality and best product, then there's no question that you are going to survive and do well in the private sector. We don't have that in the public sector, by the very fact that it's a monopolistic thing.
One of the fundamental differences between the private sector and government is that in the private sector the consumer of services and the purchaser of services -- i.e., the person who gets the service and the person who pays for the service -- is usually the same person. That person is able to make a value-cost judgment as to whether they wish to purchase that service or not. But the services delivered by government are. . . . In essence, the consumer of the service and the purchaser of the service are divorced. The taxpayer is in essence a purchaser of services. The consumer may be the recipient of welfare, the child who can get an education, the road that is paved or whatever the score may be. We have this divergence between the consumer and the purchaser. Government stands in the middle making these value judgments.
If we are ever going to resolve this as best as we can -- and I do say as best as we can -- we're going to have to bring the consumers' and the purchasers' thinking closer together, as we have in the private sector. When you go out and buy a new car for yourself, you weigh the value and the cost and make your own personal decision.
We've heard about focus groups and customer surveys and how we evaluate the methodology of whether we are getting value for money. Harry, I think you mentioned the fact that we can have a 60 percent success rate, but if 40 percent of the ones that are not resolved are the difficult cases, then perhaps it is very superficially achieving success. If we're trying to deliver a service that has complex cases and simple cases, and we're only resolving the simple cases, and all the difficult ones are falling by the wayside, then even though 60 percent may be satisfied, it may be an abysmal record. I haven't seen academia or the consultants or the researchers or a Public Accounts Committees struggle with the concept of how you try to bring the thinking of the consumer and the purchaser of goods together, as we have in the private sector, to see if there's anything there that will ensure value for money by government in the same way in which you get value for money in the private sector.
H. Hatry: One approach that's been used, as you probably know, is the use of more fees and charges for services such as garbage collection, to give some choices.
J. Williams: Which is fine when the consumer is able to afford the purchase. But by and large, government usually steps in when the consumer of the service is not in a position to also be the purchaser. I use, for example, welfare, education, health or all these things.
H. Hatry: There are also things like the defence department environment, where things become extremely difficult. Basically, as the elected officials here, you really play that representative role. That's what's happening here.
By the way, on privatization, I was with General Electric for eight years as an industrial engineer. I'm fairly familiar with some of the measurement devices that are used also. The problem with the private sector, the competition in some of the studies we've looked at, is that you have to be very careful that the private sector has certain profit incentives, which at times are not the ones that we would want. A study of hospital emergency rooms in the United States -- in comparing public versus private -- found that even though the costs were lower, they were turning people away from the emergency rooms in the private hospitals. Of course, this is a stronger tendency in the private sector where they have a choice to handle easier-to-help clients, because they're cheaper.
I think partly the answer to that part of your question is what I did not speak about but what is becoming used more throughout the world, which is performance contracting -- that is, when you contract with the private sector, you build into the contract performance indicators with certain targets on them. There have been a number of trials of that procedure. In some cases they have actually used incentive targets: if your return rate on vehicles, for example, if you're doing vehicle repair, is under a certain amount, you'll get a bonus, and if not, you'll get a small fee. There have been devices like that used in a number of cases as another mechanism for trying to make the private sector relationship take care of some of these other issues that you're worried about. In the case of hospitals, you could still use the private sector but obviously pay them more for lower-income, more needy families, and therefore switch the incentives. But the problem is very difficult in this whole business in terms of giving incentives for good performance.
R. Chappell: If I could just add this, it's obvious that our role as Members of Parliament is to provide that interface, and yes, we are all struggling in every jurisdiction with how we can do that more efficiently. Some of the things we've been talking about this afternoon are, in fact, the devices which enable Members of Parliament to perhaps make more accurate and more beneficial decisions on behalf of those who elect us -- in other words, those who want the services. But there are, as Harry says, user-pay processes and all of those things that a number of different jurisdictions have introduced. I guess there's always going to be that tension. There's never going to be one right way of doing it to get the user and the payer of the services so close together that you can get the best fit between the two.
George, I just want to go back, by way of a supplementary answer, to the last question I was asked, which is the relationship between outcomes and the budgetary process. I'm just reminded that we are currently reviewing the Public Finance and Audit Act in New South Wales, and it will be expressed specifically in the new act. The principle of establishing a close link between parliamentary appropriations and outputs and outcomes to be achieved will be expressed in the legislation. The actual mechanism, of course, for doing that, will be determined administratively and as part of the budget allocation process between the minister and the departmental head. Because it will be part of that process, it will also then be subject to scrutiny and review by the parliamentary process. In that regard, I would think the Public Accounts Committee will again have a significant role to play. So we are going to encapsulate the principle of linking those things in that new legislation.
G. Morfitt: Ladies and gentlemen, we have time for one more question or comment before we adjourn for a break. Could I invite someone to make some meaty, meaningful observation on where we are in all of this? All of us are thoroughly involved with this whole thrust towards managing for results. We're all in different places along the spectrum, but we're all moving in that direction. Is there some comment from either the auditors general or the legislators as to just what you think about all this? Do you think we're going to be successful?
Why don't I leave it to Denis to say something about this? Being the auditor general of Canada, he is fully absorbed in this whole matter at the federal level.
I might say that I was at the get-together of the legislative auditors in Australia two years ago in Hobart, and they are again dealing with all of this, at the same time as the legislators down there in New Zealand, and anywhere you want to look. Denis, how about some final remarks for this session?
D. Desantels: I guess I tend to be by nature on the optimistic side. As far as I'm concerned, it is an extremely tough challenge. Some people might argue that to a certain extent it's an academic debate, but I think it's important that we find the right answers and eventually transform the traditional culture of focusing on budgets and inputs, and link very clearly the use of tax money, the budgetary decisions, to performance by government organizations.
[3:00]
I sense from my last five years in Ottawa, particularly in the last two years or so, that there's more acceptance of those notions on the part of the bureaucracy, in particular. I think we're past the stage where people are arguing that it shouldn't be done. I think we're at the stage now where people are looking for ways to do it, as opposed to arguing that it can't be done. So we've moved somewhere on the spectrum, at least speaking for what I see at the federal level. Even at that, my own appreciation of what's happened in some other jurisdictions is that they're even ahead of that. I don't think it's just a dream. I think we can get there, but we need to get there relatively quickly.
G. Morfitt: Thank you for those closing remarks. On behalf of Fred Gingell, who has just returned from balancing the budget, I'm sure, and on everyone's behalf, I wish to thank Michel, Ray and Harry for a very meaningful presentation. [Applause.]
We're breaking for coffee, then the legislators are returning to this room for a final wrap-up, and the legislative auditors are returning to their room for a final wrap-up.
The meeting recessed from 3:02 p.m. to 3:39 p.m.
C. James: This is the final session for this year's Canadian Council of Public Accounts Committees conference in Victoria, and we are dealing with council business. I don't know if anyone has anything they would like to discuss under the item of council business, whether it relates either to the constitution or any other matter, project or issue that affects the council. I'll stop there and ask: would anybody like to discuss anything at this moment?
E. Robichaud: The only comment I would have is that in the future -- that's my opinion, and I'm a new member on Public Accounts -- more time should be given to exchanges between jurisdictions as to what is happening, what should happen and what is not happening. I think it would be quite valuable to all the members. That would be my only comment.
I will join my colleagues from Quebec and Alberta in expressing my thanks for the nice reception that we have had here. I don't know if the sun was out just because we're here, but we have enjoyed it very much.
C. James: Thank you very much. We have enjoyed having everybody here on the Public Accounts side but also in terms of the auditors, as well.
After just a couple of quick administrative notes, I can turn you loose. You will know that on your agenda at 4:30, those of you who wish are very welcome to wander over to Bastion Square where the actual office of the auditor general for the province of British Columbia is located. He is hosting a reception there. I will be leaving my office a little after 4:30, and those who would like to accompany me, feel free.
There being no official council business, what I would like to say is that ordinarily, at the conclusion of the meetings, the executive is struck for the next few years in what is a pro forma thing based on the location of the future conferences -- which are not cast in stone but for planning purposes, it's useful to know.
The past president of the council will be Mr. Fred Gingell, and because there's some expectation that Alberta may be able to host the conference next year, the president will be the Chair of the Public Accounts Committee from Alberta. The first vice-president will be the chair of the Northwest Territories Public Accounts Committee, and the second vice-president will be the chair of the Quebec Public Accounts Committee, or the Committee on Budget and Administration. After Quebec, I believe it goes on to Saskatchewan; if not, Saskatchewan follows within a year or two after that. That gives everybody an idea as to where the conferences are likely to be held, bearing in mind that it's up to the jurisdiction whether it is feasible for them to put the conference on.
The auditors will be meeting in these locations over the course of the next few years. As you know, we try to share costs, resources and some sessions, which I think all would agree are generally cost-effective and fairly productive. If we could strive toward retaining that format, I think everybody would be well served. Without further ado, if there are no further questions or comments, and on behalf of Mr. Fred Gingell, who is in the middle of a radio interview right now, I declare the conference officially closed.
The meeting concluded at 3:43 p.m.
Copyright © 1999: Queen's Printer, Victoria, British Columbia, Canada