Artigos

Conteúdo (article):

ASSESSING THE TECHNICAL EFFICIENCY OF BRAZIL’S TEACHING HOSPITALS USING DATA ENVELOPMENT ANALYSIS

ABSTRACT
The general objective of this study was to assess the technical efficiency of Brazil’s teaching hospitals using data envelopment analysis (DEA). To this end, a quantitative exploratory study was conducted with secondary data from the national health information system (DATASUS) using an output-oriented DEA model. The study population consisted of 29 large-sized teaching hospitals located in the country’s Center-West, South, Southeast, and North regions. Twelve hospitals were shown to be on the efficient frontier (technically efficient) and 17 were off the frontier. Absolute efficiency values were calculated for the hospitals that were off the frontier, using benchmarks with weighting for benchmarking. Private for-profit hospitals were shown to be the most efficient, followed by private not-for-profit and public facilities. The findings of this study suggest that DEA has potential for assessing technical efficiency in hospital settings in relation to operational capacity.
Keywords: Teaching hospitals; the Unified Health System; Health service evaluation; Benchmarking and Operational Research.
1 INTRODUCTION
In recent years in Brazil, health managers have investigated alternative ways to optimize physical and financial resources in pursuit of healthcare excellence. The financial crisis that has hit Brazil, with serious repercussions for public health, highlights the need to seek cost reduction solutions in order to improve access to quality health services.
The universalization of healthcare, enshrined in Brazil’s 1988 Constitution, brought the Brazilian population the hope that their health needs will always be met by the State. However, it is evident that maintaining a health system that is accessible to all is an extremely complex task for numerous reasons. Scarce financial resources and the lack of a physical structure capable of serving the whole population are just some of the reasons that make the administration of healthcare establishments such a major challenge 1.
The administration of a healthcare organization, whatever its size, within a complex system, requires capacity building, competency development, constant innovation, and entrepreneurship. In the pursuit of competitive advantage, there is an urgent need to add value to production through differentiation, attained through product and process innovation. This new model of production requires the professionalization of management and staff flexibility and initiative to perform multiple complex tasks simultaneously and solve unexpected problems 2.
The health sector in Brazil, and more specifically hospital establishments, face multiple challenges, ranging from sectoral policies to the healthcare model that has been adopted over recent decades. It is therefore necessary to streamline health actions, analyzing cost-effectiveness and cost-benefit relationships and minimizing mistakes in resource allocation and health policy implementation in the constant pursuit of the optimization of scarce resources 3.
Teaching hospitals have the most complex funding arrangements within Brazil’s national health system (Sistema Único de Saúde– SUS), receiving large volumes of public investment and performing the system’s most costly procedures. They provide essential health services while at the same time promoting research and teaching. Furthermore, these facilities play an important role in upholding the principles of the SUS, particularly comprehensiveness. Access to the services they deliver should be integrated with other healthcare facilities within the system to ensure that all patients have access to the most modern health services 4.
With regard to hospital services, the assessment of productive performance requires models that describe the characteristics of these organizations in an abstract, simplified, and systematic manner, so that they can be understood through the prism of interest. Models are partial representations of reality constructed for a definite purpose; however, they should be shaped around a theory and sufficiently comprehensive to reflect reality, ensure that results are consistent with the underlying theory, and serve the desired purpose. Different theories have been used to underpin hospital models depending on the research aims and the approach adopted. Furthermore, the specific characteristics of hospitals and the healthcare services they provide have tended to limit and hamper the development of comparative research 5.
Globalization has stimulated changes in the transformation process in pursuit of sustainability. Organizations from different sectors, notably the health sector, have had to change to survive, with competitiveness becoming dictated by agility, productivity, and quality, or in other words, efficiency and productive process effectiveness criteria. With regard to production systems, the concept of “lean thinking”, coined by a study of Toyota’s production system, brings numerous examples of best practices. When applied to production systems, the lean thinking methodology has a direct influence on planning, scheduling, production control, and, consequently, operational performance. The lean approach is entirely related to waste elimination. This production system emerged as a manufacturing system whose objective is to improve process and operations through the continual reduction of waste 2. Efficiency concerns have been criticized because they are said to encourage the downsizing of services; however, when efficiency is a management assessment criterion, it makes sense to use managerial methods. Technically speaking, the measurement of indicators is not an insurmountable problem; however, it depends on institutional policy and is warranted when it serves decision-making. Otherwise it ends up being an excuse for one-off processes driven by agents outside the sector with little knowledge of the specificities of its processes 6.
A study conducted in Italy highlights that at some point, a hospital goes beyond its optimal level of efficiency and begins to show diseconomies of scale. Optimum size is found when all economies of scale have been exploited, without creating diseconomies. Hospital administrators and policy makers should therefore concentrate on the efficient use of inputs to produce health care services and be aware that it is possible to reduce inputs without compromising healthcare. Understanding the key factors that lead to inefficiency allows hospitals to improve output without seeking additional inputs, or to maintain output while reducing excess inputs 7.
One of the challenges of evaluating healthcare management using a deterministic rather than probabilistic approach is finding appropriate techniques that allow managers to analyze all aspects involved simultaneously. Data envelopment analysis (DEA) is a widely used methodology for evaluating the productivity and technical efficiency of organizations that use multiple inputs to provide multiple outputs. The methodology enables the identification of best practices by estimating frontiers utilizing linear programming 8.
DEA was developed by A. Charnes, W. W. Cooper, and E. Rhodes to assess the efficiency of not-for-profit and public sector organizations, later being applied to for-profit organizations. DEA compares an organization’s service delivery units to its other units in multiple locations, classifying efficiency according to resource input and output indices. Multiple input (for example, hours worked, materials) and output (for example, sales) data are possible and desirable when measuring unit efficiency. Given the above, the linear programming model defines the efficiency margin based on the units that achieve efficiency scores of 100 per cent. Areas that need improvement are identified by comparing the operational practices of the efficient units to those of the less efficient units. Sharing the management practices of the most efficient units with those of the least efficient provides an opportunity to improve these units and increase the overall productivity of the system 9.
DEA is a non-parametric method, that is, it does not establish an explicit functional form, but rather explores the relationship between variables to obtain a comparative measure of the efficiency of Decision Making Units (DMUs) based on best practices. This technique enables managers to analyze the relative efficiency of multiple input and output productive units by constructing an efficient frontier and identifying inefficiencies generated by less than adequate decisions and actions 10.
DEA seeks to maximize the efficiency of service units, expressed as the ratio of outputs to inputs, by comparing the efficiency of a specific unit to the performance of a group of similar units carrying out the same work. Units that achieve efficiency scores of 100 per cent are referred to as efficient units, while those that fail to achieve 100 per cent are referred to as inefficient units 9.
By determining the DMUs that have the best practices, DEA constructs an empirical production frontier, where a unit’s degree of efficiency ranges between 0.0 and 1.0 (or 0 and 100 per cent) depending on its distance from the frontier. Efficiency is measured by applying the Pareto-Koopmans efficient empirical production function, where units on the frontier are only truly efficient where a reduction in an input or increase in an output does not increase another input or reduce another output 11.
The variables needed to calculate the relative efficiency of DMUs are classified as inputs and outputs. Relative efficiency is determined by the ratio of the weighted sum of the DMU’s outputs to the weighted sum of the inputs needed to generate them. These variables are adjusted using weights calculated by linear programming with the aim of maximizing the efficiency of each DMU in relation to the whole 12.
There are two more widely used DEA models: CCR and BCC. The acronym CCR derives from the names of the authors, Charnes, Cooper, and Rhodes (1978), who published the first article on DEA, while the second model is an extension of the first, developed by Banker, Charnes, and Cooper (1984) 13. The distinction between these two models is that CCR assumes a Constant Returns to Scale (CRS), whereas BCC considers a Variable Returns to Scale (VRS). Both models allow managers to assess whether productivity can be increased with a reduction in inputs or an increase in output. Moreover, by comparing the results of the CCR and BCC, it is possible to identify problems arising from changes to the scale of production or management 14.
The present study analyzed the productive efficiency of teaching hospitals in Brazil, highlighting deficiencies and potentialities and indicating better ways of streamlining existing resources guided by the following research question: What is the hospital technical efficiency score of Brazil’s general teaching hospitals based on data envelopment analysis? This study therefore sought to assess the hospital technical efficiency of Brazil’s teaching hospitals utilizing data envelopment analysis and adopting the following specific objectives: to identify technically efficient teaching hospitals using the data envelopment analysis approach; to propose output targets for hospitals identified as technically inefficient; and to compare technical efficiency according to ownership (public, private for-profit, private not-for-profit).


2 MATERIAL AND METHODS
A quantitative exploratory study was conducted using freely accessible data from the teaching hospital database of the country’s national health information system (DATASUS) collected in May 2017. Purposive sampling was used to achieve a homogeneous and comparable sample, as established by the DEA methodology. All hospitals on the National Register of Healthcare Establishments (CNES, acronym in Portuguese) that met the inclusion criteria described and justified below were included in the study:
1) Non-specialist general teaching hospitals: specialist hospitals focus on only one specialty, thereby hindering comparability.
2) Hospitals with over 150 beds (large-sized special hospitals): to ensure a sample of similar-sized establishments.
3) Hospitals with adult, pediatric, and neonatal intensive care units: to ensure a sample of hospitals with similar characteristics in terms of complexity and age groups served.
4) Hospitals with obstetric and psychiatric/mental health care beds: this criterion considered shorter hospital stays for deliveries as compared to long stays for psychiatric/mental health care.
To differentiate this study from other studies, strict inclusion criteria were proposed, particularly in relation to installed capacity and typology. We also sought to correct limitations observed in the literature, such as the use of number of hours worked rather than number of professionals, to ensure an interpretation that is closer to reality. There are few national level studies in the literature addressing the efficiency of teaching hospitals registered on the CNES, with most research focusing on university hospitals linked to the Brazilian Hospital Services Company (EBSERH, acronym in Portuguese).
The application of the DEA model can be output oriented (to achieve maximum output while keeping the level of inputs constant) or input oriented (to reduce inputs to a minimum while maintaining the same level of output) 15. Using an output maximization format including the outputs total revenue, number of hospital admission authorizations (HAAs), number of inpatient days, and number of high-complexity procedures, this application of the DEA was oriented to determine which hospital generated the highest output while maintaining the existing level of inputs and considering a constant returns scale (CCR). This orientation was chosen primarily due to the lack of capacity of municipal and state level public sector managers when it comes to the management of internal matters in contracted hospitals and also owing to resource scarcity and high demand among SUS users for tertiary care. Table 1 shows the variables used to apply the DEA methodology and the data source for each variable.
Table 1
The DEA was performed with MDeap 2 for Windows using comparable units that carry out the same functions. The comparison was made to identify inefficient units singled out for management interventions to improve performance 22.
The analysis is based on the principle that there are a number of decision making units (DMUs) that convert inputs into outputs. Hospitals use doctors, nurses, beds, etc. as inputs to generate outputs such as admissions. Thus, by using the data available for these variables for various hospitals, it is possible to determine the relative efficiency of the various hospital units by comparing inputs and outputs. Another important aspect is the difference between outputs and results. Although the objective of the decision making units is to achieve results rather than the mere generation of outputs, the former are more difficult to assess, given the number of external factors that can affect results 23.
The study was minimal risk because it only used freely accessible secondary data made available by the DATASUS, thereby dispensing the need for research ethics committee approval.

3 RESULTS
In May 2017, Brazil had 5,129 general hospitals registered on the CNES, of which only 389 (7.6%) developed some kind of teaching activity: university facilities (25), isolated higher education facilities (5), auxiliary teaching facilities (238), and teaching hospitals (121) – the object of study. These 389 hospitals accounted for 28.1% of all clinical and surgical beds and 48.3% intensive care beds.
After applying the inclusion criteria, the study population consisted of 29 eligible hospitals belonging to the Center-West, South, Southeast, and North regions (from a total of 121 teaching hospitals in Brazil). The Northeast Region was not included because it did not have an eligible hospital during the study period. Fourteen of the hospitals were public, four were private for-profit, and 11 private not-for-profit. In the DEA models, these hospitals are referred to as DMUs and were classified as efficient or inefficient.
Table 2 presents the efficiency scores of each hospital. DMUs that achieve an efficiency maximization score of 100 per cent are considered “efficient”, while those that do not are said to be “inefficient”. Twelve hospitals were shown to be on the efficient frontier, or in other words technically efficient, whereas 17 hospitals were off the frontier. Of these 17 hospitals, DMU 07 (Hospital São Paulo, the university hospital of the Federal University of São Paulo – UNIFESP) obtained the lowest score (0.50).
Table 2
Table 1 shows the benchmarks for each inefficient unit, where the higher the value, the stronger the similarity for benchmarking. DMU 16 (Clemente de Faria University Hospital) was the hospital that was identified as a benchmark of an inefficient unit most times (12 times) and obtained the highest score for similarity with DMU 19 – Hospital Nossa Senhora da Conceição AS (4.15).
Table 1: Benchmarks for inefficient hospitals
Table 2 shows the estimated output values needed for each DMU to achieve 100 per cent efficiency. For example, DMU 01 (Maria Aparecida Pedrossian University Hospital) had a revenue of R$14,111,100, processed 9,858 HAAs, approved 76,558 inpatient days, and performed 1,178 high-complexity procedures. To achieve 100 per cent efficiency, the hospital would need to increase revenue to R$18,228,600, process 10,048 HAAs, approve 78,034 inpatient days, and perform 1,201 high-complexity procedures while maintaining the same level of inputs.
Table 2: Output targets for inefficient DMUs
Hospital ownership was divided into three categories: private not-for-profit (11), public (14), and private for-profit (4). The findings show that private-for-profit hospitals were the most efficient (average efficiency score = 0.95), followed by private not-for-profit (0.93) and public (0.91).



4 DISCUSSION
The DEA model is a mathematical, non-parametric tool used to measure the efficiency of productive units called Decision Making Units. The method uses linear programming to calculate an efficiency index for each DMU and generate an empirical efficient frontier composed of units that show the best practices (benchmarks) specific to the study sample. The units on the frontier are classified as efficient and the other units as inefficient. The DEA model is appropriate for various inputs and outputs. The efficiency of a DMU is the ratio of its productivity to that of the most efficient DMU in the dataset 24. However, the results should be interpreted with caution since being classified as efficient may also be interpreted as an indicator in the balance between inputs and outputs. Furthermore, achieving a maximum score does not mean the absence of problems, but rather that the unit produced more outputs with fewer inputs. Quality was not measured due to the lack of relevant secondary data such as information on infection control and patient safety.
In the DEA model, inefficient units are benchmarked against efficient units19. The inefficient units can study the absolute values of the variables of their benchmarks and implement the necessary changes in order to find their way onto the efficient frontier. The findings of another study 13 that measured total efficiency using the CCR model showed that private hospitals were more efficient than public hospitals. The same study highlighted an important limitation in comparing public and private hospitals, pointing out that the two categories operate under a different logic. Public hospitals fulfill an important social function, whereas private hospitals are guided by the logic of the market, which is very particular in the case of health services 13. Despite this difference, we understand that this comparison is important given the increasing focus on SUS privatization. Arguments for and against privatization are scarce in the current literature and require further exploration to provide inputs to inform decision making.
It is also important to highlight that, from a financial perspective, public hospital staff costs are not directly related to monthly revenue. Despite this advantage over private for-profit and not-for-profit hospitals, public hospitals showed the lowest average efficiency score.
Unlike other studies that have used DEA methodologies to assess the efficiency of health services in Brazil, the present study was restricted to teaching hospitals with similar installed capacity, seeking to enable comparisons between intrinsically more homogeneous entities. To this end, we used classic indicators previously used by similar studies, such as inpatient days and level of output, bed capacity, and health professional worked hours.
A significant challenge is how to assess the operational context of hospitals using quantitative approaches. When it comes to SUS hospitals, contextual elements like system organization or disorganization, socio-political context, funding, time of use, and hospital condition are key contextual elements that can influence performance and explain certain results found 13.
The study findings show that 59% of the teaching hospitals have the potential to increase output maintaining the current level of inputs by increasing the provision of services to SUS users. In addition to setting output targets for inefficient units, DEA indicates benchmarks using more efficient hospitals for benchmarking. In this way, it is possible to bring the performance of these establishments closer together, seeking alternatives to improve resource efficiency and share success stories.
Although teaching hospitals use more technological resources on average than other SUS services, medium complexity care – both inpatient and outpatient cases that could be referred to other facilities within the system – makes up a large proportion of the services they provide. Performing low complexity procedures in teaching hospitals has a number of drawbacks, including: the use of services with high installed capacity and specialist staff and that are therefore generally more costly for simple procedures that could be better performed at other levels of care; patients seeking continuing care in teaching hospitals to treat basic health problems are forced to make frequent trips outside their municipality of residence because these facilities are usually located in major urban centers in larger municipalities; and, finally, low complexity cases take up appointments that would otherwise be used for high complexity cases, leading to waiting lists and suppressed demand, delaying treatment and jeopardizing patients’ health 4.
In addition to stimulating reflection on health system financing, the above considerations on technical efficiency in teaching hospitals contribute to the emerging discussion surrounding the use of existing resources and methodologies that help to identify alternatives to widen access to health services at all levels of care. The findings of this study can help inform the implementation of wider control mechanisms and health service evaluation and provide valuable information to local health managers on the current output of large public, private, and not-for-profit teaching hospitals in Brazil.

5 FINAL CONSIDERATIONS
Efficiency lies at the top of the agenda of health managers and the DEA technique enables managers to arrive at a consensus and identify priorities, strengthening its effectiveness with value judgments and interaction with experts. The present study demonstrates that DEA has the potential to become a key tool for evaluating services and assist in decision making in the area of health. To this end, a feasible path is to bring epidemiology in health services and operational research closer together 25.
Our findings demonstrate the potential benefits of using DEA to evaluate government programs and policies, set legally binding targets, prioritize corrective actions, etc., in so far as it identifies the gaps and level of effort necessary for each variable to attain efficiency. Moreover, the technique is neutral and welcomes the possible and desirable participation of managers and other agents in the assessment process. This participation is desirable in view of the SUS’s complex governance structure. Finally, it allows the determination and rating of potential paths to the efficient frontier for the purposes of monitoring proposed objectives 10.
The current reality in Brazil increases in spending out of the question, meaning that it is increasingly necessary to conduct evaluations based on available resources with the aim of maximizing service output, both in more developed regions and in areas that lack adequate facilities. Give Brazil’s vast size, it is necessary to correct systemic failures, formulating health policies and health teams so that they do not remain inoperative due to bureaucratic abuse and have the power to combat social inequalities at national, state, and local level 1.
One of the difficulties in assessing the technical efficiency of hospitals is the selection of inputs and outputs. Inappropriate measures can produce biased and inconsistent results. The current literature suggests outputs such as number of admissions and number of patient days or hospital discharges, largely ignoring the severity of the patient\'s health condition and service quality. Many studies on the productive performance of hospitals express this limitation as the lack of available data 5.
The main limitation of this study relates to the reliability of the DATASUS data, given that this depends on the data entry of individual hospitals, which often lacks appropriate inspection and auditing to ensure data validity. Thus, the dataset may not fully reflect actual hospital efficiency due to possible deficiencies in data entry. Another limitation is the lack of available data on service quality. This data is important for a more detailed analysis of output in both inpatient and outpatient services. Future studies on the technical efficiency of teaching hospitals should encompass other important dimensions such as regulation, financing, incentives, contracting models, and epidemiological profiles.
The study findings suggest that DEA has potential for assessing technical efficiency in hospital settings in relation to operational capacity. Since the present study used a national sample, we used variables with publicly accessible data sources, which unfortunately lack information on service quality in the SUS. The results are specific to the particular study context and should not be extrapolated to other situations.

REFERENCES

1 Faria TC, . Tecnologia de gestão hospitalar: proposta e aplicação em SC [Monografia]. Florianópolis: Universidade Federal de Santa Catarina; 2016. Disponível em: https://repositorio.ufsc.br/handle/123456789 /166441

2 Fonseca PC, Ferreira MAM. Investigação dos Níveis de Eficiência na Utilização de Recursos no Setor de Saúde: uma análise das microrregiões de Minas Gerais. Saúde Soc. 2009; 18(2):199-213. Disponível em: http://www.scielo.br/scielo.php?pid=S0104-12902009000200004&script=sci_abstract&tlng=pt

3 Barata LRB, Mendes JDV, Bittar OJNV. Hospitais de ensino e o sistema Único de saúde. Rev. adm. saúde, 2010; 12(46):7-14. Disponível em: http://sistema.saude.sp.gov.br/sahe/documento/ras46.pdf

4 Wolff, LDG. Um modelo para avaliar o impacto do ambiente operacional na produtividade de hospitais brasileiros [Tese]. Florianópolis: Universidade Federal de Santa Catarina; 2005. Disponível em: https://repositorio.ufsc.br/handle/123456789/102461

5 Vecina Neto G, Malik AM. Tendências na assistência hospitalar. Ciên. Saude Colet. 2007; 12(4):825-839. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232007000400002

6 Scaratti D, Calvo MCM. Indicador sintético para avaliar a qualidade da gestão municipal da atenção básica à saúde. Rev. Saúde Públ. 2012; 46(3):446-455. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-89102012000300006

7 Fitzsimmons JA, Fitzsimmons MJ. Administração de serviços: Operações, estratégia e tecnologia da informação. 7° ed. Porto Alegre: AMGH; 2014.

8 Pedroso MM, Calmon PCDP, Bandeira LF, Lucena RAV. Eficiência Relativa da Política Nacional de Procedimentos Cirúrgicos Eletivos de Média Complexidade. RAC 2012; 16(2):237-252. Disponível em:
http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1415-65552012000200005

9 Lobo, MSC. Aplicação da Análise Envoltória de Dados (DEA) para Apoio às Políticas Públicas de Saúde: o Caso dos Hospitais de Ensino [Tese]. Rio de Janeiro: Universidade Federal do Rio de Janeiro; 2010. Disponível em: http://www.producao.ufrj.br/index.php/br/teses-e-dissertacoes/teses-edissertacoes/doutorado/2010/275--240

10 Senra LFAC, Nanci LC, Mello JCCBS, Meza LA. Estudo sobre métodos de seleção de variáveis em DEA. Pesq. Operacional 2007; 27(2):191-207. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0101-74382007000200001

11 Souza PC, Scatena JHG, Kehrig RT. Aplicação da Análise Envoltória de Dados para avaliar a eficiência de hospitais do SUS em Mato Grosso. Physis 2016; 26(1):289-308. Disponível em: http://www.scielo.br/pdf/physis/v26n1/01 03-7331-physis-26-01-00289.pdf

12 Calvo MCM. Hospitais públicos e privados no Sistema Único de Saúde do Brasil: O mito da eficiência privada no estado de Mato Grosso em 1998 [Tese]. Florianópolis: Universidade Federal de Santa Catarina; 2002. Disponível em: https://repositorio.ufsc.br/handle/123456789/82445

13 Bandeira DL. Análise da Eficiência Relativa de Departamentos Acadêmicos – O Caso da UFRGS [Dissertação]. Porto Alegre: Universidade Federal do Rio Grande do Sul; 2000. Disponível em: http://www.lume.ufrgs.br/handle/10183/3752

14 Andrade EL. Introdução à pesquisa operacional: métodos e modelos para análise de decisões. 5° ed. Rio de Janeiro: LTC; 2015.

15 Silva RM, Vieira GBB, Senna ETP, Senna LAS. A aplicação da Data Envelopment Analysis (DEA) para análise da produtividade na atividade de picking em um centro de distribuição. Rev. de Eng. e Tecn. 2016; 8(1):35-49. Disponível em: http://revistas2.uepg.br/index.php/ret/article/view/11661

16 Lins ME, Lobo MSC, Silva ACM, Fiszman R, Ribeiro VJP. O uso da Análise Envoltória de Dados (DEA) para avaliação de hospitais universitários brasileiros. Ciênc. Saúde Colet. 2007; 12(4):985-998. Disponível em:
http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-8123200700040002

17 Lobo MSC, Lins MPE. Avaliação da eficiência dos serviços de saúde por meio da análise envoltória de dados. Cad. Saúde Pública 2011; 19(1):93-102. Disponível em: http://www.revistargss.org.br/ojs/index.php/rgss/article/view/248

18 Silva BN, Costa MAS, Abbas K, Galdamez EVC. Eficiência hospitalar das regiões brasileiras: Um estudo por meio da análise envoltória de dados. RGSS 2017; 6(1):76-91. Disponível em: http://www.revistargss.org.br/ojs/index.php/rgss/article/view/314

19 Silva MZ, Moreth BR, Schuster HA. Avaliação da eficiência hospitalar por meio da análise envoltória de dados. RGSS 2016; 5(2):100-114. Disponível em: http://www.revistargss.org.br/ojs/index.php/rgss/article/view/248

20 Gonçalves AC, Noronha CP, Lins MPE, Almeida RMVR. Análise Envoltória de Dados na avaliação de hospitais públicos nas capitais brasileiras. Rev. Saúde Públ. 2007; 41(3):427-435. Disponível em: http://www.scielo.br/scielo.php?pid=S0034-89102006005000023&script=sci_abstract&tlng=pt