Artigos

Conteúdo (article):

DEVELOPMENT OF A QUANTITATIVE HEALTH INSPECTION INSTRUMENT IN FOOD AND NUTRITION SERVICES, BRAZIL.

RESUMO
O estudo visou desenvolver um Instrumento Quantitativo para Inspeção Sanitária (IQIS) em serviços de alimentação e nutrição de grande porte no Brasil. Utilizou-se a tecnologia de inspeção, no Modelo de Avaliação do Risco Potencial (MARP) e legislação sanitária brasileira. Estruturaram-se 12 dimensões, 41 módulos, 57 indicadores de controle de riscos (críticos/não críticos), numa escala de 0-5, totalizando 1.512 índices com codificação de respostas fechadas. O IQIS foi validado com o Coeficiente de Kappa, com excelente concordância para atributos de clareza e relevância (k = 0,82 e k = 0,92) e boa concordância para o atributo aplicabilidade (k = 0,78). O teste de Kruskal-Wallis mostrou inexistir diferença significativa entre as avaliações (p = 0,423), o Coeficiente de Correlação Intraclasse foi satisfatório (CCI = 0,53), o Alpha de Cronbach (α = 0,71) aceitável. O resultado final possibilitou classificar o serviço como tendo risco sanitário inaceitável. Considera-se o IQIS com conteúdo validado, tendo confiabilidade e reprodutibilidade para avaliação higiênico-sanitária, sendo uma inovação tecnológica para serviços de alimentação e nutrição e vigilância sanitária, possibilitando inspeção detalhada e rigorosa.
Palavras-chave: Inspeção Sanitária, Risco Sanitário, Serviços de Alimentação, Estudos de Validação.
ABSTRACT
The study aimed to develop a Quantitative Health Inspection Instrument (IQIS) in sizeable Brazilian food and nutrition services. The inspection technology based on the Potential Risk Assessment Model (MARP) and the Brazilian Health Legislation was used. Twelve realms, 41 modules, and 57 risk control (critical/non-critical) indicators were associated with an interval scale of 0-5, totaling 1,512 indices with closed-ended response coding. The IQIS was validated with the Kappa Coefficient, with excellent concordance for the attributes of clarity and relevance (k=0.82 and k=0.92) and good concordance for applicability (k=0.78). The Kruskal-Wallis test showed no statistically significant difference between the evaluations (p=0.423), the Intraclass Correlation Coefficient was satisfactory (ICC=0.53), and Cronbach’s Alpha (α=0.71) was acceptable. The final result of the evaluation allowed us to classify the service as having an unacceptable health risk. The IQIS has a validated content, is reliable and reproducible to evaluate the hygienic-sanitary conditions, and is technological innovation for food and nutrition services and health surveillance, allowing a detailed and thorough inspection.
Keywords: Health Inspection, Health Risk, Food Services, Validation Studies.


INTRODUCTION
Foodborne diseases (FBD) are common and underreported in Brazil. The World Health Organization (WHO) estimates that one-third of the population will suffer from FBD annually, but only a small part will make the notification (1) (2) (3). According to the Centers for Disease Control and Prevention (CDC), 48 million people fall ill each year from FBD in the U.S. In 2014, the CDC reported 864 outbreaks, resulting in 13,246 diseases, 712 hospitalizations and 21 deaths (4). In Brazil, 11,241 outbreaks were recorded in the period 2000-2015. Of these, 218,507 people became ill, and 2,121,110 were exposed. Fifteen percent of the outbreaks were related to food consumed in restaurants and bakeries, and 8.2% with housing and work (5). Food contamination can occur throughout the production chain, and risk management is a significant health control measure, which is essential to avoid it (6) (7). A considerable proportion of food outbreaks arise from the association between the consumption of food contaminated by improper handling and conservation or distribution (8) (9).
Regulation of health risk in food services
The oldest technical health control regulation is the Codex Alimentarius, established by the United Nations Food and Agriculture Organization and the WHO (10). In 1993, the Food Hygiene Committee of the Codex published a guide for the application of the Hazard Analysis and Critical Control Point (HACCP) system, whose application in Brazil is based on Ordinance Nº 1.428/1993 (11) (12) of the Ministry of Health. In Brazil, the publication of RDC Nº 216/2004 is a health control milestone, establishing minimal conditions for food and nutrition services, but without an inspection roadmap (13). The Health Surveillance Center of the State Health Secretariat of São Paulo published Ordinance Nº CVS 5/2013 (14), complementing the procedures of RDC No. 216/2004, defining an inspection roadmap with statewide coverage.
The health inspection technology is highlighted as an instrument of risk management, assessing compliance with health legislation (7) throughout the food chain. The usual health inspection routines evaluate the services using dichotomous variables with compliant and non-compliant indicators, which show the level of compliance with the legislation, without judging the criticality of the analyzed items, such as the Roadmap for the Assessment of Hygienic-Sanitary Conditions in Food Services (15). The Integrated Handbook for Prevention and Control of Foodborne Diseases (7), published by the Ministry of Health, provides for an inspection roadmap that defines a criterion for classifying indicators based on the relevance of individual risk regarding product quantity and safety and workers’ safety. The School Feeding Good Practices Checklist (16) considers the consequence potential of each indicator. On May 10, 2013, the Ministry of Health approved Ordinance Nº 817/2013, with the national guidelines for the elaboration and implementation of the Food Service Categorization Pilot Project (17), implemented in the host cities of the 2014 FIFA World Cup, providing for the classification of the criticality of each indicator, health and safety consequences, level of association concerning outbreak-associated flaws, and weighting to establish values (18).
Technological innovation for large food services
Health surveillance has three sets of practices, with varying risk notions depending on the strategy. Health promotion actions aimed at group education aim at increasing the quality of health of the population and are unrelated to a specific risk factor. Risk or harm prevention actions act on specific factors, based on epidemiological risk, to reduce or eliminate new occurrences. Health protection actions seek to strengthen defenses. They address risk as possible harmful events to health (19). Considering its dynamic nature, it is necessary to search for new technologies that consider the complex and cross-cutting nature of the processes and manage health risks (20). All the health inspection guidelines found in the literature (15) (16) (18) (21) are based on the probabilistic risk concept. The notion of risk concept, of great relevance in the area of health surveillance, was thus proposed (22) to cover the complexity of a risk concept.
The potential risk has two essential characteristics that differentiate it from the concept of classic risk, relating to the possibility and not to the probability of occurrence of the unexpected. A classical risk evaluation is based on events that have occurred, while the potential risk builds on those that are occurring and the effects that may or may not occur. The potential risk can be quantified and classified into levels of acceptability, and its operationalization enables the monitoring and comparison of several objects under health surveillance control. In this context, the Potential Risk Assessment Model (MARP) was developed to gauge risk and its classification in the space of acceptability (19). It classifies the potential risk into acceptable, tolerable and unacceptable through mathematical formalism. MARP’s application is particularized from the risk control indicators, based on a defined acceptability scale classified into critical and non-critical (22) categories.
This study aimed at developing the IQIS for large food services based on the MARP and Brazilian legislation. The use of the potential risk concept evidenced advances in health risk management in related areas such as hemodialysis (20) and radiodiagnosis (22) services. Its application in food and nutrition services is a technological innovation, enabling risk anticipation and health protection.
METHODS
This study was developed from October 2015 to May 2016, with the following steps: i) Elaboration of the instrument by sector health professionals; ii) IQIS evaluation concerning content validity through submission to the Expert Committee, as well as external validity and reliability for the performance of Pretest and Test, applied in large food and nutrition service in São Sebastião do Passé, Bahia, Brazil, which prepares 2,541 daily meals for workers: 276 breakfasts, 1,393 lunches, 243 dinners and 629 snacks. The Research Ethics Committee of the Bahia School of Medicine and Public Health approved the research under reference CAAE 46698415.3.0000.5544. Figure 1 summarizes the IQIS development and validation process.
FIGURA 1
Elaboration of the IQIS
This stage counted on a group of seven professionals, of which two are doctors in public health and risk management researcher, two are ergonomists and managers in workers’ health, two are nutritionist experts in public health and experience in collective nutrition, and one is a health surveillance expert working in the area of risk management. The literature review allowed the analysis of the existing instruments and methods, as well as the theoretical basis for the decision and elaboration of a new instrument. Following the identification of primary sources, review of secondary sources, critical reading and summary of the literature was performed using keywords or descriptors: risk, health risk, epidemiological risk, potential risk, health surveillance, health inspection, food health inspection, health inspection roadmap, health inspection in restaurants roadmap, industrial kitchen health inspection roadmap, health inspection validation, questionnaires, evaluation instruments, and elaboration and validation of instrument (21) (23) (24) (25) (26) (27) (28) . The IQIS is based on the MARP, whose mathematical formalism is detailed by Navarro (22), applied to food and nutrition services. This model proposes underlying processes according to the natural flow of the inspector. The main activities that are a potential risk for each of these processes are defined, as are the risk control indicators for each of these activities. Finally, each indicator is classified as critical (IC) and non-critical (INC) and associated with an interval scale from zero to 5, where zero evidences nonexistent or inadequate risk control and five is excellent risk control, with the following classification: 0 – nonexistent or inadequate; 1 – insufficient; 2 – reasonable; 3 – good; 4 – very good and 5 – excellent. The coding of closed-ended answers for each index of the scale was developed to reduce the subjectivity of the evaluator, with six possibilities for each indicator. The food production flow was used as a criterion to define the realms, which were broken down into activities, areas or equipment that generate potential risk, called modules. These developed into risk control indicators and were associated with indices with quantitative variables, coded in closed-ended responses. Fourteen eight-hour meetings were held with professionals, who employed RDC No. 216/2004. The requirements to meet the resolution were considered in index 3 of each indicator, and the lower and upper grades were defined from the experience and literature researched. A panel was developed with the professionals to classify indicators into critical and non-critical; these professionals used their knowledge for this task and, later, the table of the association of risk factors for the occurrence of outbreaks was elaborated by National Health Surveillance Agency specialists (18). The panel was developed from three 8-hour meetings, in which each member expressed his opinion on the indicator in question, achieving the group’s consensus.
IQIS validation
Many validity and reliability methods are mentioned in the research literature (29) (30) (31) (32) and, concerning methodological conduction, we chose the definitions that establish validity as how much a test measures what we wish to gauge and what reliability is related to the accuracy and precision of the measurement procedure (33). A Judges’ Committee was established to validate the content and consisted of three nutritionists, with average professional experience in the areas of food safety and health surveillance of 17 years; one worked as a teacher and two at the Regional Council of Nutritionists. One had a master’s degree, and two had specialization in health surveillance. The first version of the IQIS was submitted to the Judges’ Committee, along with the electronic spreadsheet for content evaluation, regarding the attributes of applicability, clarity, and relevance. Applicability was evaluated on a scale with four alternatives: 1 – fully agree, 2 – partially agree, 3 – no opinion, and 4 – disagree. Clarity was evaluated on a scale with three alternatives: 1 – high, 2 – fair and 3 – low, to verify whether the indicators were written intelligibly. Relevance was assessed on a scale with three alternatives: 1 – relevant, 2 – not relevant, and 3 – not applicable, noting if the items reflected the concepts involved and if they were adequate to achieve the proposed objectives. An eight-character code was created for each set of realm, module, indicator and index to organize the database generated in this stage. Experts gathered for two days in a quiet and uninterrupted environment, answered the 1,344 evaluations and were instructed not to exchange information with each other. The Kappa coefficient was used to analyze the results, considering k> 0.80-1.00 as excellent concordance; k> 0.60-0.79, good concordance; k> 0.40-0.59, moderate concordance; k> 0.20-0.39, poor concordance and k> 0-0.19, no concordance (34). All results with a Kappa Coefficient above 0.60 were accepted, i.e. showing good concordance. The Pretest checked whether all items were understandable to members of the target population. To that end, we invited five nutritionists, with a mean age of 42 years and an average of 14 years of experience in this area, all with specialization in health surveillance or related areas. The IQIS was applied on the same day, by direct observation for all the risk control indicators, except for the documentary ones, answered by interviews with the sector’s managers, which was conducted over eight hours during the food service’s administrative hours. The evaluators received the printed instrument and were instructed to assess the modules at the same time, as well as not to exchange information among themselves, to avoid influencing the answers. We used interobserver frequency statistical tests in the pre-test analysis, discussing the results in an interactive process between researchers and nutritionists to clarify controversial points. All items with a percentage of concordance less than 80% were analyzed to eliminate or adjust ambiguous indicators or indices or that carried value judgments. We assessed whether the concepts were drafted in a way that was comprehensible to what was expected to be measured and adequate for the proposed objectives. The documents and their modification proposals were analyzed and accepted, generating the second version of IQIS. The test mainly aimed to evaluate the psychometric characteristics of the instrument. To this end, we invited seven nutritionists with a mean age of 38 years and an average of 11 years’ food production experience. All of them had specialization in health surveillance or related areas, and applied the second version of IQIS in the same food and nutrition service of the pre-test, following the same previous conditions. The data were collected with the printed forms imported to the SPSS software version 24.0 to perform the analysis (35).
Statistical analysis
Data statistical treatment was performed through descriptive and exploratory analyses to investigate the accuracy of entries, the distribution of missing cases and the distribution of frequencies. The non-parametric analysis of variance was used to compare the percentages of the adequacy of hygienic-sanitary conditions of the food and nutrition service, employing the Kruskal-Wallis test and considering the significance level of 5% in all analyses (36). The Intraclass Correlation Coefficient was applied to verify the reliability and usability of the instrument in other contexts using the same Test data, considering that in the case of ICC ≥ 0.75, reliability should be considered excellent; an ICC of 0.40-0.75 shows satisfactory reliability; and an ICC <0.40 (37) reflects poor reliability. Finally, the Cronbach Alpha coefficient was calculated for each realm to verify the internal reliability, where α ≥ 0.70 values were considered acceptable and α ≥ 0.80 (38) (39) were highly reliable values.
RESULTS
The IQIS was defined in its first version based on the requirements of RDC No. 216/2004 and the production process flow. It contained 12 realms and 41 modules, spread into 76 risk control indicators and 1,344 indices associated with quantitative variables. Also, indicators and indices were associated with each realm and module to assess the respective health risk, resulting in 224 different verification items. Each realm-module-indicator-index combination features an IQIS item. Box 1 shows an example of an IQIS Verification Item.
QUADRO 1
The panel of experts initially ranked 42% of the 76 indicators as critical. After the Pre-test and evaluation of the concordance rate, 57 indicators were redefined and evaluated in light of the risk factors table for the event of outbreak (18). The Kappa Coefficient was used to validate the content. The IQIS items were evaluated for applicability, clarity and relevance, and results showed that the instrument has a high interobserver concordance, as per Table 1.
TABELA 1
The applicability attribute obtained 78% of the option I fully agree in the response of the judges. Clarity was evaluated as high, obtaining 85% of the answers. The relevance attribute achieved the option relevant with 91% of the answers. The IQIS was considered validated with an excellent agreement for the attributes of clarity and relevance, with k=0.82 and k=0.92, respectively, and good agreement for the applicability attribute, with k=0.78. Suggestions to change the index texts (30%) were analyzed and accepted and gave rise to the second version of IQIS, adjusted to the opinion of the Committee. The results of the application were analyzed as to the rate of agreement between them. Of the 224 responses of each evaluator, 115 had an agreement of less than 80%, which were reevaluated by the researchers and adjusted, giving rise to the second version. Following content validation, the IQIS was restructured with 12 realms, 41 modules, 57 indicators (30 critical and 27 non-critical), 252 items and 1,512 indices.
The Kruskal-Wallis test was applied to the data resulting from the evaluation of the seven nutritionists in the Test and showed no statistically significant difference (p=0,423) between the evaluations obtained for both the instrument as a whole and its realms. The items were evaluated according to the risk for the maintenance of hygienic-sanitary quality, classified in the 12 realms, as shown in Table 2.
TABELA 2
When calculating the proportion of adequate hygienic-sanitary conditions of the food service and of each realm, we observed that the IQIS instrument allowed us to identify the items that required correction in order to comply with the legal requirements, since there was agreement on the answers indicated. There was no statistically significant difference between the evaluations of nutritionists by the Kruskal-Wallis test, showing that the instrument is reliable and reproducible. Using the same database, the statistical analyses of the Cronbach Alpha and the Intraclass Correlation Coefficient by realm were performed to verify IQIS’ reliability, which evidenced acceptable reliability. The Kruskal-Wallis test showed that there was no statistically significant difference between the evaluations (p=0.423), the Intraclass Correlation Coefficient was satisfactory (ICC=0.53), and the Cronbach\'s Alpha was considered acceptable (α=0.71).
The Test’s database was used, and the mathematical formalism of the MARP was applied to each realm of the IQIS to evaluate the potential risk range of the food and nutrition service. The rates of compliance were analyzed to evaluate its effectiveness, in which the responses indicated in indices 3, 4 and 5 were “compliant”. Table 3 shows the classification of each realm vis-à-vis the potential risk variation range, as acceptable, tolerable or unacceptable.
TABELA 3
The results of the application of the IQIS showed that the food and nutrition service evaluated had a mean compliance rate of 35% in the evaluation of the risk control indicators and the potential risk classified in the range of risk variation was deemed unacceptable.
DISCUSSION
In the current Brazilian social and economic context, health control measures require risk management of hygienic-sanitary conditions of food and nutrition services as an essential factor to reduce the incidence of FBDs. Thus, the preparation and validation of a quantitative instrument for inspection of large food and nutrition services, based on the MARP, support prevention and control measures. The IQIS was developed and tested for its validity to corroborate this, with consistent results that allow the immediate application. Its elaboration considered the RDC Nº 216/2004 in full, which contributes to standardized terminology, besides ensuring that all national legal requirements be considered. Similar studies, such as the Food Service Categorization Assessment List (18) and the Best Practices in School Food (BPAE) (16), used methods to prioritize items in RDC Nº 216/2004, which were more relevant to health risk control, as well as developed their respective risk classification methods at the end of the application of the lists. Despite the different methodologies for classification of risk control indicators as critical and non-critical, a similarity between the IQIS critical indicators and the Food Service Categorization Checklist is observed (18). The MARP applied to food and nutrition services resembles HACCP in its purpose of ensuring food security, but evidences different methodological aspects, since it is used to analyze potential hazards of operations, based on the concept of probabilistic risk. On the other hand, the MARP works with the potential risk, regarding a possible health problem, without necessarily describing the problem and its probability. It is a concept that expresses the value judgment about the potential exposure to a possible risk (19), and this is an advance in food safety assurance.
Although it is usual to use the health inspection instrument to collect nonconformities in food and nutrition services, it is necessary to use methods to classify the risk control indicators, defining their respective criticalities, for the adequate health risk management. Table 3 data analysis showed that the proportion of nonconformities in each realm is not always equivalent to their level of criticality. We observed that, concerning realms VII. Raw materials, Ingredients, and packaging; VIII. Food preparation; and X. Exposure to the consumption of the prepared food, while more than 60% of their items were evaluated with appropriate hygienic-sanitary conditions, they were classified with a potentially unacceptable risk and jeopardized the whole system. The MARP facilitates the comparison of the potential risks evaluated, guiding crucial stages of risk management, as well as implementing corrective measures in cases of failure. A similar study evaluates the risk classification of food and nutrition services as a reliable strategy for risk communication and food security promotion, significantly contributing to reduced foodborne diseases (40).
The validation of quantitative instruments, with a view of risk management for food safety control in collective food and nutrition, is useful and gains substantial significance by supporting various professionals in the field of research and practice, enabling an accurate view of the most critical issues, without losing sight of the other items required by law. IQIS has been shown to be a viable application instrument, with a reduced level of interference of the evaluator due to the objective response options, appropriate for a detailed and thorough inspection, which takes time to complete in large food and nutrition services. It should be noted that this time difference to implement the IQIS and dichotomous instruments is not significant, which reinforces its applicability by private nutritionists, as well as by regulatory bodies linked to health surveillance. The instrument is available to the entire scientific community and may be requested from authors by sending an e-mail to lilianmferrari@gmail.com.
The lack of validation studies involving quantitative evaluation tools for food and nutrition services – which include all the items required in RDC No. 216/2004 – and of a gold standard appear as hindrances to the design of this study, which led to the combination of several methodological strategies to ensure the validation of the IQIS. Among the main limitations of this research is that sample size was restricted by the need of the group of evaluators to apply the IQIS concurrently, since the reality of food production is dynamic and changes its hygienic-sanitary conditions in the respective sectors as per the production flow throughout the day.
CONCLUSIONS
The results of the IQIS evidence content validity and it can be used with good reliability and reproducibility by nutritionists to evaluate the hygienic-sanitary conditions of the large food and nutrition services and to manage the priority risks, supporting the adoption of best practices of food handling and favoring the prevention of diseases. The contributions evidenced the relevance of developing the IQIS, taking into account the need for nutritionists to have a specific technological innovation for the segment of collective feeding that meets the requirements of RDC Nº 216/2004 and allows health risk management, covering all its complexity. Similar work will generate knowledge and tools to ensure health risk management in food and nutrition services, thus contributing to the provision of safer food with a lower probability of foodborne diseases. An essential benefit of the work is the appropriate management of resources in the hierarchy of improvements, fundamental in the current socioeconomic context of crisis and scant resources. We recommend that the technical application of IQIS be extended to other food and nutrition services, as well as its possible be considered for use by regulatory agencies such as Health Surveillance. As a perspective for future studies, we suggest the preparation of instructional material for a better understanding of the instrument by professionals employing the IQIS and a more objective and appropriate version for application in realities with reduced availability of resources or health emergency situations.

BIBLIOGRAPHY
1. Brasil. Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento das Doenças Transmissíveis. Vigilância Epidemiológica das Doenças Transmitidas por Alimentos - VE - DTA. Brasília: Editora do Ministério da Saúde; 2014.
2. Almeida de OA, Minéia Daniel de PC, Capalonga R, Ribeiro de ICM, Cesar TE. Doenças transmitidas por alimentos, principais agentes etiológicos e aspectos gerais: uma revisão. Revista HCPA [Internet]. 2010 [cited 6 June 2018];30(3):279-285. Available from: http://hdl.handle.net/10183/157808
3. Gould LH, Walsh KA, Vieira AR, Herman K, Williams IT, Hall AJ, Cole D .Surveillance for foodborne disease outbreaks - United States, 1998-2008. Morbidity and Mortality Weekly Report: Surveillance Summaries [Internet]. 2013 [cited 6 June 2018];62(2):1-34. Available from: https://www.ncbi.nlm.nih.gov/pubmed/23804024
4. United States.Centers for Disease Control and Prevention (CDC). US Department of Health and Human Services. Surveillance for Foodborne Disease Outbreaks. Atlanta: Annual Report: 2014; 2016.
5. Brasil. Ministério da Saúde, Secretaria de Vigilância em Saúde. Vigilância Epidemiológica das Doenças Transmitidas por Alimentos. São Paulo: Editora do Ministério da Saúde 2015
6. Souza E, Silva C, Sousa C. Bacteriocins: molecules of fundamental impact on the microbial ecology and potential food biopreservatives [Internet]. 2005 [cited 6 June 2018]. Available from: http://www.scielo.br/pdf/babt/v48n4/25718.pdf
7. Brasil. Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento de Vigilância Epidemiológica. – Brasília: Editora do Ministério da Saúde; 2010
8. Greig J, Ravel A. Analysis of foodborne outbreak data reported internationally for source attribution. International Journal of Food Microbiology [Internet]. 2009 [cited 6 June 2018];130(2):77-87. Available from: https://www.ncbi.nlm.nih.gov/pubmed/19178974
9. Scallan E, Griffin P, Angulo F, Tauxe R, Hoekstra R. Foodborne Illness Acquired in the United States—Unspecified Agents. Emerging Infectious Diseases [Internet]. 2011 [cited 6 June 2018];17(1):16-22. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3204615
10. Codex AC. Hazard analysis and critical control point (HACCP) system and guidelines for its application. Annex to CAC/RCP. 1997;1(1969):Rev 1.
11. Virgínia MSA, Ricardo SDP. Programa Boas Práticas de Fabricação. Recife: EDUFRPE; 2012.
12. Brasil. Portaria Ministério da Saúde Nº 1.428, de 26 de novembro de 1993. Aprova o regulamento técnico para inspeção sanitária de alimentos, as diretrizes para o estabelecimento de boas práticas de produção e de prestação de serviços na área de alimentos. Diário Oficial da União 1993; 02 dez
13. Brasil. Resolução RDC n°216, de 15 de setembro de 2004. Dispõe sobre Regulamento Técnico de Boas Práticas para Serviços de Alimentação. Diário Oficial da União 2004; 15 set
14. São Paulo. Portaria CVS N º5, de 09 de abril de 2013. Aprova o Regulamento Técnico sobre Boas Práticas para Estabelecimentos comerciais de alimentos e para serviços de alimentação, e o roteiro de inspeção. Diário Oficial do Estado de São Paulo 2013; 09 abr
15. Mello A, Carmo C, Leite S, Miguel M, Colares L. Elaboração, validação de conteúdo e da confiabilidade do instrumento para avaliação higiênico-sanitária de serviços de alimentação. Vigilância Sanitária em Debate [Internet]. 2014 [cited 6 June 2018];2(3). Available from: https://www.researchgate.net/profile/Cleber_Carmo/publication/273992644
16. Stedefeldt E, Cunha D, Silva Júnior Ê, Silva S, Oliveira A. Instrumento de avaliação das Boas Práticas em Unidades de Alimentação e Nutrição Escolar: da concepção à validação [Internet]. 2013 [cited 6 June 2018]. Available from: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232013000400006
17. Brasil. Portaria Nº 817, de 10 de maio de 2013. Aprova as diretrizes nacionais para a elaboração e execução do projeto-piloto de categorização dos serviços de alimentação para a Copa do Mundo FIFA 2014. Diário Oficial da União 2013; 10 maio.
18. Brasil. Agência Nacional de Vigilância Sanitária. Categorização dos Serviços de Alimentação: Elaboração e Validação da Lista de Avaliação. São Paulo: Editora do Ministério da Saúde; 2013
19. Leite DLH, Navarro M. Risco potencial: um conceito de risco operativo para vigilância sanitária. In: Alves Costa E. Vigilância Sanitária: temas para debate. Salvador: EDUFBA; 2009. p. 61-82
20. Leite DLH. Vigilância sanitária em serviços de saúde: risco e proteção da saúde em serviços de hemodiálise [Tese]. Salvador: Universidade Federal da Bahia; 2007
21. Spexoto AA. Aplicação do sistema de análise de perigos e pontos críticos de controle (APPCC) em propriedades leiteiras [Tese]. São Paulo: Universidade de São Paulo; 2003
22. Navarro VTNM. Risco, Radiodiagnóstico e Vigilância Sanitária. Salvador: EDUFBA; 2009.
23. Tobias W, Ponsano E, Pinto M. Elaboração e implantação do sistema de análise de perigos e pontos críticos de controle no processamento de leite pasteurizado tipo A [Internet]. 2014 [cited 6 June 2018]. Available from: http://www.scielo.br/scielo.php?pid=S0103-84782014000901608&script=sci_abstract&tlng=pt
24. Belik W. Perspectivas para segurança alimentar e nutricional no Brasil. Revista Saúde e Sociedade [Internet]. 2003 [cited 6 June 2018];12(1):.12-2. Available from: http://www.scielo.br/pdf/sausoc/v12n1/04.pdf
25. Cavalli S, Salay E. Gestão de pessoas em unidades produtoras de refeições comerciais e a segurança alimentar [Internet]. 2007 [cited 6 June 2018]. Available from: http://www.scielo.br/scielo.php?pid=S1415-52732007000600008&script=sci_abstract&tlng=pt
26. Figueiredo A, Miranda M. Análise de Risco aplicada aos alimentos no Brasil: perspectivas e desafios [Internet]. 2011 [cited 6 June 2018]. Available from: http://www.scielo.br/scielo.php?pid=S1413-81232011000400024&script=sci_abstract&tlng=pt
27. Kepple A, Segall-Corrêa A. Conceituando e medindo segurança alimentar e nutricional [Internet]. 2011 [cited 6 June 2018]. Available from: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232011000100022
28. Maluf SR, Menezes F, Valente LF. Contribuição ao Tema da Segurança Alimentar no Brasil. Revista Cadernos de Debate [Internet]. 1996 [cited 7 June 2018];IV:66-88. Available from: https://www.sswm.info/sites/default/files/reference_attachments/MALUF%20et%20al%20(1996).pdf
29. Hernández SR, Fernández CC, Baptista LP. Metodología de la investigación. 4th ed. México: McGraw-Hill; 2007.
30. D\'Ancona CA. Metodología cuantitativa: estrategias y técnicas de investigación social. Síntesis; 1999
31. Polit D, Hungler B, Palacios MR, Féher de la TG. Investigación científica en Ciencias de la Salud. México, D.F.: McGraw-Hill Interamericana; 2007.
32. Coluci M, Alexandre N, Milani D. Construção de instrumentos de medida na área da saúde. Ciência & Saúde Coletiva [Internet]. 2015 [cited 7 June 2018];20(3):925-936. Available from: http://www.scielo.br/pdf/csc/v20n3/1413-8123-csc-20-03-00925.pdf
33. Cooper D, Schindler P. Métodos de Pesquisa em Administração. 12th ed. São Paulo: McGraw Hill; 2016.
34. Landis JR, Koch GG. The measurement of observer agreement for categorical data. 1st ed. Biometrics. 1977. p. 159-74.
35. IBM Corp.IBM SPSS Statistics [Windows]. Versão 21.0. New York; 2012.
36. Marôco J. Análise Estatística com Utilização do SPSS. 3rd ed. Edições Sílabo; 2007.
37. Wynd C, Schmidt B, Schaefer M. Two Quantitative Approaches for Estimating Content Validity. Western Journal of Nursing Research [Internet]. 2003 [cited 7 June 2018];25(5):508-518. Available from: https://www.ncbi.nlm.nih.gov/pubmed/12955968
38. Streiner D, Norman G. Health measurement scales. Oxford: Oxford University Press; 2015.
39. Hair J, Black W, Babin JB, E. Anderson R. Análise multivariada de dados. 6th ed. Bookman; 2009.
40. Cunha, DT, Freitas S, Tondo AL, Oliveira EC, Ginani AB, Araújo VC. Inspection score and grading system for food services in Brazil: the results of a food safety strategy to reduce the risk of foodborne diseases during the 2014 FIFA World Cup. 7th ed. Frontiers in microbiology; 2016.