Artigos

Conteúdo (article):

Content validity and reliability of a university food environment assessment instrument


Amanda da Silva Franco,1 Daniela Silva Canella,2 Letícia Ferreira Tavares,3 Alessandra da Silva Pereira,4 Roseane Moreira Sampaio Barbosa,5 Gilson Irineu de Oliveira Junior,6 Claudia Roberta Bocca dos Santos,4 Inês Rugani Ribeiro de Castro2

1 Nutrition Coordination, Health Sciences Center. Serra dos Orgãos University Center
2 Institute of Nutrition.State University of Rio de Janeiro
3 Institute of Nutrition Josué de Castro. Federal University of Rio de Janeiro
4 School of Nutrition. Federal University of the State of Rio de Janeiro
5 Department of Nutrition. Fluminense Federal University

Corresponding author
Amanda da Silva Franco
amandafranco@unifeso.eud.br

Abstract
This study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment. A checklist was developed to assess establishments that sell food and beverages in the university environment. The content validation encompassed the development of the instrument, expert evaluation and pretest performance. Reliability was evaluated using a convenience sample (n = 64) of establishments distributed across seven campuses of three public universities and was carried out using interobserver (IO) and test-retest (TR) evaluations. Categorical and count variables were analyzed by calculating the percentage agreement (PA), kappa coefficient (k) and prevalence-adjusted, bias-adjusted kappa (PABAK), and continuous variables were analyzed by the intraclass correlation coefficient (ICC). The checklist consisted of 204 items distributed in seven domains. The instrument\'s performance was considered excellent or very good for 91.3% (PA) of the items when evaluated. For IO, 68.3% (k) and 96.5% (PABAK) had excellent, very good or good agreement, while for TR, 65% (k) and 96.5% (ka) had excellent agreement. The instrument showed satisfactory content validity and reliability for characterizing the food environment at Brazilian universities.

Descriptors: Psychometric Properties, Reproducibility, Higher education Institutions


Introduction
Efforts have been made to translate concepts and constructs about the food environment into measurement tools that can generate empirical evidence.1-6 The consumer food environment,1 which was the object of interest in the present study, includes factors related to the food itself, i.e., the way in which it is provided or presented and the price, nutritional quality and nutritional labeling (including nutrition claims), among other factors. The consumer\'s food environment may be inserted into organizational environments, such as companies and universities.1 In the literature, the most commonly reported method for evaluating the consumer food environment is the auditing of establishments that sell food. Researchers evaluate the variety of products, nutritional information, healthy options, prices, advertisements and other aspects of food.2,7-9
The university food environment, which was the scenario of interest of this study, can be characterized by the availability of foods, preparations and beverages (FPB), physical and financial accessibility, the promotion of FPB, nutritional information and advertising within the campus and may also include the campus surroundings. It is known that this environment influences the eating habits of the individuals who are exposed to it,10 since it can act as a facilitator of or barrier to healthy choices1,10 given that meals consumed at the university may be an important part of the diet of students, teachers and employees.11,12
As of 2021, Brazil has 302 public higher education institutions (HEIs) and 2,306 private HEIs, which together comprise 8,603,824 students and 386,703 teachers,13 in addition to thousands of technical-administrative employees. Therefore, these locations should be valued as a strategic setting for promoting healthy eating.
The literature on the evaluation of the university food environment is still expanding. Countries such as the United States and Australia have published studies on the characterization of the food environment at universities and their surroundings.14,15 In Brazil, to date, few such studies have been found.16-19
The selection or development of a data collection instrument is an essential step that, if not performed with care and methodological rigor, can be a source of error and misleading conclusions.20 Although many studies have been conducted to evaluate the food environment, a small portion of them have assessed the psychometric properties of the instruments used in them.21,22 Among the studies on the university food environment, only two analyzed the reliability of the instrument used,14,16 and none evaluated its validity. In addition, of the instruments that have been identified to date, none is able to fully capture the specificities of the Brazilian university food environment or highlight the extensive variety of convenience items and the existence of mixed establishments (that have characteristics of both restaurants and cafeterias). In addition, these instruments do not use the NOVA classification of foods, which has been adopted in the Brazilian guidelines for public policies on food and nutrition23 and studies on food, diet quality and health24, to characterize commercial foods and preparations.25
Considering the importance and specificities of the university food environment, the scarcity of instruments for its evaluation and the scarcity of evaluations of the performance of these instruments in different contexts, the present study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment.

Methods
The present study is an initiative of the Collaborative Group for Studies on the University Food Environment (Grupo Colaborativo de Estudos sobre o Ambiente Alimentar Universitário (CALU) in Portuguese), which is composed of professors from public universities located in the state of Rio de Janeiro that have a course in nutrition, namely, the Universidade do Estado do Rio de Janeiro (UERJ), Universidade Federal do Rio de Janeiro (UFRJ), Universidade Federal do Estado do Rio de Janeiro (UNIRIO) and Universidade Federal Fluminense (UFF).

Content validation
The content validation of the instrument involved three stages: the development of the instrument, its evaluation by experts26,27 and the performance of a pretest.

Development of the instrument
The preparation of the instrument was based on two theoretical frameworks: the approach of Caspi et al.9 regarding access to food and the approach of Glanz et al.1 regarding the consumer food environment. Of the five dimensions proposed by Caspi, those that could fully apply to commercial establishments located in universities were considered, namely, availability (including the variety of food available), affordability (including the price of food) and convenience (e.g., open hours, payment methods accepted).9 These dimensions were complemented by two components proposed by Glanz: nutritional information and food promotion.1
The first stage of construction of the instrument involved mapping the existing instruments for the evaluation of the consumer\'s food environment and the selection of two that closely matched the characteristics of the food environments of Brazilian public universities.14,17 The elaboration of the instrument encompassed three phases, which will be described in detail in the results section: the identification of the domains that should compose the instrument (e.g., characterization of the establishment, available foods, prices, advertisements), the preparation of the items and the structuring of the instrument. During this process, the inclusion of foods based on the recommendations of the Dietary Guidelines for the Brazilian Population23 and international guidelines,28 the NOVA classification of foods25 and foods commonly found in universities14,17 was guaranteed. The items included in the checklist comprised fresh or minimally processed foods, processed foods and culinary preparations based on these foods, as well as ultra-processed foods and culinary preparations containing these foods. Together, these are markers of healthy and unhealthy eating.

Expert evaluation
For the evaluation of the instrument by experts, the literature suggests the participation of five to 20 individuals.26,27 In the present study, 12 researchers in the field of nutrition from different institutions were invited. Their expertise was in epidemiology, public health, collective food or gastronomy, and they had experience in topics related to the study and/or construction of instruments.
The preliminary version of the instrument was qualitatively evaluated and was performed in two stages: in the first phase, the proposed domains were evaluated, and in the second phase, the items were evaluated, including their relevance and clarity. Regarding relevance, whether each item expressed the expected concept, its relevance and its adequacy for the proposed objectives were evaluated. Regarding clarity, whether the wording of each item expressed the concept in a way that was understandable and consistent with what was expected to be measured was evaluated.
The experts also evaluated the instrument as a whole, examining its scope, i.e., whether each domain was adequately covered by the set of items presented and whether the dimensions proposed by Caspi et al.9 and the components of the food environment proposed by Glanz et al.1 that were expected to be covered by the instrument had been adequately considered.

Pretest
The final step was a pretest of the instrument. The pretest was conducted by the first author of this article at a commercial establishment located near one of the universities studied that had characteristics similar to those of the establishments of the universities participating in this study.

Assessment of the reliability of the instrument
To evaluate the reliability of the instrument, a convenience sample of establishments with different characteristics distributed over seven campuses of three universities was studied, namely, UFRJ: Cidade Universitária, Praia Vermelha and Professor Aluísio Teixeira (Macaé) campuses; UFF: Gragoatá and Valonguinho campuses; and Unirio: Pasteur and Biomedical Institute campuses. Based on publications on the topic,14,17 it was determined at least 50 establishments should be included. All establishments on the campuses of the universities that agreed to participate in the study were included.
Data collection was performed by 15 trained field researchers (nutrition and gastronomy students) using a printed form. The training lasted six hours and was performed in two stages: first, a theoretical stage, in which the purpose of the study, the instrument to be applied and the data collection manual were presented; and second, a practical stage, in which the instrument was applied by the researchers in at least two establishments under the supervision of the study coordinators.
The establishments included in the study were evaluated three times: two researchers (A and B) visited each establishment on the same day at a maximum interval of 30 minutes, and one of the researchers from each pair (A or B) performed a new data collection visit 15 to 30 days after the first evaluation. Data collection occurred between November 2015 and February 2016, which was an academic period at the universities.
Data entry was performed by double entry using Excel software version 2016, which was also used to evaluate the consistency of the database.
The reliability of the instrument was determined by interobserver and the test-retest evaluations.20 For the categorical and count variables, the stability of the instrument was evaluated by calculating the percent agreement (PA)29 and the kappa index (k), kappa index adjusted for prevalence and interviewer bias (ka) (the prevalence-adjusted, bias-adjusted kappa (PABAK)) for all variables.30 For food prices, which were analyzed as continuous variables, the intraclass correlation coefficient (ICC)20 was calculated. The PA, k, ka and ICC values were classified according to the criterion of Byrt:31 excellent agreement (≥0.92), very good agreement (0.80 to 0.91), good agreement (0.60 to 0.79), fair agreement (0.40 to 0.59), slight agreement (0.20 to 0.39), poor agreement (0.20 to 0.39), and no agreement (< 0.00).
All analyses were performed using the Statistical Package for Social Science (SPSS 21) software. To calculate the PABAK and the ICC, it was necessary to create a specific macro.
The study was submitted to the Ethics Committee of the Pedro Ernesto Hospital of the State University of Rio de Janeiro and approved under opinion number 49988015.6.0000.5259.

Results

Content validation
In general, the content validity of the developed instrument was considered satisfactory by the experts, considering that the proposed domains and items included the conceptual dimensions of interest and that these dimensions covered the essential elements of the consumer food environment in the university context.
Based on the experts’ assessment, items were changed or added to or removed from the checklist. The main changes were inclusion of a new domain (information); the removal of a domain (review of the menu); reorganization of the “environmental observation” domain and the inclusion of 13 items in it; addition of the category “convenience items” to the “beverages, foods and preparations” domain and the inclusion of eight new types of ultra-processed beverages; inclusion of the form of presentation of advertisements in the “advertisements” domain.
The results of the pretest did not indicate a need for changes in the content or form of the instrument. All items were suitable for completion, and no situation of interest was identified that was not covered by the instrument, nor were any unnecessary items noted. The ordering of the modules in the instrument, as well as its formatting, was also shown to be adequate.
The final version of the developed instrument consisted of a checklist that covered the dimensions availability, affordability, convenience,9 nutritional information and food promotion1 and included 204 items distributed across seven different domains, namely, characterization of the establishment; observation of the environment; information made available; beverages, foods and preparations; prices and promotions; availability of healthy substitutions; and advertisements (full text of the instrument is available athttp://www.observatoriodeobesidade.uerj.br/?p=3294), as detailed below:
● Characterization of the establishment: type and location of the establishment; types of food offered; days and hours of operation and most frequent customers; forms of payment.
● Observation of the environment: availability of specific items (microwave, structures for on-site consumption, open-access shelves and counters displaying certain items, display refrigerators and freezers, available water dispenser and free filtered water supply) and organization of buffets (order of presentation).
● Information available: availability of information to the customer (menu, price, nutritional information) and form of presentation of this information (on the preparation or product, table display, banner, internet).
● Beverages, food and preparations:
o Preparations, meals, snacks and other: availability of complete meals (portioned or a la carte); brown rice; legumes; fruits and fruit salads; raw and cooked vegetables; natural seasonings and sauces; natural fruit juices and soft drinks; savory snacks and sandwiches; and other items (water, table sugar and sweetener).
o Convenience items: availability and variety of the following convenience items (i.e., different types of the same item): ultra-processed foods (candies, bonbons, sweet cookies with and without filling, packaged snacks, whole cookies, cereal bars, breakfast cereal), ultra-processed beverages (flavored water, coconut water, soft drinks, fruit juice or fruit nectar drinks, ready-made teas, isotonic beverages, energy tea, soy-based drinks, guarana soft drinks, flavored milk or milk or yogurt beverages, alcoholic beverages, self-service coffee) and elaborate sweets (honey bread, cake, brigadeiros).
● Prices and promotions: prices of the smallest portions available for each of the foods, or, when available, the price of meals by the kilogram; supply and prices of promotions/combos (e.g., foods/products plus an ultra-processed beverage), supply and price of larger portions of certain products.
● Supply of healthy substitutions: options to replace white rice with brown rice and French fries with salads or vegetables in meals or combos.
● Advertising: existence of advertisements for fruits and vegetables and ultra-processed beverages, ice cream and desserts; number, presentation and messages of advertisements.
The types of establishments were classified according to their predominant activity: by-the-kilo restaurants (essentially sells meals and foods by weight); a la carte/portioned/executive-style restaurants (essentially sells meals at a fixed quantity and price); free/rotating restaurant (essentially sells meals using a free (unweighed) or rotating system (variable quantity/weight and a fixed price) system; snack bar (sells mainly snacks/sandwiches and savory snacks and candy items (candy, sweets, ultra-processed beverages); bar (mainly sells alcoholic beverages, cigarettes, candies, snacks and food items); cafeteria (sells coffee and other beverages and, sometimes, foods that can be eaten quickly or light dishes); bomboniere (sells sweets, candies, chewing gum, chocolates, soft drinks, juices, sweets in general and ice cream); and mixed establishment: snack bar/cafeteria + meals by the kilo or snack bar/cafeteria + a la carte meals/portioned items/executive-style (sells different typical cafeteria products with portioned menu items or meals sold by weight).32
To categorize the beverages included in the instrument, the grouping used by Euromonitor International33 for sugary or sweetener-based beverages was used and adapted for the Brazilian market. The adopted categories were flavored (bottled) waters, carbonated or uncarbonated (they contain natural or artificial fruit flavors, usually do not contain artificial colors, may or may not contain sugar); soft drinks: cola and non-cola beverages, regular or low calorie; liquid or powder concentrates for reconstituted beverages; 100% fruit juices; juice-based beverages; nectars; ready-to-drink teas (e.g., iced-tea, green tea and mate with added sugar or other sweeteners); isotonic or replenishing beverages (e.g., sport drinks); energy drinks, generally those containing caffeine and taurine; guarana soft drinks; soy-based beverages, plain or flavored; coconut water; milk-based beverages: whey-based dairy beverages, flavored milks, and mixed milk and fruit preparations, with or without cereals or other items.
Food advertisements were categorized according to their form of presentation. Any word or phrase about the food/product that was presented in a banner/poster produced by the supplier, a banner/poster produced by the establishment, on employees’ clothing, on the product (advertising added to product packaging), on a replica of the product, on display in the establishment or even on the menu was considered.

Reliability of the instrument
A total of 64 establishments were included in the first phase of data collection, and 54 were included in the second round (10 refused to participate). The average duration of data collection at each establishment was 18 minutes.
Regarding the performance of the instrument, excellent or very good agreement was observed for 91.3% of the items evaluated. Regarding the interobserver test, for among the evaluated items (158), 68.3% and 96.5% had excellent, very good or good agreement according to the k (range of -0.51 to 1.00) and ka (range from 0.44 to 1.00), respectively (Table 1). Poor or null agreement was observed for 12% and 0.6% of the items according to the k and ka, respectively; these items were in the domains of characteristics of the establishment, healthy substitutions and nutritional information. For the items in the food prices domain, the ICC ranged from 0.05 to 1.00. Of the evaluated items (n = 28), 88% showed excellent, very good or good agreement (Table 2). In this domain, 23% of the convenience items showed poor agreement.
In the test-retest evaluation, of the evaluated items (158), 65% had excellent, very good and good agreement according to k, which ranged from -0.05 to 1.00. Regarding the ka, which ranged from 0.37 to 1.00, 96.5% of the items showed excellent, very good or good agreement (Table 3). Considering k, there was poor or null agreement for 4.8% of the items, which were concentrated in the domains of environmental characteristics; beverages, foods and preparations; and information. For ka, there was no poor or null classification of any item. For the items in the food prices domain (n = 28), the ICC ranged from 0.05 to 1.00; 77% showed excellent, very good or good agreement, and none of the items had poor or null agreement (Table 2).

Discussion
The results indicated satisfactory content validity and reliability (interobserver and test-retest) of the instrument for characterizing the consumer food environment at universities in the Brazilian context. Even for the test-retest evaluation, for which some of the differences can be explained by real variations in the environment, particularly with regard to food supply, good performance of the instrument was observed.
Although the performance of the instrument was generally satisfactory, some items showed low performance, especially in the interobserver reliability evaluation. This result suggests the need to improve the training provided for the application of the instrument. A similar situation was observed in a study that evaluated a checklist applied in a hospital food environment.34 The food environment is a complex context with significant diversity of establishments, foods, beverages and information, and thorough and standardized evaluation through the appropriate selection and training of evaluators is necessary.4 Although the results of the present study were good, the duration of training was shorter than that reported for other studies.3,35
Few studies in the literature were identified that determined the content validity of instruments for assessing the food environment, and none involved the university food environment. A review on the assessment of the food environment that included 432 studies published between 2007 and 2015 showed that 28.1% evaluated the validity of the instrument used, and 2.1% assessed the content validity of the instrument.22 Among those studies, that of Pomerleau et al.36 is noteworthy, as it conducted a content validation procedure similar to that used in the present study; for the development of the instrument it used (the EURO-PREVOB Community Questionnaire), a literature review, expert consultations and a pretest were performed.33
In the international and national literature, there are few studies on the reliability analysis of instruments for assessing the food environment in universities. In addition, because the existing studies use different instruments to evaluate the environment and because reliability is a context-dependent measure,37 the existing studies are not fully comparable with the present evaluation. Nevertheless, it is worth commenting on their results. In Australia, Roy et al.15 developed a checklist for the evaluation of food availability, physical accessibility (e.g., products near the checkout counter, free access to shelves) and food promotions. The authors performed an interobserver reliability analysis through percentage agreement for two establishments and found 96.0% agreement.15 In Brazil, Rodrigues et al.17 adapted a checklist to evaluate the university food environment that included items related to the description of the establishments and the evaluation, the availability of foods/preparations and facilitators and barriers to healthy eating. The interobserver and test-retest reliability was measured by kappa for 64 establishments. For interobserver agreement, the kappa ranged from -0.04 to 1.0. For the test-retest agreement, a variation of 0.06 to 1.0 was observed. The mean kappa index was 0.54 for interobserver agreement and 0.72 for test-retest agreement.17
It should also be noted that some instruments have been developed with the objective of evaluating the food environment in other scenarios, such as the community or other organizational spaces, such as work and school environments.3,37,38 In Brazil, it is worth mentioning three studies conducted in the state of São Paulo that evaluated the reliability of instruments for evaluating the food environment of restaurants3,4 and food retailers.3,37,38 The authors evaluated the interobserver and test-retest reliability using the kappa and ICC and generally found good reliability for both tools.3,4,38 Our study adds to these findings, providing evidence for an instrument that addresses the specificities of the university food environment, including establishments that sell food for immediate consumption (and thus have characteristics similar to those of restaurants and snack bars) but also sell items such as packaged snacks and cookies that are typically sold in supermarkets/markets/grocery stores and are not usually available in restaurants. The instrument evaluated in the present study assesses all types of food available at the university and avoids the need to use two different instruments.
Although most studies on the university food environment have been conducted in public HEIs, given the similarities between the contexts of the food environments of public and private universities, the instrument presented here may also be useful for the latter.
The study has some limitations. Because it used a convenience sample of public universities, its external validity is limited. In addition, although the elaboration of the instrument included consultation with experts in the field, quantitative measures were not used to verify the content validity. Another aspect that warrants mention is the fact that the data collection process lasted three months; therefore, it is possible that seasonal differences in food supply were not captured. Consequently, the possible implications of seasonality, if present, for the performance of the instrument are unknown. A fourth limitation is the fact that, for some of the items that were used to calculate kappa and PABAK, these statistics could not be computed due to the absence of one or more levels in the cross-referenced tables. However, this affected only a small portion of the items (approximately 10%). Thus, the generally good performance of the other items suggests that the performance of these items, if calculated, would not significantly impact the overall good performance of the instrument.
Among the strengths of this study is the evaluation of test-retest reliability, since most evaluations of audit tools only evaluate interobserver reliability at a single moment.15 Another strength of the study is the broad scope of the adopted instrument. In addition to being a compilation and adaptation of elements from other instruments to better reflect the characteristics of the Brazilian university food environment, the present instrument was conceived and evaluated by researchers with different training profiles and who worked on university campuses with different characteristics, such as being located in the state capital vs. other municipalities and in an urban center vs. in a neighborhood outside the urban center; having two (morning and afternoon) vs. three (morning, afternoon and night) shifts; and having fewer vs. more structured establishments. Another aspect to highlight is the use of ka, which is still infrequently used in the health literature. In addition, the application of the instrument was tested on university campuses that had different characteristics.
Conclusion
The proposed instrument performed well in the context in which it was tested. It can support the characterization of the food environment in both public and private universities (and their surroundings, if the establishments have characteristics similar to those present on the university campuses), monitor their dynamics over time and support interventions for their improvement.

Acknowledgements
We are grateful to all undergraduate students involved in the data collection for this study and to the members of the Collaborative Study Group on University Food Environment.
Funding
Center on Food and nutrition in public policies. Institute of Nutrition, State University of Rio de Janeiro.

Author contributions
Conceptualization. ASF, IRRC; investigation. ASF, LFT, ASP, RMSB, GIOJ, CRBS; supervision. IRRC, DSC; writing—original draft preparation. ASF, DSC, IRRC; writing—review and editing. ASF, LFT, ASP, RMSB, GIOJ, CRBS, DSC, IRRC. All authors have read and agreed to the published version of the manuscript.


References
1. Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: concepts and measures. Am J Health Promot 2005; 19(5):330-333, ii.
2. Bethlehem JR, Mackenbach JD, Ben-Rebah M, Compernolle S, Glonti K, Bárdos H, Rutter HR, Charreire H, Oppert JM, Brug J, Lakerveld J. The SPOTLIGHT virtual audit tool: a valid and reliable tool to assess obesogenic characteristics of the built environment. Int J Health Geogr 2014; 13:52.
3. Duran AC, Lock K, Latorre Mdo R, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.
4. Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53:91.
5. Altenburg T, Te Velde S, Chiu KJ, Moschonis G, Manios Y, De Bourdeaudhuij I, Vik FN, Lien N, Brug J, Chinapaw M. Interrater reliability of the ENERGY photo-rating instrument for school environments related to physical activity and eating. J Phys Act Health 2016; 13(4):433-439.
6. Giskes K, Van Lenthe FJ, Brug J, Mackenbach JP, Turrell G. Socioeconomic inequalities in food purchasing: the contribution of respondent-perceived and actual (objectively measured) price and availability of foods. Prev Med 2007; 45(1):41-48.
7. Glanz K, Sallis JF, Saelens BE, Frank LD. Nutrition environment measures survey in stores (NEMS-S): development and evaluation. Am J Prev Med 2007; 32(4):282-289.
8. Gustafson A, Hankins S, Jilcott S. Measures of the consumer food store environment: a systematic review of the evidence 2000-2011. J Community Health 2012; 37(4):897-911.
9. Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: a systematic review. Health Place 2012; 18(5):1172-1187.
10. Roy R, Soo D, Conroy D, Wall CR, Swinburn B. Exploring university food environment and on-campus food purchasing behaviors, preferences, and opinions. J Nutr Educ Behav 2019; 51(7):865-875.
11. Perez PMP, Castro IRR, Canella DS, Franco ADS. Effect of implementation of a University Restaurant on the diet of students in a Brazilian public university. Cien Saude Colet 2019; 24(6):2351-2360.
12. Perez PMP, De Castro IRR, Da Silva Franco A, Bandoni DH, Wolkoff DB. Práticas alimentares de estudantes cotistas e não cotistas de uma universidade pública brasileira. Cien Saude Colet 2016; 21:531-542.
13. Ministério da Educação. Instituto nacional de estudos e pesquisas educacionais anísio teixeira (INEP). Resumo técnico do censo da educação superior 2019. Brasília: Ministério da Educação; 2021.
14. Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Health Promot 2013; 27(4):e81-e90.
15. Roy R, Hebden L, Kelly B, De Gois T, Ferrone EM, Samrout M, Vermont S, Allman-Farinelli M. Description, measurement and evaluation of tertiary-education food environments. Br J Nutr 2016; 115(9):1598-1606.
16. Da Silva Franco A, Canella DS, Perez PMP, Bandoni DH, De Castro IRR. Ambiente alimentar universitário: caracterização e mudanças no período de 2011 a 2016 em uma universidade pública brasileira. Rev Nutr 2020; 33:e200058.
17. Rodrigues CB, Monteiro LS, De Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139.
18. Barbosa R, Henriques P, Guerra H, Emerentino J, Soares D, Dias P, Ferreira D. Ambiente alimentario de una universidad pública brasileña: desafíos para promover una alimentación saludable. Rev Chil Nutr 2020; 47(3):443-448.
19. Pulz IS, Martins PA, Feldman C, Veiros MB. Are campus food environments healthy? A novel perspective for qualitatively evaluating the nutritional quality of food sold at foodservice facilities at a Brazilian university. Perspect Public Health 2017; 137(2):122-135.
20. Streiner DL, Norman GR, Cairney J. Reliability. In: Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. USA: Oxford University Press; 2015. p. 179-199.
21. McKinnon RA, Reedy J, Morrissette MA, Lytle LA, Yaroch AL. Measures of the food environment: a compilation of the literature, 1990-2007. Am J Prev Med 2009; 36(4 Suppl):S124-S133.
22. Lytle LA, Sokol RL. Measures of the food environment: a systematic review of the field, 2007-2015. Health Place 2017; 44:18-34.
23. Ministério da Saúde, Secretaria de Atenção à Saúde, Departamento de Atenção Básica. Guia alimentar para a população brasileira. Brasília: Ministério da Saúde; 2014.
24. Monteiro CA, Cannon G, Lawrence M, Costa Louzada ML, Pereira Machado P. Ultra-processed foods, diet quality, and health using the NOVA classification system. Rome: FAO; 2019.
25. Monteiro CA, Cannon G, Levy RB, Moubarac JC, Louzada ML, Rauber F, Khandpur N, Cediel G, Neri D, Martinez-Steele E, Baraldi LG, Jaime PC. Ultra-processed foods: what they are and how to identify them. Public Health Nutr 2019; 22(5):936-941.
26. Lynn MR. Determination and quantification of content validity. Nurs Res 1986; 35(6):382-385.
27. Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ 1998; 317(7153):273-276.
28. World Health Organization. Diet, nutrition and the prevention of chronic diseases. Geneva: WHO; 2003.
29. Luiz RR. Métodos estatísticos em estudos de concordância. In: Medronho RA, Bloch KV, Luiz RR, Werneck GL, organizers. Epidemiologia. São Paulo: Atheneu; 2009. p. 343-369.
30. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46(5):423-429.
31. Byrt T. How good is that agreement? Epidemiology 1996; 7(5):561.
32. Tavares LF, Perez PMP, Dos Passos MEA, De Castro Junior PCP, Da Silva Franco A, De Oliveira Cardoso L, De Castro IRR. Development and application of healthiness indicators for commercial establishments that sell foods for immediate consumption. Foods 2021; 10(6):1434.
33. Euromonitor International. International marketing data and statistics. London, UK: Euromonitor I Messias GM. Ambiente alimentar hospitalar: desenvolvimento e avaliação de confiabilidade de um instrumento na rede pública de uma metrópole brasileira [Tese de Doutorado]. [Rio de Janeiro]: UERJ; 2019.nternational Limited; 2013.
34. Messias GM. Ambiente alimentar hospitalar: desenvolvimento e avaliação de confiabilidade de um instrumento na rede pública de uma metrópole brasileira/Giselle Moura Messias. 2019.
35. Saelens BE, Glanz K, Sallis JF, Frank LD. Nutrition environment measures study in restaurants (NEMS-R): development and evaluation. Am J Prev Med 2007; 32(4):273-281.
36. Pomerleau J, Knai C, Foster C, Rutter H, Darmon N, Derflerova Brazdova Z, Hadziomeragic AF, Pekcan G, Pudule I, Robertson A, Brunner E, Suhrcke M, Gabrijelcic Blenkus M, Lhotska L, Maiani G, Mistura L, Lobstein T, Martin BW, Elinder LS, Logstrup S, Racioppi F, McKee M. Measuring the food and built environments in urban centres: reliability and validity of the EURO-PREVOB community questionnaire. Public Health 2013; 127(3):259-267.
37. Kelly B, Flood VM, Bicego C, Yeatman H. Derailing healthy choices: an audit of vending machines at train stations in NSW. Health Promot J Austr 2012; 23(1):73-75.
38. Martins PA, Cremm EC, Leite FH, Maron LR, Scagliusi FB, Oliveira MA. Validation of an adapted version of the nutrition environment measurement tool for stores (NEMS-S) in an urban area of Brazil. J Nutr Educ Behav 2013; 45(6):785-792.