Artigos

Conteúdo (article):

Indicators for monitoring oral health services in primary care: Content validation and measurability
Indicators for monitoring oral health services in primary care: Content validation and measurement

Raquel Conceição Ferreira. School of Dentistry, Federal University of Minas Gerais.raquelcf@ufmg.br. ORCID: https://orcid.org/0000-0001-8897-9345.
Loliza Luiz Figueiredo Houri Chalub. School of Dentistry, Federal University of Minas Gerais.lolischalub@gmail.com. ORCID: https://orcid.org/0000-0002-0892-9047
João Henrique Lara do Amaral. School of Dentistry, Federal University of Minas Gerais.jhamaral.1@gmail.com. ORCID: https://orcid.org/0000-0001-6900-7559
Rafaela da Silveira Pinto. School of Dentistry, Federal University of Minas Gerais. rafaelasilveirapinto@gmail.com. ORCID: https://orcid.org/0000-0002-6169-7708
Jacqueline Silva Santos. Minas Gerais State Department of Health.jacqueline.silva@saude.mg.gov.br. ORCID: https://orcid.org/0000-0003-0572-4668
Fernanda Lamounier Campos. School of Medical Sciences of Minas Gerais.nanda_lamounier@yahoo.com.br. ORCID: https://orcid.org/0000-0002-4772-8088
Elisa Lopes Pinheiro. School of Dentistry, Federal University of Minas Gerais.elisalp92@gmail.com. ORCID: https://orcid.org/0000-0002-3390-1062
Maria Inês Barreiros Senna. School of Dentistry, Federal University of Minas Gerais.mariainessenna@gmail.com. ORCID: https://orcid.org/0000-0002-0578-8744

Abstract
This is a methodological study regarding the development, content validation, and measurability assessment of indicators for monitoring oral health services in primary health care (APS). The indicators were constructed using variables recorded in individual dental care records and collective activities within the e-SUS APS system. A panel of 46 experts evaluated these indicators, determining their percentage of agreement regarding the relevance of each measure, the transparency of their calculation method, and whether they captured the theoretical dimension. Measurability was assessed using 2022 data from the Primary Care Health Information System (SISAB) for each Brazilian municipality. The percentage of municipalities with nonnull data for the numerator and/or denominator was thus obtained. Sixty-eight indicators were validated (agreement >75%), with measurability demonstrated among 53 across the following subdimensions: Access to Oral Health Services (9), Oral Health Surveillance (5), Diagnosis, Treatment, and Rehabilitation in Oral Health (16), Promotion and Prevention (14), Intersectoral Action/Popular Participation (4), and Work Process (5). The percentages of municipalities with calculated indicators thus ranged from 0.97% to 95.40%. While valid and measurable indicators can enhance evaluative capacity, they may, therefore also signal underreporting or issues in the provision of health services to SUS users.
Keywords: Primary Health Care. Oral Health. Health Services. Indicators of Health Services. Health Care Evaluation Mechanisms.

INTRODUCTION
The monitoring corresponds to the continuous and systematic follow-up of services and should be carried out based on information1 about the health and disease process of the population and related to the executed actions2. Monitoring through indicators is a tool for decision-making on goals and the definition thereof by managers to qualify health interventions3 and reflect the repercussions of health policies and programs4.
In the Brazilian context, the evaluation and monitoring of oral health services are among the guidelines of the Brazilian National Oral Health Policy (PNSB)5. The Ministry of Health has established municipalities\' responsibility for achieving goals concerning a minimum list of indicators in primary health care (APS). Oral health indicators have been included in goals agreed upon in the different management models over time6,7, and analyses of their evolution have highlighted their advances and setbacks, including the need for new indicators that can contribute to the consolidation of evaluation practices in the area of oral health in the Unified Health System (SUS)6,7,8. The fragmentation of such monitoring systems and the lack of indicators for evaluating and guiding policies are recognized global challenges9. A recent World Health Assembly resolution has recommended that governments facilitate the developing and implementation of effective surveillance and monitoring systems10.
This study proposes an evaluation matrix of indicators obtained from data generated periodically in health services and made available in the Health Information Systems (SIS), which can be estimated at the national, state, or municipal level or by health teams. In Brazil, the SIS is an information network composed of systems of epidemiological rationality, health care (service production), monitoring, and service management, among others. It has been increasingly used in health research and evaluation. These systems offer comprehensive population coverage, broad availability, and low-cost data collection while favoring longitudinal follow-up2,11. In APS, the Health Information System for Primary Care (SISAB) and the e-SUS APS strategy for its operation have been in force since 201312. The use of this data source favors periodicity within an evaluation cycle. Another characteristic that qualifies an indicator is its measurability, which corresponds to the possibility of calculating this indicator with the available data and the complexity of this calculation3.
The evaluation matrix, although including some indicators already recommended by the Ministry of Health (e.g., (i) coverage of first dental appointment, (ii) ratio between completed dental treatment and first dental appointment and (iii) dental care for pregnant women), broadens the scope of evaluation by considering aspects of the provision and management of oral health services. In addition, the proposal is that it be used as a monitoring tool, helping change the focus of evaluation from the verification of the achievement of goals to regular and continuous use for the monitoring of health services and planning at various levels of management. Additionally, the broader use of data can qualify and strengthen SIS.
Thus, it is posited to contribute to the evaluation process in the SUS, expanding the evaluative capacity of managers at the local and central levels through information supported by valid and reliable data that contribute to the qualification of care via a reduction in inequities and improvement in the health of the population. This methodological study aims to develop, assess the content validity, and evaluate the measurability of indicators for the monitoring of oral services in APS based on data from the e-SUS APS.
METHODS
Theoretical and conceptual framework
This methodological study adopted the principles and guidelines of the PNSB5 and the concepts in the National Policy for Primary Care (PNAB)13. It was also based on the model for evaluating the effectiveness of oral health care (Figure 1), adapted from the model proposed by Nickel (2008)14 and modified by Colussi (2010)15. The model proposed by Nickel includes two dimensions of the quality of oral health services: Management of Oral Health and Provision of Oral Health Services. The former is the political-organizational dimension of the model, arranged into four subdimensions: Intersectorality, Popular Participation, Human Resources and Infrastructure.
The Provision of Oral Health Services dimension is related to care for the individual14 and includes the following subdimensions: (i) Promotion, Prevention and Diagnosis and (ii) Treatment, Recovery and Rehabilitation by Life Cycle. Colussi (2010)15 has included SUS principles and guidelines in this model, such as Integrality, Universality, and Equity. In the present study, this model was adapted according to the theoretical affinity of the indicators for each dimension/subdimension and to cover all aspects evaluated by the proposed indicators (Chart 1).
Data sources
The data sources comprised the health (production) and collective activity reports on APS, publicly extracted from SISAB16. The secondary data were generated from the routine records of appointments and activities performed in APS found in the Individual Dental Care Forms and Collective Activity Forms entered in the e-SUS APS through Simplified Data Collection (CDS) or Electronic Citizen\'s Record (PEC), e-SUS APS application on Android® devices or through its system using Apache THRIFT transport technology17,18. SISAB allows the extraction of data disaggregated by municipality, state, health region, or Brazilian macroregion (North, Northeast, Southeast, South, and Center-West). However, the indicators can be estimated by the health team using management reports at the local level as a source.
The Individual Dental Care Form is the instrument for recording the dental clinical care of each user by the Oral Health team (eSB) concerning the following: the identification of the professional and the health unit where the care took place and the identification of the user and characteristics of the care performed (type of consultation, type of care, occurrence of diseases for oral health surveillance, procedures performed, conduct and outcome). These records are made individually for each user and each new instance12. The Collective Activity Sheet is the instrument for recording the actions collectively performed by the teams and is used by all professionals on APS teams. There is a field for filling in collective health actions for the population (Health Education, Group Care, Collective Assessment/Procedure, Social Mobilization) and for team organization (Team meeting, Meeting with Other Health Teams, Intersectoral Meeting/local Health Council/Social Control). The topics of the team meetings can be administrative issues, work processes, the diagnosis/monitoring of territory, the planning/monitoring of health actions, case discussion/singular therapeutic project, and continuing education. Actions aimed at the population are characterized by their target audience (community in general, according to life cycle, sex, and health conditions). These actions are further described in terms of health topics and practices. The records also include data on the identification of the professional responsible, the location of the activity, and the participants12.
Development of indicators
The indicators were developed through extensive discussion and collective interinstitutional work by the Working Group (WG) composed of representatives of the UFMG School of Dentistry (five professors in the areas of Public Health and Epidemiology, with experience in studies evaluating health policies and services, two undergraduate students from the School of Dentistry, UFMG, and one fellow for technical support) and of the State Health Coordination Department in the State Department of Health of Minas Gerais (technical assistant, also a doctoral student in Public Health), as well as two workers in oral health APS.
Initially, a study of the theoretical framework, the data sources and the e-SUS APS and SISAB systems was performed. The WG also sought to understand the guidelines for completing the Individual Dental Care Forms and Collective Activities Forms by the APS teams. The referenced sources were the Manual for Completing the CDS Forms12 and the experience of oral health service professionals in APS. In this process, the data of interest for monitoring oral health services were selected, and the numerators and denominators were defined according to their intended measurement.
An internal evaluation of the 1st version of the indicator matrix was conducted by pairs of WG members who presented their considerations, and the matrix was revised by seeking the consensus of the WG to obtain the version for content validation.
The evaluation matrix of indicators was organized by dimension, and each indicator was defined in a standardized qualification form based on the following attributes: name of the indicator, measure (information that defines the indicator and the way it is expressed; if necessary, aggregating elements to understand its content), interpretation of the result (explanation of the type of information obtained and its meaning), uses (main purposes for using the data to be considered in the analysis of the indicator), limitations (factors that restrict the interpretation of the indicator, referring to both the concept itself and the sources used), calculation method (formula used to calculate the indicator, defining the variables that compose it), data source (institutions responsible for producing the data used in the calculation of the indicator and by the information systems to which they correspond), and parameter (element or characteristic defined as a reference for comparison, when present). Otherwise, the indicator qualification sheets dictate that the parameters can be established via analysis of the historical series in each context or from the national, state average values or those obtained at the local level), observations (complementary information on the data sources or the data used in the calculation of the indicators), and references (material consulted for the preparation or interpretation of the indicators)3,19.
Validation of the content of the indicators
The modified e-Delphi method was adopted to obtain the opinions of a panel of professionals (panelists) on the indicators, with interactive and anonymous evaluation rounds20. These panelists were researchers in the field of public health and epidemiology (n=12), managers of oral health services (n=16), and professionals in oral health services (n=19) with experience in both the production and analysis of data and information for research purposes or in the management of health services. Due to the substantial number of indicators, the groups of panelists were composed of representatives of all profiles. For each group of indicators, 10 to 12 panelists were invited by e-mail to obtain the participation of at least seven in each group. Upon acceptance, the material for validation was sent to a panelist. This stage took place from 03/2020 to 06/2021.
Survey Monkey® software was used to format the validation tool and to manage the responses obtained, generating a link for access by the panelists. In the first section of this tool, a description of the methodology and theoretical framework used for the development of the indicators was presented, as well as the validation process, followed by the Free and Informed Consent Form (ICF), whereby the panelists could register their acceptance and participate in the process. In the second section, the qualification form for each indicator was presented, followed by the validation script for the panelist to enter their evaluation. This script included the following statements: 1) The indicator "x" is relevant for the evaluation of the quality of oral health services; 2) The method of calculating the "x" indicator is easily understandable; 3) The method for calculating the "x" indicator is easily reproducible; and 4) The "x" indicator measures aspects of the "y-dimension/z-subdimension". The response options were based on a Likert scale as follows: 0 disagree, 1 partially disagree, 2 partially agree and 3 agree. Additionally, an open field was included to record any "observations, criticisms or suggestions regarding the evaluated indicator" of the panelists.
The percentage of agreement of the panelists for the response options "I partially agree" or "I agree" was obtained for each indicator concerning its relevance, understandable calculation method, reproducible calculation method, and measurement of the dimension aspects. For the permanence of the indicator, a threshold of 75% agreement was adopted for each evaluated aspect21. Indicators with agreement <75% were reassessed and, when maintained in the matrix, subjected to another evaluation round. Even those indicators that obtained ≥ 75% agreement in the previous round were sent back to the panelists, followed by an open field for the insertion of new comments and/or suggestions if they considered them necessary. When five or fewer panelists participated, the criterion of 100% agreement was adopted for the maintenance of the focal indicator21.
Between one round and another, a report was prepared, grouping the suggestions and similar comments as well as the changes made in the matrix with justifications based on the percentage of agreement for each indicator. The report protected the identities of the panelists and was made available to all of them in the subsequent rounds of validation, along with the updated indicator matrix.
Evaluation of measurability and percentage of municipalities with nonzero data for calculating the indicators
Measurability was assessed using data from the dental appointments and collective activities performed in each Brazilian municipality during 2022, extracted from SISAB, separately, for the numerator and denominator of each indicator, according to the guidelines described in the Dictionary ofIndicators19,22. The Brazilian Institute of Geography and Statistics (IBGE) code for each municipality, present in the databases extracted from SISAB, was used to link the databases before calculation. The percentage of municipalities that had nonzero data for the calculation of the indicator was obtained by dividing the number of municipalities with no data in SISAB for the numerator and/or denominator by the total number of municipalities (N=5565).
All panelists agreed to participate in the study by signing the informed consent form (CAAE: 68646217.0.0000.5149).
RESULTS
A total of 70 indicators were originally developed, 58 comprising the Oral Health Services Provision dimension (Access to Oral Health Services: 11, Oral Health Surveillance: 10, Diagnosis, Treatment and Rehabilitation in Oral Health: 22, Promotion and Prevention: 15) and 12 falling into the Oral Health Management dimension (Intersectoral Action/Popular Participation: 4, Structure of Health Services: 3, eSB Work Process: 5). Sixty-two panelists were invited to participate in the validation process; 46 (response rate: 74.2%) responded in the first round and 31 (response rate: 67.4%) in the second round. For each set of indicators, 5 to 9 panelists participated in the evaluation rounds. The revisions performed in these two rounds have been detailed in a previous publication23.
Provision of Oral Health Services Dimension
The validation of the 21 indicators of Access to Oral Health Services and Oral Health Surveillance subdimensions was performed in two rounds. Agreement < 75% was observed for three indicators in at least one of the aspects evaluated, which were subjected to a 2nd validation round (Table 1). The revisions performed mainly related to the inclusion of limitations (ind. 1, 2, 3 and 9), to observations facilitating interpretation or clarification of the data used to calculate the indicator (ind. 5 and 11), to modifications to the name of the indicator (ind. 5, 6, 7, 11, 12 and 14 to 21) and to the wording of the interpretation for greater clarity of the measure (ind. 7). The descriptions of the measures of indicators 12, 14 to 21 were modified because they refer to the attendance rate, not a proportion, as originally proposed. For all the indicators, the data source descriptions in the qualification form, individual dental care forms, and collective activities forms were replaced by SISAB health reports. In the second round, indicator number 10 ten was excluded because it is an absolute number, making it impossible to compare its results over time or across different locations, limiting its use in monitoring oral health services. For the other indicators, agreement >85.7% was obtained, and no further revisions were needed, having been validated in the second round.
In the Diagnosis, Treatment, and Rehabilitation in the Oral Health subdimension, three groups of different panelists participated in two rounds of validation of the 22 indicators. In the 1st round, agreement < 75% was obtained for ten indicators in at least one of the aspects evaluated (Table 1). The indicators were revised to include any measurement limitations (ind. 23, 28, 37 to 42), a description of the procedures that derive the numerator/denominator and/or clarification of the data sources (ind. 23, 24, 32, 34, 38, 39, 42), any change in the calculation method, especially in relation to the delimitation of age groups (ind. 25, 26, 30, 31, 38, 42), revisions to the text interpreting the measure (ind. 30, 31, 36, 37, 39 to 42) and any change to the name of the indicator (ind. 22 to 24 and 28). For indicators 22 to 24, the term "dental" was inserted into their name to delimit the types of procedures. In indicator 28, permanent dentition was delimited into the name thereof. In the 2nd round, there was an increase in the percentage of agreement for 9 indicators. The indicator rate of alveolitis was excluded because it did not reach an agreement and the panelists\' arguments showed little relevance. Although there was an increase in agreement for the indicator Proportion of appointments to the NASF originating in appointments provided by the eSB, it was ultimately excluded it because it should have reached a percentage of agreement of 100% due to the reduction in the number of panelists in the second round (Table 1). The agreement regarding the indicator measuring aspects of the dimension did not increase in round 2 for the Mean Referrals for Dental Prosthesis indicator. However, it was decided to maintain it and revise the subdimension concept to include care for rehabilitation, as suggested by the panelists. A new indicator, Treatment of mucosal lesions in APS, was introduced between the 1st and 2nd rounds based on the comment of one of the panelists on the inclusion of this procedure in the Table of Procedures, Medications, Orthoses, Prostheses and Special Materials (OPM) of the SUS in mid-2020.24
Among the 15 indicators in the Promotion and Prevention subdimension, as originally created, agreement >75% was obtained for 12 of them by consensus among eight panelists participating in the 1st round (Table 1). Three indicators were subjected to the 2nd round of validation, and the percentage of agreement was increased. Between the 1st and 2nd rounds, the method for calculating nine indicators was revised to define better and describe the age groups or the types of activities/procedures included in the numerator and/or denominator. Two indicators were excluded (Supply of Toothbrush and Supply of Toothpaste), while the indicator that assessed the concomitant supplies of toothbrushes and toothpaste was maintained because it was considered sufficient for measuring the supply of these inputs. Other wording changes or text inclusions were made to clarify the interpretations, measurements, limitations, and uses of the indicators. There was a change to the name of indicator 44 to represent the measure better. After the 2nd round, at the suggestion of a panelist, three new indicators were created to break down and replace indicator 56 for each of the types of collective activity (health education, group care and collective evaluation/procedure). The three indicators maintained the same calculation method; however, the numerator variable became each separate type of collective activity. Because they are very similar to indicator 56, which the panelists already validated, it was decided to include these indicators without revision.
The Oral Health Management dimension and the Intersectoral Action/Popular Participation, Structure of Health Services, and eSB Work Process subdimensions
There was agreement > 75% for all evaluated aspects of the indicators in this dimension, and this review was completed in a single evaluation round. The comments of the panelists were considered in the review and included limitations for indicators 59, 63 and 65, a correction to the name of indicator 61 according to the measure; indicators 67 and 68 had their calculation method modified to offer a better delimitation of their denominator in relation to the topics discussed in the team meetings.
Validated indicator matrix
Six of the 70 indicators in the initial matrix were excluded, and 64 were validated after revisions in the 1st and/or 2nd rounds. A new indicator was created in the dimension Diagnosis, Treatment and Rehabilitation in Oral Health, and three new indicators were created in the dimension Promotion and Prevention. Thus, the final matrix consisted of 68 indicators, 56 of which referred to the Provision of Oral Health Services in the following subdimensions: Access to Oral Health Services (10), Oral Health Surveillance (10), Diagnosis, Treatment and Rehabilitation in Oral Health (21), and Promotion and Prevention (15). The Management of Oral Health Services dimension included 12 indicators, four in the subdimension Intersectoral Action/Popular Participation, three for evaluating the Structure of Health Services, and five related to the Work Process of the eSB.
Measurability
Fifteen indicators were not measurable due to their lack of identification of PNE in SISAB (Proportion of people with special needs having their first programmatic dental appointment, Mean number of PNE referrals to specialized care), the impossibility of extracting relevant data from a population registered by age (Rate of dental care of children due to dentoalveolar trauma, Rate of dental care of adolescents due to dentoalveolar trauma, Rate of dental care of adults due to dentoalveolar trauma, Rate of dental care of children with moderate or severe dental fluorosis, Rate of dental prosthesis installation in the adult population, Rate of dental prosthesis installation in the elderly population, Mean number of extractions of permanent teeth, and Mean number of deciduous teeth extraction), or absence of records of treatment of mucosal lesions in SISAB until May 2023 (Treatment of mucosal lesions in APS) as well as the unavailability of data on the supply of toothbrushes and toothpaste (Supply of toothbrush and toothpaste).
Table 3 shows the 53 measurable indicators, the final name and numbering and description of each measure, and the percentage of municipalities with records in the numerator and/or denominator, enabling their calculation. The results on the indicators, calculated for each municipality and Brazilian Federation Unit, are shown in spreadsheets, graphs, and maps available in a publicly accessible dashboard, and the indicator qualification sheets have been published in the Dictionary of Indicators19. The percentage of municipalities with nonzero data for fifteen indicators was >70%. A very low percentage of records was observed for the indicator Rate of dental care for users with cleft lip and palate. The indicators referring to the mean number of referrals to dental implants, pediatric dentistry, orthodontics/orthopedics and diagnostic radiology support for dental procedures and the mean number of fitting procedures per denture installed were obtained in a low percentage of Brazilian municipalities (<10%). For <20% of municipalities, it was possible to estimate the indicators Scheduling users for group activities by the eSB, Educational action for children in very early childhood (0 to 3 years old), Educational action for tobacco control, and Collective practices in oral health. All indicators in the dimension Management of Oral Health Services were estimated for <15.11% of Brazilian municipalities.
DISCUSSION
The validated indicators cover various aspects for monitoring and evaluating the performance of oral health services in APS based on secondary data from a national health information system. As a whole, the evaluation matrix will enable the monitoring of oral health services regarding the coverage of individual and collective APS actions and specialized care in the SUS, the resoluteness of dental care, the evaluation of the predominant care model, and the analysis of integration between the eSB and the family health team, in addition to the calculation of rates of dental care for specific conditions. These indicators can be analyzed by considering geographical and temporal variations, helping identify inequalities and trends that require specific actions and studies. The inclusion of data on oral health in SISAB/e-SUS APS represents an advance in the analysis of the oral health situation at the local level in the territories where the eSBs operate. In this sense, the matrix of indicators prepared from these data constitutes a relevant contribution to APS\'s management, monitoring, and evaluation.
The panelists contributed to the content validation of the indicator matrix by using the e-Delphi method to improve the clarity of measurements, the names of the indicators, and their methodological transparency. Regarding this last aspect, the panelists expressed whether the procedures for constructing the indicators were sufficiently clear and transparent and whether the methodological decisions were justified. These attributes are essential for ensuring the legitimacy of the indicators in technical and scientific circles, which is crucial for their political and social acceptance25. The professionals presented different perspectives on the indicators based on their varied experiences and training levels, demonstrating the technical and scientific relevance of the proposed measures. The percentages of agreement obtained in the review rounds were sufficient for most indicators, indicating that they can measure aspects of the quality of services defined by the theoretical conceptual model adopted. The e-Delphi method allowed the participation of professionals from various parts of Brazil and made it possible to conduct the study during social isolation due to the COVID-19 pandemic20.
The measurability of most indicators was demonstrated, except for those with data unavailable in SISAB, even if their calculation method uses variables contained in APS registration forms. The nonmeasurability of two indicators in the Structure of Services dimension showed that the data generated from routine records are insufficient for evaluating the characteristics of the physical spaces and environments of the Basic Health Units. Regarding the indicator Treatment of mucosal lesions in APS, the absence of these data can be explained by the fact that this procedure was included in the SUS Procedures, Medications, and OPM Table from mid-202024. Thus, the calculation of this indicator will be possible based on the availability of data for the years following this change to the table. Although measurability has been demonstrated for the year 2022, data extraction can be performed by month, allowing less temporal disaggregation. The indicators can be calculated as soon as the information is made available in the system, ensuring regularity in measurement and contributing to the timely evaluation and monitoring of social policies and programs25.
The evaluation of the percentage of municipalities with nonzero data regarding individual dental appointments and collective activities (numerator and/or denominator of the indicator) may indicate failures in properly completing the collection instruments, pointing out weaknesses and potentialities of the data produced by this information system3,26. On the other hand, as they are records of outpatient production, the absence of data likely corresponds to the nonperformance of care or action by the eSB or APS teams, signaling characteristics of the offer concerning the modalities of individual or collective actions in public health services. However, the SISAB technical notes and guidelines for completing the collection forms notably do not record any nonperformance of actions/attendance. When extracting data from SISAB, "zero" values cannot be obtained, i.e., it is impossible to determine whether a particular municipality did not perform a specific action/service in the focal period or if the corresponding records exist. Therefore, the quality of such records may be one of the challenges for using this data source, especially regarding referrals of users to secondary care and the development of collective actions.
This study has performed content validation on the proposed indicators. However, a study on implementing this evaluation matrix in municipal contexts should include the evaluation of the acceptability, adoption, feasibility and suitability of the indicators by SUS professionals. In addition, an analysis of the data quality should be included, considering low rates of records or extreme records and possible classification errors in the records. This should involve verifying the compatibility between the information in the medical records and the data available in the Health Information System for Primary Care (SISAB). It is also recommended that future efforts involve a thorough review of the technical documents related to the classification of variables and registry guidelines and an evaluation of the understanding and practices of professionals concerning these processes.
A previous case study among Family Health Strategy professionals and a health manager of a Brazilian municipality has investigated the main problems encountered in registering and retrieving information in the e-SUS APS. Its findings show weaknesses in the training process of the professionals on these teams, including challenges to completing data record forms, difficulty using electronic systems, and a significant lack of knowledge on the requisite information storage processes. The authors thus underscore the need for trained professionals who work directly with the e-SUS APS strategy, as well as an increase in their and their managers\' awareness of the importance of the quality of health information in promoting improvements in the recording and retrieval of information27. Therefore, the training of health professionals involved in the process, from obtaining to recording the data, can ensure the indicators validly reflect the reality of oral health services in APS, allowing a more robust and effective evaluation of the health system. It is also suggested that the implementation of e-SUS APS will be consolidated in the country, helping qualify the use of SISAB and the routine data generated. Progress in the implementation of the e-SUS APS strategy has been demonstrated, existing at some level in 92.2% of the municipalities in 201928. The consolidation of the use of PEC in the national territory, from the perspective of digital health, should allow, in the future, the integration and standardization of APS records, the prevention of typing errors, or even the elimination of variable incompatibility due to municipalities using their systems.
As a limitation of this study, the lack of participation of panelists in the 2nd validation round stands out, which reduced the response rate and the possibility for these professionals to present additional contributions in relation to the revised version of the evaluation matrix. In these cases, the highest threshold was adopted for the percentage of agreement for maintaining the indicator in the evaluation matrix21. Another limitation is that the measurability of the indicators was affected by the characteristics of their data sources; not all the variables recorded in the APS user service records are available in the SISAB. This aspect mainly compromised the estimates of indicators relevant to certain specific groups defined by age or health condition (PNE). However, secondary data was sufficient, considering that their source was a national public information system containing continuous records of dental appointments performed in APS and of collective actions in the national territory. Hence, using such indicators in the monitoring actions of teams should contribute to the quality of the data generated based on the involvement of the actors in the process of discussing the results observed in each municipality.
The indicators have shown content validity and are measurable; thus, they may support the management, planning, and evaluation of public policies within the SUS. This can help improve the National Oral Health Policy, improve the quality of services, reduce inequalities and strengthen social controls. The indicators created can also be used for evaluative studies that characterize oral health services in the SUS to understand the different levels of observed results, including comparative analyses based on geographical and temporal determinants. The low percentage of municipalities with an estimated indicator may also indicate an underreporting of individual procedures or collective activities in oral health or problems when offering these actions to SUS users in APS.
Contributions
RCF participated in the conception, design, analysis, and interpretation of data, as well as in writing and approving the version of the article to be published. LLFHC participated in the conception, design, and data analysis, offered a critical review of the article, and approved the version to be published. JHLA participated in the conception, design, and data analysis, offered a critical review of the article, and approved the version to be published. RSP participated in the data analysis, offered a critical review of the article, and approved the version to be published. JSS participated in the data analysis, offered a critical review of the article and approved the version to be published. FLC participated in the analysis and interpretation of data, and writing and approving the version of the article to be published. ELP participated in the analysis and interpretation of data and writing and approving the version of the article to be published. MIBS participated in the conception, design, data analysis, and critical review of the article and approved the version to be published.

Acknowledgments
To Marina Fajardo and Debora Egg for participating in the validation process of the matrix of indicators. This study was conducted with support from the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior- Brazil (CAPES 001), Fundação de Amparo à Pesquisa de Minas Gerais (PPSUS APQ-00763-20 and PPM-00603-18). RCF is a CNPq productivity fellow (310938/2022-8).

References
1. Hartz ZMA, Silva LMV. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Rio de Janeiro: Fiocruz; 2005.
2. Branco MAF. Sistema de informação em saúde no nível local. Cad Saude Publica 1996; 12(2):267-270.
3. OPAS. Indicadores de saúde: elementos conceituais e práticos. Washington, D.C.: Organização Pan-americana de Saúde; 2019.
4. Santos JLD, Ferreira RC, Amorim LP, Santos ARS, Chiari APG, Senna MIB. Oral health indicators and sociodemographic factors in Brazil from 2008 to 2015. Rev Saude Publica 2021; 55:25.
5. Brasil. Diretrizes da Política Nacional de Saúde Bucal [Internet]. Brasília: Ministério da Saúde; 2004 [cited 2023 Jun 06]. Available from: https://bvsms.saude.gov.br/bvs/publicacoes/analise_pacto.PDF
6. França MASA, Freire MCM, Pereira EM, Marcelo VC.Indicators proposed by the Brazilian Ministry of Health for monitoring and evaluating oral health actions in the National Health System: documentary research, 2000-2017. Epidemiol. Serv. Saúde 2020; 29(1):e2018406.
7. França MASA, Freire MCM, Pereira EM, Marcelo VC. Oral health indicators in the Interfederative Pacts of the Unified Health System: development in the 1998-2016 period. Rev Odontol UNESP 2018; 47(1):18-24.
8. Scherer CI, Scherer MD. Advances and challenges in oral health after a decade of the "Smiling Brazil" Program. Rev Saude Publica 2015; 49:98.
9. Agrasuta V, Thumbuntu T, Karawekpanyawong R, Panichkriangkrai W, Viriyathorn S, Reeponmaha T, Jaichuen W, Witthayapipopsakul W, Gaewkhiew P, Prasertsom P, Tangcharoensathien V. Progressive realisation of universal access to oral health services: what evidence is needed? BMJ Glob Health 2021; 6(7):e006556.
10. WHO. World Health Assembly resolution: WHA 74.5 Oral Health. [Internet]. 2021 [cited 2023 Oct 10]. Available from: https://apps.who.int/gb/ebwha/pdf_files/EB148/B148_R1-en.pdf
11. Coeli CM, Pinheiro RS, Carvalho MS. Neither better nor worse, simply different. Cad Saude Publica 2014; 30(7):1363-1375.
12. Brasil. e-SUS Atenção Básica: manual de uso do sistema com prontuário eletrônico - PEC (versão 4.2) [Internet]. Brasília: Ministério da Saúde; 2021 [cited 2023 Jun 06]. Available from: https://cgiap-saps.github.io/Manual-eSUS-APS/
13. Brasil. Política Nacional de Atenção Básica [Internet]. Brasília: Ministério da Saúde; 2012 [cited Available from: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2436_22_09_2017.html
14. Nickel DA. Modelo de avaliação da efetividade da Atenção em Saúde Bucal [dissertação]. Florianópolis: Universidade Federal de Santa Catarina; 2008.
15. Colussi CF. Avaliação da qualidade da atenção em saúde bucal em Santa Catarina [tese]. Florianópolis: Universidade Federal de Santa Catarina; 2010.
16. Sistema de Informação em Saúde para a Atenção Básica - SISAB. [Internet]. Brasília: Ministério da Saúde; 2022 [cited 2023 Jun 06]. Available from: https://sisab.saude.gov.br/index.xhtml.
17. Brasil. Nota técnica explicativa: relatório de saúde (Produção) [Internet]. Brasília: Ministério da Saúde; 2021 [cited 2023 Jun 06]. Available from: https://sisab.saude.gov.br/resource/file/nota_tecnica_relatorio_saude_producao_210802.pdf
18. Brasil. Nota técnica explicativa: relatório de saúde (Atividade Coletiva) [Internet]. Brasília: Ministério da Saúde; 2021 [cited 2023 Jun 06]. Available from: http://189.28.128.100/dab/docs/portaldab/documentos/esus/sisab_nota_tecnica_relatorio_atividade_coletiva.pdf
19. Ferreira RC, Chalub LLFH. Dicionário de indicadores para avaliação dos serviços de saúde bucal na Atenção Primária à Saúde. Belo Horizonte: FAO UFMG; 2023.
20. Toronto C. Considerations when conducting e-Delphi research: a case study. Nurse Res 2017; 25(1):10-15.
21. Alexandre NM, Coluci MZ. Validade de conteudo nos processos de construcao e adaptacao de instrumentos de medidas. Cien Saude Colet 2011; 16(7):3061-3068.
22. Lima CR, Schramm JM, Coeli CM, da Silva ME. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109.
23. Ribeiro MF. Avaliação dos serviços de saúde bucal na Atenção Primária: desenvolvimento e validação de indicadores elaborados a partir de variáveis do e-SUS APS [Trabalho de Conclusão de Curso de graduação]. Belo Horizonte: Faculdade de Odontologia da UFMG; 2021.
24. Brasil. Portaria no. 526 de 24 de junho de 2020. Inclui, altera e exclui procedimentos da Tabela de Procedimentos, Medicamentos, Órteses, Próteses e Materiais especiais do SUS. Diário Oficial da União 2020; 02 jul. Available from: https://www.in.gov.br/en/web/dou/-/portaria-n-526-de-24-de-junho-de-2020-264666631
25. Jannuzzi PdM. Indicadores para diagnóstico, monitoramento e avaliação de programas sociais no Brasil. Revista do Serviço Público 2005; 56(2):137-160.
26. Correia LO, Padilha BM, Vasconcelos SM. Metodos para avaliar a completitude dos dados dos sistemas de informacao em saude do Brasil: uma revisao sistematica. Cien Saude Colet 2014; 19(11):4467-4478.
27. Tenório-Filho L. Sistemas de informação em saúde: um estudo dos processos de registro, integração e recuperação da informação das bases SISAB e ESUS-AB [Mestrado]. Maceió: Universidade Federal de Alagoas; 2022.
28. Cielo AC, Raiol T, Silva END, Barreto JOM. Implementation of the e-SUS Primary Care Strategy: an analysis based on official data. Rev Saude Publica 2022; 56:5.
29. Donabedian A. The quality of medical care. Science 1978; 200(4344):856-64.