1.Introduction
Academic Engagement is a complex and multidimensional construct that has been approached from various perspectives ( Fredricks et al., 2016). Although there is consensus about its relevance to students' educational process, its definition has generated a considerable division of opinions and positions regarding measurement ( Medrano et al., 2015) ( Fredricks et al., 2016).
It is a positive predictor of academic performance ( Acosta-Gonzaga & Ramírez-Arellano, 2020; Delfino, 2019; Dunn & Kennedy, 2019; Lei et al., 2018; Maluenda-Albornoz et al., 2022a; Maluenda-Albornoz et al., 2023; Ribeiro et al., 2019) and a greater sense of academic belonging ( Maluenda-Albornoz et al., 2022a; Maluenda-Albornoz et al., 2022b; Maluenda-Albornoz et al., 2023; Wong et al., 2019). It has also been shown to positively predict motivation in students with greater resilience, persistence, emotional connection, and selfefficacy ( Abreu-Alves et al., 2022). Furthermore, it is positively related to the perception of positive academic emotions and academic adaptability ( Zhang et al., 2020a), favoring the permanence and completion of studies in higher education (Kor‐honen et al., 2019). Along the same lines, it has shown an inverse relationship with dropping out of studies ( Maluenda-Albornoz et al., 2021; Maluenda-Albornoz et al., 2022; Maluenda-Albornoz et al., 2022a; Maluenda-Albornoz et al., 2023; Nickerson & Shea, 2020; Zhang et al., 2020b; Díaz-Mujica et al., 2018). On the other hand, a negative relationship has been observed between Academic Engagement and disruptive variables of the university process such as burnout and academic procrastination ( Abreu-Alves et al., 2022; Aspeé et al., 2019; Liebana-Presa et al., 2018; Marôco et al., 2020; Morales-Rodríguez et al., 2019; Paloș et al., 2019; Rahmatpour et al., 2019).
The main approach in the academic context arises from the Self-Determination Theory, which proposes the emergence of intrinsic motivational states when the basic psychological needs of human beings are satisfied: autonomy, competence, and relationship (Deci & Ryan, 2018). From this approach, Academic Engagement is understood as a three-dimensional meta-construct comprised of three interrelated dimensions: behavioral, emotional, and cognitive commitment ( Allen & Boyle, 2023; Fredricks et al., 2016). Academic Engagement is a high motivation state that manifests itself in effects in these three dimensions ( Maluenda-Albornoz et al., 2023).
In the academic context, autonomy is satisfied when a student feels that he can make decisions and is motivated by intrinsic rather than extrinsic factors, competition is stimulated when the structure of the class allows the achievement of the results expected by the student and the need for relationship is covered when training occurs in an environment of support and concern between teachers and peers ( Fredricks et al., 2016).
Based on this conceptual approach, Marôco et al. (2016) developed an operationalization of Academic Engagement called University Student Engagement Inventory (USEI). Its self-report instrument integrates behavioral, emotional, and cognitive aspects, inviting students to measure their commitment to the teaching-learning process. The cognitive dimension of the USEI addresses the thoughts, strategies, and efforts that students make to acquire new knowledge and skills. The emotional dimension refers to the positive and negative feelings and emotions experienced during the learning process concerning classroom activities, classmates, and teachers. The behavioral dimension includes the actions associated with involvement carried out in learning spaces ( Marôco et al., 2016).
The instrument comprises fifteen items, with five items for each factor, and has shown an adequate factor structure for the threefactor structure in the Portuguese population with favorable indicators of criterion validity [χ²/df=2.26; CFI=.97; TLI=.97; RMSEA=.06] ( Marôco et al., 2016).
Another study with a Portuguese population also obtained favorable results for the three-factor structure [χ²(87) = 286.665; p.001; RMSEA= .051 (90%CI.045-.058); CFI=.987; TLI=.985; NFI=.982]. Additionally, invariance was found in terms of gender and degree areas ( Sinval et al., 2021).
A cross-cultural study carried out in nine countries on four continents with Portuguese, English, Finnish, Serbian, and Chinese languages through a Confirmatory Factor Analysis found that the instrument has the same factorial structure proposed by the original authors with a second-order factor. The study also found strong measurement invariance for gender and study area and weak invariance for country ( Assunção et al., 2020).
In a population of Italian university students studying Psychology and Biology, previous research findings are reaffirmed, obtaining a factorial structure of 3 components with good test-retest reliability. However, the instrument showed weak invariance for gender and area of study, which may be due to the poor cultural appropriateness of the instrument ( Esposito et al., 2022); these findings are similar to those obtained in a previous investigation in that country ( Assunção et al., 2020).
The version adapted and validated for Iran also obtained a good factor structure composed of behavioral, emotional, and cognitive engagement, retaining the 15 items of the original version ( Sharif Nia et al., 2022). In another study with an English-language Arab population, the original version of the instrument was used, obtaining similar results with a good 3-component factor structure and measurement invariance by sex ( Sharif-Nia et al., 2023).
A recent study in the Chinese population, found that the USEI has good construct validity, internal consistency, and reliability with a 3-factor structure despite the elimination of item 6 (worded in the negative) due to its low factor loading. Gender invariance was additionally observed ( She et al., 2023).
In an effort to obtain a Spanish adaptation of the USEI, Maluenda-Albornoz et al. (2020) carried out a study of adaptation to Spanish with cultural adaptations for use in the Chilean university context with a sample of engineering students, obtaining the same factorial structure with good adjustment indices [χ² (75) = 210.276, p .001; RM‐SEA= .047 (95%CI: .040-.055); CFI= .967; TLI= .954], good indicators of reliability and criterion validity.
Finally, to obtain a Spanish version for different Spanish-speaking countries, the research evaluated the psychometric properties of a unified version of the USEI in 3 Spanish-speaking countries, finding similar outcomes to previous works. The instrument presented optimal internal consistency with a 3-factor structure and a secondorder factor ( Freiberg-Hoffmann et al., 2022).
The previously reviewed research shows progress in the study of instruments adapted to measure Engagement in the Spanish-speaking university context. These efforts are developed due to the lack of valid and reliable instruments in said context to use with validity and reliability ( Guzmán-Arellano et al., 2024; Maluenda-Albornoz, 2021a).
There are instruments for the Spanish-speaking university context, but they arise from adaptations from other contexts, not from a specific design for university students. For example, the Classroom Engagement Inventory ( Leal-Soto et al., 2023), the School Engagement Instrument ( González et al., 2022), or the Utrecht Work Engagement Scale ( Guerra y Jorquera, 2021).
Despite their availability in Spanish, these instruments do not necessarily consider adequate adaptations for measurement in university students; this increases the importance of having an instrument designed for this context with good psychometric properties for its reliable and valid use.
1.1. The present study
As has been reported, evidence supports favorable metric properties for the USEI in various cultural contexts and even with indicators of cross-cultural invariance. In the case of the Spanish-speaking context and specifically in the Chilean university context, the studies replicate the three-factor structure with their respective items. However, no studies have been observed that test its invariance. Consequently, the objective of the present research was to evaluate the invariance of the instrument by sex in the Chilean first-year university population to contribute to the analysis of the metric properties of the USEI in said context.
2. Methods
A convenience sample comprised 468 first-year university students, 174 male (37.2%) and 294 female (62.8%). Ages ranged from 17 to 28 years (M=19 years; SD=4.5 years).
2.1. Design
The study was conducted with an instrumental design in a cross-section of time ( Ato et al., 2013). The version of the USEI used was its adaptation to the Chilean university context, which consists of fifteen items with five items per factor ( Maluenda-Albornoz et al., 2020). First, construct validity was evaluated through a Confirmatory Factor Analysis considering the aforementioned factorial structure. The correlation matrices, factor loadings of each item to the corresponding factor, and the fit indices of the analyzed model were analyzed. The WLSMV (Weighted Least Squares Mean Variance) method was used to extract factors. The fit indices considered to evaluate the factor model were the root mean error of approximation (RMSEA), the non-normative fit index (NNFI), the Tucker-Lewis index (TLI), the comparative fit index (CFI), and the non-normalized fit index (NNFI). The cut-off values used as reference were Chi-Square (X2), not significant p>.05 ( Hu et al., 1999); RMSEA less than .08 acceptable; CFI, TLI, and NNFI higher than .90 ( Hair et al., 2014).
To examine the factorial invariance of the instrument, multisample confirmatory factor analyses were carried out with JASP program version 0.17.2, using the WLSMV (Weighted Least Squares Mean Variance) method. The factorial invariance contrast was carried out by examining the goodness-of-fit of the structure of each instrument in each of the samples of men and women (base‐line). Next, configural invariance (Model 1), equivalence in factor loadings (Model 2), and equivalence in intercepts (Model 3) were examined. The comparison of Models 2 and 3 indices with those obtained in Model 1 was considered an indicator of non-significant practical difference.
As statistical criteria, the evaluation of the goodness of fit of each model was used with the same cut-off points indicated above (RMSEA, NNFI, CFI, TLI). Additionally, because the comparison between the different nested models using the maximum likelihood ratio is very sensitive to the sample size and the lack of normal distribution of the data ( Hair et al., 2014), Cheung Rensvold (2002) proposed using the increase of the CFI to determine if the compared models are equivalent. When the difference between the CFI of the two models is less than .01, equivalence is considered to exist. Additionally, it is possible to compare the increase in RMSEA between the different models where values less than .015 indicate equivalence between models ( Putnick Bornstein, 2016).Finally, the estimate of the correlation between the scores of the global inventory and each factor of the Multidimensional School Engagement and Disengagement Sale ( Wang et al., 2017) was incorporated as a measure of criterion validity. A strong positive correlation with the Engagement factor and a strong negative correlation with the Disengagement factor of said instrument were considered favorable criterion validity indicators.
2.2. Procedure
The recruitment of students was carried out through the career chair to obtain permits and manage the applications, which were carried out in the rooms where the students regularly carried out their activities before the start of one of their classes through paper questionnaires. Before distributing the questionnaire, every participant signed the informed consent, which incorporated all the ethical aspects necessary for research in the human sciences. It is essential to indicate that the University of Concepcións Ethics Committee evaluated and approved the project and related materials.
No incentive was provided for participation. The information was collected during the first semester of 2021 (the first academic semester in Chile).
3. Results
he descriptive statistics (Table 1) show similar mean values and standard deviations for men and women on the global scale and each subscale. The skewness and kurtosis indices have values between 0 and 2 that are acceptable to assume a distribution of values approximate to the normal distribution ( Bollen Long, 1993).
The confirmatory factor analysis was carried out considering the three factors proposed by the original instrument and its adaptation to the Chilean context. The analysis showed factor loadings between .555 and .933, appropriate following the cut-off point established in the literature (Table 2).
The fit indices showed results within the parameters accepted by the literature ( Hu and Bentler, 1999). The RMSEA index showed a value of .034 (95% CI: .022-.045), the CFI index value was .997, the TLI index was .997, the NNFI index was .997, and the NFI was .997. .993. Although the χ² index showed a significant value [χ² (87) = 134.170; p.001], this tends to overestimate with high sample sizes, so, as a complement, the χ²/df ratio was calculated to obtain a value within the values accepted in the literature (χ²/df = 1.54).
| Global USEI | Behavioral | Affective | Cognitive | |||||
|---|---|---|---|---|---|---|---|---|
| Female | Male | Female | Male | Female | Male | Female | Male | |
| Valid | 294 | 174 | 294 | 174 | 294 | 174 | 294 | 174 |
| Missing | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Mean | 5.666 | 5.431 | 5.790 | 5.641 | 5.648 | 5.079 | 5.759 | 5.572 |
| Std. Deviation | .617 | .679 | .806 | .875 | .630 | .625 | .866 | .1021 |
| Skewness | -818 | -365 | -717 | -579 | -983 | -912 | -599 | -500 |
| Kurtosis | 844 | -702 | 463 | 319 | 2.000 | 1.489 | 339 | -564 |
| Minimum | 3.333 | 3.667 | 3.000 | 2.600 | 2.200 | 2.080 | 2.080 | 3.000 |
| Maximum | 6.667 | 6.667 | 7.000 | 7.000 | 6.800 | 6.400 | 7.000 | 7.000 |
| 95% Interval | Confidence | ||||||
|---|---|---|---|---|---|---|---|
| Factor | Item | Beta | Est. error | z-value | p | Infer. | Sup. |
| Behavioral | U1 | .633 | .021 | 29.862 | .001 | .591 | .675 |
| U2 | .666 | .023 | 29.275 | .001 | .621 | .710 | |
| U3 | .622 | .022 | 28.720 | .001 | .580 | .664 | |
| U4 | .629 | 021 | 29.504 | .001 | .588 | .671 | |
| U5 | .555 | 022 | 24.935 | .001 | .511 | .598 | |
| Affective | U6 | .819 | 013 | 61.630 | .001 | .793 | .845 |
| U7 | .879 | 012 | -70.742 | .001 | -.904 | -.855 | |
| U8 | .900 | 013 | -70.841 | .001 | -.925 | -.875 | |
| U9 | .933 | 013 | -74.445 | .001 | -.957 | -.908 | |
| U10 | .520 | 020 | -26.165 | .001 | -.559 | -.481 | |
| Cognitive | U11 | .622 | 020 | 31.143 | .001 | .583 | .661 |
| U12 | .616 | 020 | 30.098 | .001 | .576 | .657 | |
| U13 | .569 | 020 | 28.220 | .001 | .529 | .609 | |
| U14 | .848 | 019 | 43.827 | .001 | .811 | .886 | |
| U15 | .793 | 019 | 41.792 | .001 | .756 | .830 | |
When evaluating the pearson correlation, as a measure of criterion validity, between
the global scale of the USEI and the Engagement dimension of the Multidimensional School Engagement and Disengagement Sale, a significant, positive, and strong correlation was observed (r=.834, p.001). The Disengagement measure obtained a significant, negative, and strong correlation with the USEI scale (r=-.626, p.001).
The reliability measures tested for the USEI global scale showed values higher than the cut-off point accepted in the literature in both the Cronbachs Alpha index (α=.758) and the McDonalds Omega index (Ω=. 760).
The analysis of invariance between men and women for the USEI scale followed the standard procedures proposed in the literature: the study of configural, metric, scalar, and strict invariance as previously indicated. The analysis of all levels of invariance showed fit indices within the parameters accepted by the literature, the only exception being the examination of strict invariance because the ∆NNFI, ∆CFI, and ∆TLI values exceeded the accepted limit, set at a maximum variation of .01 (Table 3).
4. Discussion
Measuring the invariance of psychometric instruments is valuable since it provides complementary evidence to examine whether the theories and instruments developed to evaluate human beings in one culture are applicable in another ( Spontón et al., 2018).
Measurement invariance is defined concerning a group or form of a test so that the formal and substantive meaning of the measurement is independent of them ( Elosua, 2005). Configural Invariance assumes that the same indicators in all groups measure the latent construct; Metric Invariance restricts the factor loadings so that they are the same in all groups; Scalar Invariance implies that the difference in means of the latent factor captures all the mean differences in the shared variance between items; and Strict Invariance implies that the specific variance (what is not shared with the factor) and the error variance (measurement error) are similar in the comparison groups ( Elosua, 2005).
Evaluating based on the erroneous assumption that the scale measures the same construct in the same way in all groups ( Byrne van de Vijver, 2010) can lead to incorrect results and decisions. Suppose the equivalence or invariance of an assessment instrument is not met. In that case, the validity of inferences and interpretations drawn from the data may be flawed ( Byrne, 2008), and conclusions based on group comparisons may not be valid.
The present research sought to contribute to this direction regarding the University Student Engagement Inventory in the Chilean population. The studys main objective was to evaluate the instruments invariance and psychometric properties in Chilean university students.
The results found allow us to add evidence in favor of preliminary studies that have shown evidence of validity for a three-factor structure ( Assunção et al., 2020; Espósito et al., 2022; Sharif et al., 2022; She et al., 2023) and those who have studied this same composition in the Spanish-speaking population ( Freiberg-Hoffmann et al., 2022; Maluenda-Albornoz et al., 2020). Additionally, favorable results were found for criterion validity and reliability in ranges similar to preliminary studies ( Freiberg-Hoffmann et al., 2022; Maluenda-Albornoz et al., 2020). These results imply the possibility of evaluating student engagement compared to academic activity at a global level and disaggregated by subscale, contributing to a more detailed analysis of the various academic situations. The above allows, in practical terms, to advance in concrete actions to promote actions aimed at improving Engagement levels according to the specific needs of each educational system.
In global terms, the multigroup analysis that compared the factor models between men and women showed favorable evidence for configural, metric, and scalar invariance in Chilean university students. Both the goodnessof-fit indices, such as ∆RMSEA and ∆CFI, showed evidence in favor of invariance.
| χ² | df | RMSEA | NNFI | CFI | TLI | ∆RMSEA | ∆NNFI | ∆CFI | ∆TLI | |
|---|---|---|---|---|---|---|---|---|---|---|
| Config | 339.017* | 189 | .05895% IC [.048-.068] | .935 | .942 | .935 | ||||
| Metric | 336.415* | 186 | .05995% IC [.049-.069] | .934 | .942 | .934 | .001 | .001 | .000 | .001 |
| Escalar | 369.667* | 198 | .06195% IC [.051-.070] | .929 | .933 | .929 | .003 | .006 | .009 | .006 |
| Estrict | 419.910* | 213 | .06495% IC [.055-.073] | .921 | .920 | .921 | .006 | .014 | .022 | .014 |
*: P .001
The strict invariance analysis showed good overall fit indices, a good indicator for ∆RMSEA, but, by a small margin, did not meet the criterion for ∆CFI. Although this may indicate some variation in the measurement parameters between the groups, this variation is not significant enough to invalidate comparisons made with the USEI. Furthermore, since scalar invariance is sufficient to make statistical comparisons between group means and patterns of covariates, the level of strict invariance is often not estimated ( Beaujean, 2014; Davidov et al., 2014). In this way, evidence is provided in favor of the use of the USEI regardless of the sex of the participants.
The joint results allow us to appreciate that, similar to preliminary studies ( Sharif-Nia et al., 2023), favorable results are observed for invariance in the multigroup analysis by sex, and good psychometric properties obtained from Confirmatory Factor Analysis, validity judgment, and reliability. These characteristics would allow the measurement of Engagement in its three dimensions, behavioral, affective, and cognitive, in addition to the measurement of global Engagement in both populations in the Spanish-speaking context.
A relevant limitation of the present study is that the sample is limited to first-year students, and differences specific to different degrees students take are not established because this variable was not recorded and, consequently, not analyzed. Additionally, due to the crosssectional nature of the research, causality should not be inferred from the results, and it is suggested that causal interpretations of the results should be avoided, as is typical of longitudinal and experimental methods. Thus, the correlation between USEI scores and criterion variables should be analyzed cautiously.
As projections, it is suggested that the analysis of the invariance between various cultures of the Spanish-speaking context be advanced to avoid interpretive errors when there are relevant cultural differences between various contexts. Likewise, it would be relevant to examine possible differences between the degrees being studied that may add differentiating components to the use of this instrument.
5. Conclusions and contributions
This article contributes new evidence about the validity of this inventory by using invariance (by gender), construct, and criterion validity tests. It also presents evidence about the reliability of the global scale and by dimension. All these results confirm previous research that showed its quality and contribute to consolidating knowledge that allows scholars, educational managers, and others to use it in university students to measure Engagement during the educational process.
Conflict of interest statement
The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author declares no sources of funding for this research.
Ethical approval
The study conforms to the ethical principles of the Declaration of Helsinki and was authorized by the Ethical Research Committee of the University of Concepción.
Informed consent
Participation was voluntary, and informed consent was sought from each participant itself.