Written October 11, 2022 for MHST602: Transforming Health Care Through Informatics
The way information is shared and reviewed has changed tremendously during the digital revolution (Tonsaker et al., 2014). Now, more than ever, consumers are taking to the Internet for their health information (Dobbins et al., 2018; Tonsaker et al, 2015). There are many reasons why that might be happening for Canadians: difficulties with access to care, greater involvement of allied health providers in the care continuum, and an increase in health literacy of the general population, among others (Chu et al., 2017; Hanif et al., 2009; Tonsaker et al., 2015).
The internet is an accessible way to promote health literacy (Chu et al., 2017; Hanif et al., 2009). An increase in health literacy improves health outcomes and appropriate health care utilization; limited health literacy detrimentally impacts health care costs (Tonsaker et al., 2014). A study by Chu et al. (2017) found that regardless of the severity of a health condition, those people who seek their health information online, did so as the first source of information. Having reputable information be easily identifiable online is an important strategy to promote health literacy for Canadians (Dalhousie University, 2021). Hanif et al. (2009) interpret a study published by Fox et al. in 2007, where 40% of online health information seekers reported that the advice they obtained online improved their self-care, and 50% reported that the information they found online caused them to ask their physician new questions regarding their health condition (Hanif et al., 2009). Even with increased health literacy, evaluating health information online is complex, and the average Canadian is not equipped to fully assess the quality of information found online (Dalhousie University, 2021; Dobbins et al., 2018). Accountability to website information through governing bodies that use standardized tools to evaluate digitally published information helps ensure public trust (Hanif et al., 2009; Wilson, 2002). It is widely accepted that health information found on the internet should be assessed across several standard realms: credibility, content, disclosure, links, design, and interactivity (Dalhousie University, 2021; Kim et al., 2009).
Health information on the internet is a constantly evolving environment, and as such, it would be impossible for a governing body to continuously evaluate all information presented (Tonsaker et al., 2014). Therefore, health care providers (HCPs) need to be able to implement tools to share health information websites with their patients quickly and safely and obtain high-level health information quickly for professional purposes without examining multiple peer-reviewed journals (Dobbins et al., 2017). Having easy-to-use, reliable, and valid tools at our disposal can facilitate HCPs being leaders in improving electronic health literacy. Here, the Web Resources Rating (WRR) tool is used to evaluate the Brain Injury Canada website across the different dimensions of evidence, transparency, and usability. The questions posed by the WRR tool evaluate across all of Kim et al.’s suggested classes, except for expressly considering interactivity, which is analyzed separately (Kim et al., 1999).
Rationale for Website Evaluation
Brain injury affects diverse populations across the lifespan (Public Health Agency of Canada [PHAC], 2020). Since 2002, brain injury (BI) incidence in Canada has stayed relatively steady, with a slight decrease (1.1% per annum) among males and a slight increase (0.5% per annum) among females (PHAC, 2020). The incidence of BI in infants is 137/100,000, and the incidence in those over 85 years old is 897/100,000 (PHAC, 2020). BI is commonly caused by: motor vehicle accidents, falls, domestic violence or assault, sport, suicide attempts, or insidious onset of illnesses such as stroke (Brain Injury Canada [BIC], n.d.; PHAC, 2020). Even mild BI can have serious long-term cognitive consequences and exists under a continuum of severity (PHAC, 2020). Both long-term and acute complications and the diversity of cause and severity create the need for this vulnerable population and their caregivers to seek relevant health information online. The BIC website, in particular, was chosen for evaluation because this population typically has greater accessibility concerns with information retrieval (BIC, n.d.). The BIC website also boasts information portals targeted to specific demographics such as teens, those living with BI, caregivers and family members of those living with BI, and HCPs (BIC, n.d.). A 2021 survey concluded that those living with BI had decreased access to care providers during the COVID-19 pandemic, which caused poorer health outcomes. Still, the BIC website also houses a service directory for many common care needs of this population (Brain Injury Canada, 2022). Those living with the effects of BI complications are often members of marginalized populations with higher rates of incarceration, homelessness, and mental illness, or victims of intimate partner violence, therefore it is crucial for caregivers and knowledgeable others to be able to easily find information to help those with secondary complications (BIC, n.d.).
The WRR Tool
Rationale for selection of tool Health website quality initiatives have been setting standards for ethics, conduct, and information management since the 1990s, and various tools exist to guide users through the evaluation process (Wilson, 2002). User guidance systems enable a reviewer to quickly, easily, and inexpensively see if a health website follows specific quality standards by accessing a series of questions about the health material and providing a ‘score’ to the quality of the website (Dobbins et al., 2018; Wilson, 2002). There are contextual complexities in digital evaluation tools; it is essential to recognize that all tools are not well suited to evaluating the specific information each patient is seeking or requires since it is difficult to generalize the needs of a variety of stakeholders (Gonzales et al., 2013). Most tools fail to address the complexities that arise from patients’ needs and environmental constraints (Gonzales et al., 2013). Kaphingst et al. (2012) state that often, health literacy demands of online health information place a heavy weight on overall literacy by evaluating readability and grade level difficulty, something that is widely varied in the population that would be accessing health information through the BIC website. According to a literature review performed by Dobbins et al. (2018), a considerable amount of effort has been invested in the development of quality assessment tools, but at the time of their publication, it was not previously determined which tool was superior to others in terms of usability and reliability, therefore the WRR tool was developed. This tool was chosen because it has been evaluated for ease of use, as well as reliability and validity in its use on health websites that meet the aforementioned criteria (Dobbins et al., 2018). This tool was designed with high inter-rater reliability so that novice evaluators will obtain similar results as an experienced subject matter expert (SME) (Dobbins et al., 2018). The WRR tool is intended for quality assessment of online web resources, which are: not directly funded by a company trying to sell a product or service and less than five years old (Dobbins et al., 2018). While the BIC website does not share its initial publishing date, and updates to information are seemingly continuous, this website was determined to fit the tool's criteria of age <5 years since the earliest dated information available on the website is from 2021, and the website was copyright updated to 2022 (BIC, n.d.). BIC is funded by the government of Canada, and there are no listed products or professional affiliations that pull secondary revenue for the website (BIC, n.d.). The tool uses a 13-question dichotomous (yes/no) questionnaire that rates health websites based on evidence-base, transparency, and usability (Dobbins et al., 2018). This tool was chosen to evaluate the BIC website because the information on the website is targeted for use by patients, caregivers, and medical professionals alike, so it needs to be assessed across these specific criteria, and especially for quick ease of access to accurate, up-to-date information informed by Canadian best practice guidelines for HCPs, as well as usability for the vulnerable population the website is intended to serve.
Quality of evaluation criteria This evaluation tool was rated and ranked against several other evaluation tools during its development. The WRR tool was developed by researchers at McMaster University, who created a website which determines the quality of resources available on the internet for older adults (Dobbins et al., 2018). They decided that there was a need for an evaluation tool with higher reliability and ease of use than existing tools such as DISCERN, the Information Quality Tool, the Quality Scale, and the LIDA tool (Dobbins et al., 2018). Dobbins et al. (2018) then used an intraclass correlation coefficient (ICC) to determine the correlation between a rating on one tool versus a rating on the other tools (Dobbins et al., 2018). They also assessed these measures for inter-rater reliability (Dobbins et al., 2018). An ICC value is considered ‘good’ at a score of 06.-0.74 and ‘excellent’ if the ICC was >0.75; the WRR tool ranked highest in this assessment (ICC = 0.988), followed by DISCERN (ICC = 0.823), Quality Scale (ICC = 0.79), LIDA (ICC = 0.611), and the Information Quality Tool (ICC = 0.543); while most of the tools rated as highly reliable with ‘good’ or greater quality, the WRR tool was selected since it boasts the greatest reliability with the fewest questions. The dichotomous nature of the questionnaire improves inter-rater reliability, and so as a novice evaluator is performing this evaluation, it is expected that an evaluator with more experience could use the tool and obtain very similar results, which would support a one-rater system of health websites which would save a great deal of professional resources (Dobbins et al., 2018). Dobbins et al. (2018) also measured the WRR tool for ICC scores by question against the other tools, with each question having ‘good’ to ‘perfect’ single scores. Therefore, the tool appears to be both highly reliable and highly valid. It is accepted that health websites should be evaluated across the criteria implemented by the WRR tool, and the WRR tool leaves little room for interpretation. Additionally, it is designed primarily to assess the quality of information presented while evaluating other visually appealing aspects with less weight (ex/ ease of scrolling, size of headings, size of paragraphs, targeted reading levels).
Evaluating the Brain Injury Canada Website
Application of the WRR tool
Upon opening the BIC website, one is immediately struck by an explanation of BIC, a search bar, the option to change between Canada’s official languages, and a bright yellow accessibility menu. The website's homepage has interactive links to obtain information on BI, review a service directory for BI, and reference a new program in development targeted to teens. The WRR tool independently examined many aspects of this thorough website. Appendix A contains the full WRR scorecard for the BIC website.
Evaluating Evidence-base The WRR tool has the user begin by determining if the web resource is informed by evidence (Dobbins et al., 2018). Choosing ‘information on brain injury’ from the navigation menu takes the user to a secondary home page, where information can be obtained by further selecting if the website user is an individual with BI, a family member or caregiver of someone with BI, or an HCP. This evaluation begins by reviewing the information for those who have endured a BI and their lay care providers. Each article has a link to a source list which includes peer-reviewed single studies, relatively current statistics, hospital-based sources, and other reliable published multimedia. There are also cited randomized-controlled trials and systematic reviews, which can be found through secondary sourcing, suggesting that SME authors utilize an adequately reliable evidence base. Best practice guidelines are published in the HCP section of the website and cited in the resource lists of many of the informative articles. The WRR also asks if the website reports on the quality of evidence, which is indirectly conveyed by the website, through a published disclaimer that information is primarily compiled and developed by BIC, that all external influence is properly cited, and that all content is further reviewed by members of the site’s advisory committee, all of whom are SMEs. The website does not report on the strength of the recommendations provided and doesn't utilize GRADE criteria to determine the strength of the recommendation.
Evaluating Transparency The next section of the WRR tool is for transparency of the website and its resources. The BIC website minimally uses in-text citations to substantiate claims and recommendations published on its website. The authors’ or editors’ names and affiliations are not disclosed on every article on this website. There is no advertising. It appears as though the web resource has been updated within the last three years. The website’s copyright is up-to-date, and each source can be followed to an up-to-date webpage or article, including references published within the last three years. The website includes a ‘contact us’ link to provide feedback on the sourced material.
Evaluating Usability The final section of the WRR tool assesses usability. The resource is laid out with a logical flow, but it is not easy to follow. It is likely that it would not be easy to follow for someone who has suffered a BI, as headings and sections are relatively unclear and the article’s central message is not always summarized in an easy-to-understand way. As well, the subject matter is not targeted to a lower than grade 8 reading level, which has been widely accepted as the reading benchmark for published health information online, so for those reasons, the evaluator chose ‘no’ as their dichotomous option as they believed that the bad outweighs the good here. One highlight of the BIC website is its robust accessibility features; while there is no embedded screen reader, there is the option for many visual accessibility adaptions such as increasing or decreasing the size of the text, changing the colour to grayscale, high contrast, negative contrast, or light background; underlining any linked materials; or modifying the font to be more readable (BIC, n.d.).
Scoring using the WRR tool The WRR tool calculates the score by weighting evidence-based criteria as 75% of the final score and transparency and usability as the remaining 25%. The evaluator rated evidence-based as 5/6 criterion being met for a score of 62.5/75. The website scored 5/7 for transparency and usability, with an overall score of 17.86/25. The total score determined for this website is 80.36/100. The WRR tool does not directly indicate what is considered to be a “good” score.
Interpretation of results Unlike tools with Likert-scaled answers, the WRR tool is entirely dichotomous. This leaves little room for the interpretation of results. Keeping the target population in mind, the way the citations and sources were managed for this resource was appropriate since many in-text citations or utilizing Vancouver-style or APA formatting can be distracting or confusing. Overall, the quality of the health information on this website is comprehensive, and non-health information, such as approaching holidays or ordering takeout during the COVID-19 pandemic as an individual with a BI, is appropriate to the user population (BIC, n.d.). Knowing that the BIC website is funded, in large part by the government of Canada enables a website viewer to identify that it is a reputable website easily.
Limitations
The WRR tool references using ‘GRADE’ or ‘GRADE criteria’ but does not describe what that criteria are, or the value of utilizing such criteria, so without further research, a user could not easily determine if the website meets that criterion. The WRR tool was designed for ease of use to get a high-level overview of the quality of health information provided on a website, and so evaluation is dichotomous (yes/no). However, as an educated reviewer, there are limitations in deciding if one wants to accept that dichotomous decision. For example, when reviewing whether there are peer-reviewed sources for each claim/ recommendation on the BIC website, the toll had the evaluator choose ‘yes’ because the claims are substantiated. However, not every aspect of the claim or information provided had a citation attached, so it was impossible to tell if the authors had substantiated every facet of their claim with the overarching cited source, implying the reader should trust that an authoring SME could back up each and every claim. One important consideration that needs to be made with future website evaluation tools is the likelihood that health technology websites utilize social media platforms to disseminate health information as well, which is an informal means of health information sharing but also a common way that the Canadian public receives information on a variety of topics (Goth, 2008). Therefore, as usability and interactivity evolves, so must our evaluation tools, as these websites and health advocacy groups are increasingly linking with their social media pages to increase the spread of their information (Goth, 2008). Conclusion
The Brain Injury Canada website is a high-quality website with quality health information, but like many funded foundations, is at it risk of becoming outdated. It is essential too, to note that this review is only a snapshot in time of this website, and as political interest groups and funding for projects changes, or as health information on the internet evolves, this website might change in its reliability of health information. While the WRR tool is quite helpful in determining accuracy and evidence-supporting information that is disseminated on the BIC website, the limitations of the tool fail to help the user determine if the information shared to their linked social media or whether the format of the information assessed is appropriate to the population that would benefit most from accessing the health information website.
Comments