Background: Many individuals living with dementia want to live in their own homes for as long as possible. To do so, they frequently require assistance with activities of daily living, which is often provided by friends and relatives acting as informal care partners. In Canada, many informal care partners are currently overworked and overwhelmed. Although community-based dementia-inclusive resources are available to support them, care partners often struggle to find them. Dementia613.ca was created to make the process of finding community dementia-inclusive resources simpler and more straightforward by bringing them together in one eHealth website.
Objective: The objective of our study was to determine if dementia613.ca is meeting the goal of connecting care partners and persons living with dementia to dementia-inclusive resources in their community.
Methods: A review and assessment of the website was conducted using 3 evaluation methods: web analytics, questionnaires, and task analysis. Google Analytics was used to collect data related to website use over a 9-month period. Data on site content and user characteristics were collected. Furthermore, 2 web-based self-administered questionnaires were developed: one intended for care partners and persons living with dementia, and the other intended for businesses and organizations interested in serving persons living with dementia. Both gathered data on user characteristics and included standard questions used in website evaluations. Responses were collected over a 6-month period. Scenarios, tasks, and questions were developed for the moderated, remote, and task-analysis sessions. These tasks and questions determined how effectively persons living with dementia and their care partners can use dementia613.ca. Overall, 5 sessions were held with persons experiencing moderate cognitive decline and with care partners of persons living with dementia.
Results: This evaluation showed that the idea behind dementia613.ca is strong and appeals to persons living with dementia, their care partners, and the businesses and organizations serving this market. Participants indicated that it is a useful community resource that meets a previously unfulfilled need in the area, and highlighted the benefits of bringing community resources together on 1 website. In our questionnaire, >60% (19/29, 66%) of people living with dementia and their care partners and 70% (7/10) of businesses and organizations agreed that the website made it easier to find relevant dementia-inclusive resources. There is room for improvement; participants indicated that the navigation and search features could be further developed.
Conclusions: We believe that the dementia613.ca model could be used to inspire and guide the creation of dementia resource websites in other regions in Ontario and beyond. The framework behind it is generalizable and could be replicated to help care partners and persons living with dementia find local resources more easily.
In 2020, it was estimated that 597,300 Canadians were living with dementia . As of 2021, there are an estimated 36,991,981 Canadians, meaning that approximately 1.6% of the population is living with dementia [ , ]. The number of Canadians with dementia will continue to rise, with 78,600 new cases being diagnosed every year in Canada [ ]. By 2030, the number of people living with dementia in Canada is projected to be close to 1 million [ ].
Most persons living with dementia want to live in their own homes for as long as possible, and approximately 61% of persons living with dementia in Canada are able to live at home [, ]. Those living with dementia at home frequently require assistance with activities of daily living, which is often provided by their friends and relatives acting as informal care partners [ , ]. In Canada, informal care partners spend an average of 26 hours a week supporting their loved one with dementia [ ]. Many care partners are currently either overworked or overwhelmed. The Change Foundation’s 2019 Spotlight on Caregiving Report [ ] presents current data on the well-being of care partners, and many are struggling with this role. The report noted that 75% of care partners wished that there was somewhere they could go for advice. In addition, >45% of care partners of older adults with dementia exhibit symptoms of emotional distress, which is almost double the number of care partners of older adults without dementia [ ].
To maintain a good quality of life, persons living with dementia must engage in physically  and mentally stimulating activities [ ] and avoid social isolation [ ]. A lack of these enriching experiences can lead to boredom, which is linked to anxiety, apathy, wandering, and agitation [ ], and contributes to many of the “challenging” or “difficult” behaviors associated with dementia [ ]. It can also be challenging for already overwhelmed care partners to determine the best way to provide these activities. With the many community-based dementia-inclusive resources that are available, finding the appropriate ones can be difficult or even overwhelming for care partners who are already managing multiple tasks.
Currently, there are few community-based platforms with organized and reliable information regarding dementia-inclusive resources in an easy-to-search place . Web-based resource directories could provide this service and transform how persons living with dementia and care partners access information regarding community resources. Although some studies have evaluated projects that aim to connect persons with dementia and care partners with information regarding community resources [ , ], there is a lack of web-based Canadian dementia-friendly resource directories.
The website, dementia613.ca, was created to make the process of finding community dementia–inclusive resources simpler and more straightforward by bringing them together in one website. The “613” in the name of the site refers to the area code for the greater Ottawa area. Before its launch, there was no similar directory in the Ottawa and Renfrew County areas. This website was created through a partnership between The Dementia Society of Ottawa and Renfrew County (DSORC) and the Bruyère Research Institute, supported by funding from the Centre for Aging and Brain Health Innovation Spark program. There were 3 distinct phases involved in the development of dementia613.ca: phase 1, obtaining end user feedback to guide website development; phase 2, low-fidelity prototyping and participatory evaluation and design development; and phase 3, launch and evaluation. This study focuses on the third phase.
The Creation of Dementia613.ca
Input from care partners and other stakeholder groups (eg, individuals with cognitive impairment, clinicians, and DSORC staff member) was sought during the entire design and development process. During the first phase of this project, held before the development process began, stakeholders were invited to participate in a web-based questionnaire to help inform website content and development. It was promoted to memory clinic physicians, as well as DSORC staff members, volunteers, and clients. The questionnaire included 17 questions, and responses were collected using a mixture of open-ended prompts and Likert-type scales. It gathered data on the participants’ characteristics, the types of information they wanted the website to contain, and suggestions for ease of use. Responses informed and guided website development and were referred to throughout the development process. The complete questionnaire and all the collected responses are shown in. The questionnaire was available in English and all 46 respondents who started the questionnaire completed it. Most respondents identified as woman (37/46, 80%) and were aged between 45 and 64 years (25/46, 54%). Half of the respondents (23/46, 50%) identified their role as being a care partner for individual living with memory difficulties or dementia.
On the basis of the feedback gathered from these stakeholders, a website format was selected; searching on the internet was the second most common method for finding dementia-related resources. In addition, the questionnaire results indicated that the website should be mobile friendly so that it could be easily accessed on multiple types of devices. Several stakeholders indicated that they preferred to directly call the DSORC to receive information regarding resources. However, this service is only available during business hours. Stakeholder feedback provided evidence that creating a searchable web-based directory to bring existing resources together would provide a valuable service.
The second phase of this project, the development and design of the website, involved 2 rounds of participatory design testing, using 2 distinct methods. In both rounds, wireframe mock-ups of the website were created, and the participants were asked to complete simple tasks and provide feedback. Between rounds, the website design was refined, and after the second round, the final design was reached.
The first round involved unmoderated, remote tree testing. It focused on testing the overall ease of use of the website’s navigation structure and how easily the information could be found. The participants completed a series of simple tasks and questions to evaluate their ability to follow the proposed navigation logic of the website. A total of 83% (25/30) of individuals completed the study (of the participants who started the study). This round of testing was conducted for 2 weeks in May 2020. The results indicated that the overall navigation structure was efficient, and only minor adjustments were needed where participants struggled. The second round of testing involved live one-on-one task-analysis sessions. This study focused on testing the ease and efficiency of its use. During the sessions, participants were asked a series of questions and navigated the wireframe to find answers. In total, 5 individuals participated in this round of testing for over 1 week in June 2020. The results indicated that users had trouble finding and using the filter-and-sort functions of the website.
On the basis of the results of the second phase, the main features incorporated into dementia613.ca included web-based filtering tools for finding relevant resources from the database with a category search and map view, organizational listings that included specific environmental features to help users better plan a visit, information for businesses about training to become dementia-inclusive, and the ability for organizations and businesses to submit a request for their resources to be considered for inclusion on the website.
The dementia613.ca website is also fully bilingual, available in French and English, because approximately 16% of Ottawa’s population and approximately 5% of Renfrew County’s population is primarily French speaking .
Goal of This Study
With the launch of demetia613.ca, this project entered its third phase, which is the focus of this study. We wanted to determine if dementia613.ca meets its goal of connecting care partners and persons living with dementia to dementia-inclusive resources in their community. The specific objective of our analysis was to evaluate feedback obtained from website users on the utility of dementia613.ca. Thus, a review and assessment of the website is required. In this paper, we present the findings from the multiple evaluation methods used during this phase, including data from web analytics, a questionnaire, and task analysis.
This study focused on evaluating website user demographics, reasons for website use, website use characteristics, and suggestions for ongoing improvement of the website.
In phase 3 of this project, several methods were used to evaluate the launch of dementia613.ca. This included web analytics, self-administered questionnaires, and task analysis.
Google Analytics was used to collect data related to website use . We collected information regarding the number of individuals who used dementia613.ca; the number of sessions (ie, the period in which a user is actively engaged with the website) that have occurred since the launch of the website; as well as demographic information regarding the visitors, geographic locations of the visitors, the frequency of the sessions, when the sessions occurred, and the top site contents viewed. Google Analytics was used to collect information from January 2021 to October 2021.
With regards to web analytics, a “session” refers to the time period users are actively engaged with the website from when they click on the link to enter the site, to when they exit the site; it includes all and any form of interaction (eg, viewing the screen or scrolling through pages on the website) per session by a single user. A page view is the number of pages viewed by a user, and the repeated views of a single page are counted.
A total of 2 web-based, self-administered questionnaires were developed, one was intended for care partners and persons living with dementia and the other was intended for businesses and organizations interested in serving persons living with dementia. Both gathered data on users’ characteristics and included standard questions used in website evaluations. The questionnaires were reviewed and piloted internally by hospital colleagues who did not have background information on the website evaluation project.
The version for individuals collected general demographic information regarding the individual, such as gender, age, and role (role options included individual living with memory difficulties or dementia, care partner to an individual living with memory difficulties or dementia, individual who works with persons living with dementia, health care professional or other). The version for organizations collected information such as organization type and number of employees. Both versions of the questionnaire assessed satisfaction with the design, content, and navigation of the website; frequency of use; users’ perceived trust of the information; and future intentions. Responses to the questions were acquired using a Likert-type scale. In addition, there were open-ended questions for additional comments regarding the website, including reasons for use and features that users would like to see added. The complete questionnaires are shown inand . Both questionnaires were developed to align with others in the literature, including evaluation of eMentalHealth.ca by Jeong et al [ ]. The questionnaire used in that study was based on the Commission of the European Communities’ quality criteria for health-related websites [ ]. In addition, questions were added to both versions of the questionnaire to ensure that they addressed the 7 categories in the User Experience Honeycomb developed by Peter Morville [ , ], that is, useful, valuable, usable, credible, accessible, desirable, and findable. The questionnaires were customized for this study, including questions regarding the specific and unique features of the design, such as map and search functionality. Finally, the questionnaire was developed to align with previous work done in phases 1 and 2, including questions that followed up on the topics of ease of use and relevancy of information.
Questionnaire responses were collected over a 6-month period, from January 2021 to July 2021. The questionnaire was made available through a link on dementia613.ca from March 2021 to July 2021 in both French and English languages. The questionnaire was promoted on posters displayed in a Geriatric Day Hospital and Memory Clinic and in the Dementia Society Monthly Newsletter. It was also sent to the Champlain Dementia Network, a collection of organizations that support persons living with dementia, to share among their networks. To encourage participation, a draw for gift cards was done over an 8-week period.
The testing team developed a scenario, tasks, and questions for a moderated, remote, task analysis session to be held with persons experiencing moderate cognitive decline and the care partners of persons living with dementia. These tasks and questions aimed to determine how effectively persons living with dementia and their care partners could use dementia613.ca. The methods and sample sizes used to evaluate dementia613.ca aligned with the best practices in usability testing. To reach close to the user testing’s maximum benefit:cost ratio, it is standard to perform the test with 5 users per round [- ]. This allows for multiple rounds of testing on iterations of the application, allowing designers and developers to identify and fix problems and then test the redesigned versions [ ].
Tasks were created to test the 2 main features of the website (ie, web-based filtering tools and organizational listings with detailed information). The tasks centered on participants finding resources using the categories in the directory and using the map view to find resources near them. Participants who were not located in the Ottawa or Renfrew County areas were asked to complete the tasks assuming that they lived in downtown Ottawa. The tasks were based on 7 categories in User Experience Honeycomb by Morville [, ]. To drill deeper into issues of accessibility, questions were also developed to determine whether the site content aligned with Web Content Accessibility Guidelines 2.1 [ ]. Questions were designed to reach at principle 1: perceivable and principle 3: understandability [ ].
The tasks and questions were reviewed and piloted internally by DSORC colleagues who did not have background information on the website evaluation project. The participants were recruited from the DSORC mailing list. The list included persons experiencing cognitive decline, current and former care partners, and DSORC dementia care coaches.
Between March 3, 2021 and March 15, 2021, five task analysis sessions, with one participant per session, were conducted. They were conducted remotely over Zoom (Zoom Video Communications, Inc) and video recorded. The participants had varying levels of experience with dementia613.ca; 3 of them had never used the site and the remaining 2 had. Sessions took between 30 and 40 minutes to complete and started with the participant accessing dementia613.ca from their own laptop or tablet and screen sharing their view with the moderator. The moderator started the task analysis by describing the scenario to the participant and then asked them to complete a series of tasks and respond to the questions. Although there were multiple ways to complete the tasks, they did have the correct end state. As such, they were marked “complete” or “incomplete” in the analysis. The time taken by each participant to complete each task was recorded. The questions were open-ended, asking about participants’ opinions and beliefs regarding the tasks and the experience of using dementia613.ca. The responses were analyzed thematically using categories from the User Experience Honeycomb of Morville [, ].
In this scenario, participants were asked to imagine that their relatives had recently been diagnosed with dementia. The participant knew that over time, their relatives’ care needs will increase, and so the participant wanted to learn more regarding local organizations and businesses that were dementia inclusive. They planned to do this using the website dementia613.ca. For the first task, participants started on the dementia613.ca home page and were asked to find information regarding courses or programs related to the arts and crafts. The correct end-state for this task was navigating to the categories Fitness, Exercise & Learning. During the second task, the participants were asked to determine which listings were close to their location. To do this, they had to select See Map View and navigate the provided map. In the third task, the participants were asked to obtain more detailed information on an individual resource. To do this, they had to click on an individual resource, which took them to the resource’s full listing page. For the final task, participants were asked to find information regarding the organizations that run dementia613.ca. The correct end-state for this task was to find and select the About link. After completing the tasks, the participants were asked 4 follow-up questions to examine their overall impressions.
The ethics application was submitted to and approved by The Bruyère Research Institute Ethics Board (# M16-19-042).
Between January and October 2021, Google Analytics reported 3924 sessions and 11,127 page views of dementia613.ca. The average length of each session was 1:40 minutes, and during that time, each user viewed an average of 2.84 pages. The list of Resource Categories and the French version of the site were among the top 5 viewed site features.and and summarize the web analytics collected via Google Analytics.
The vast majority of users (3143/3453, 91.02%) were new to dementia613.ca. More than two-thirds (2326/3453, 67.36%) were Canadian, and more than one-third (1212/3453, 35.1%) were located in Ottawa, Ontario. Approximately 10% (258/3453, 7.47%) were located near Ottawa, such as Gatineau, Quebec, and Ashburn, Ontario. This means that <45% (1470/3453, 42.57%) of users were in relevant locations.
Most users (62/74, 84%) accessed the website using desktop computers. In most sessions, users (2402/3924, 61.21%) reached the website directly. In other sessions, users (736/3924, 18.76%) reached the website with a referral, of which dementiahelp.ca, facebook.com, and bruyere.org were the 3 most common websites.
|Analytic||Data collected from dementia613.ca|
|Timeframe||January 21, 2021 to October 28, 2021|
|Average session duration||1:51 minutes|
|New visitors, n (%)||3143 (91.02)|
|Returning visitors, n (%)||310 (9.86)|
|Values, n (%)|
|Device of access to website (n=74)|
|Website traffic (n=3924)|
|Organic search||556 (14.17)|
Google Analytics data on top website content viewed collected from dementia613.ca from January to October 2021.
Top website content viewed
For the English and French versions of the website, 64% (39/61) of the respondents completed the questionnaire (out of the respondents who started it). Of these completed responses, 96% (28/29) of the responses were to the Care Partner and Persons Living with Dementia Questionnaire in English, 3% (1/29) to the Care Partner and Persons Living With Dementia Questionnaire in French, and 10 responses to the Business and Organization Questionnaire in English.
Most Care Partner and Persons Living with Dementia respondents (26/29, 90%) identified as woman and were aged >65 years (15/29, 52%). The majority of respondents (19/29, 66%) identified their role as being a care partner to an individual living with memory difficulties or dementia. Most Business and Organization respondents (6/10, 60%) worked for a health care organization.
Most Care Partners and Persons Living With Dementia respondents had been to dementia613.ca >5 times. The largest segment of Business and Organization respondents had used dementia613.ca between 2 and 4 times.presents the results of the dementia613.ca questionnaires.
|Characteristics||Care partner and person living with dementia questionnaire (N=29), n (%)||Business and organization questionnaire (N-10, n (%)|
|Individual living with memory difficulties or dementia||0 (0)||N/A|
|Care partner to an individual with memory difficulties or dementia||19 (66)||N/A|
|Individual who works with people living with dementia||5 (17)||N/A|
|Health care professional||3 (10.3)||N/A|
|DSORCb volunteer||2 (7)||N/A|
|Health care||N/A||6 (60)|
|Approximate number of employees|
|Between 20 and 99||N/A||3 (30)|
|Between 100 and 499||N/A||1 (10)|
|Frequency of use|
|Once||2 (7)||1 (10)|
|<5 times||8 (28)||4 (40)|
|Between 5 and 9 times||10 (34)||3 (30)|
|>10 times||9 (31)||2 (20)|
aN/A: not applicable
bDSORC: Dementia Society of Ottawa and Renfrew County
Content and Ease of Use
More than 70% (21/29, 72%) of the care partners and Persons Living respondents and 100% (10/10) of Businesses and Organization respondents strongly agreed or agreed that the website contained relevant information.presents the respondents’ ratings of the design, content, and ease of use of dementia613.ca.
Usefulness of Features
Most of the questionnaire respondents found features of dementia613.ca helpful (). Approximately 70% (20/29, 69%) of Care Partners and Persons Living With Dementia respondents strongly agreed or agreed that the way the resources were organized in categories was helpful, and 90% (9/10) of Businesses and Organizations respondents strongly agreed or agreed that the information provided on becoming dementia friendly was helpful.
However, some features of dementia613.ca were not viewed favorably. Only 48% (14/29) of Care Partners and Person Living with Dementia respondents strongly agreed or agreed that it was helpful to search for resources using the map view.
User Satisfaction With Website
As shown in, the overall user satisfaction with dementia613.ca was very high. More than 90% (27/29, 93%) of Care Partners and Persons Living With Dementia respondents and 100% (10/10) of Businesses and Organizations respondents were very satisfied or somewhat satisfied with the website. In addition, users came away with an overall positive impression of dementia613.ca. More than 80% (24/29, 83%) of Care Partners and Persons Living With Dementia respondents and 100% (10/10) of Businesses and Organizations respondents plan to visit dementia613.ca again ( ).
The respondents were asked to answer specific questions at the end of the questionnaire. Descriptive coding, a type of first-cycle qualitative coding, was used to analyze the comments. In “descriptive coding,” codes are used to summarize the main topic of a passage of qualitative data . This method allows responses to be grouped by theme. All the comments can be found in .
Most comments were positive, stating that the website was a good resource for finding local resources, and that there was no equivalent local resource. Examples of the positive comments for each identified theme are listed in.
Negative comments were helpful because they often either critiqued specific features or offered advice on how to improve websites. Several comments noted that the lack of a search function made it challenging to find specific resources. Other comments noted that although the categories used to organize resources were helpful, the resources in each category appeared disorganized. Representative negative comments for each theme are presented in.
|Question and theme||Questionnaire type||Comment|
|If you only used the website once, why did you NOT return?|
|Newly launched resource|
|What is the reason or reason(s) you are using the website?|
|Supporting friends and family|
|Supporting care partners|
|Supporting business development|
|Education on dementia|
|Do you have any other thoughts you would like to share about the website?|
|Website is helpful|
|Question and theme||Questionnaire type||Comment|
|If you only used the website once, why did you NOT return?|
|Hard to navigate|
|Are there any other features you would like us to add?|
|Do you have any other thoughts you would like to share about the website?|
|Hard to navigate|
We conducted 5 task analysis sessions. Three participants had been care partners for a family member with dementia. Two participants had mild cognitive impairments. Furthermore, 2 of the participants were in Ottawa and the other 3 were in rural communities. Two participants self-identified as older adults (aged ≥65 years). Further details can be found in.
During the task analysis sessions, the participants were given a scenario and asked to complete 4 related tasks. The amount of time it took for users to finish the tasks was recorded, and the average successful completion time for each task is presented in. The reasons for unsuccessful task completion were as follows: in task 1 a participant did not know what category to select and gave up on the task; in task 2, a participant selected individual listings one-by-one and looked at their addresses on the full resource listing page; in task 4, a total of 2 users selected Contact Us instead of the correct choice About.
The data collected during the task analysis sessions were synthesized into takeaways based on 7 categories from User Experience Honeycomb of Morville [, ]. Overall, most of the feedback participants provided during these sessions was positive.
Fundamentally, participants found the idea behind dementia613.ca to be good and they found the information provided to be helpful. Participants thought that the information was reliable, in large part, because of the positive reputation of the DSORC. The participants also found dementia613.ca to be accessible. The positive takeaways are listed in. However, participants also identified several areas in which dementia613.ca could be improved. The area that needs the most improvement is the overall website navigation. Users found that the organization of the website content could be improved and that specific listings were difficult to find. In addition, some users would have liked to see that business and organization listings contained more specific information. Negative and neutral takeaways are presented in .
|Code||Previously used dementia613.ca||Device||Location||Age (years)||Relation to research|
|P1||Yes||Desktop||Ottawa, Ontario||<65||Care partner|
|P2||Yes||Tablet||Renfrew, Ontario||≥65||Care partner|
|P3||No||Tablet||Ottawa, Ontario||≥65||Mild cognitive impairment|
|P4||No||Desktop||South Hampton, Ontario||<65||Mild cognitive impairment|
|P5||No||Desktop||South Hampton, Ontario||<65||Care partner|
|Task||Question asked||Success rate, n (%)||Average time for successful completion (min:sec)|
|Task 1: finding information about courses or programs related to arts and crafts||Where would you expect to find information about courses or programs related to arts and crafts?||4 (80)||1:31|
|Task 2: finding which listings are close to your location||How would you go about finding which listings are close to your location?||4 (80)||1:27|
|Task 3: finding detailed information about a resource||Once you have found a resource that interests you, how might you find out more detailed information about it?||5 (100)||0:24|
|Task 4: finding information about the organizations(s) that run dementia613.ca||Where would you expect to find information about the organizations(s) that run this website?||3 (60)||1:11|
|Category and key takeaways||Comments|
|The resources provided are relevant to user needs.||“These listings look like places I would go.”|
|The full listing pages of resources provide important and relevant information.||“This page provides a good overview of information (contact, blurb, etc.).”|
|Provides enough information for a user to know if a resource is relevant.||“Gives the information you need to make a decision.”|
|Provides users with multiple ways to connect organizations to find more information.||“Gives enough information to know if organization is of interest and to contact them to learn more.”|
|Strong positive reputation of DSORCa.||“I assume that I can trust it because it was put there by The Dementia Society.”|
|Can directly contact organizations from listings.||“I see this information as reliable. I like that there are multiple methods to contact the organization.”|
|Content is legible.||“Content is easy to see; bold items, font size, it’s easy to read, and gives the right information.”|
|Content is easy to understand.||“Wording was good, and language was understandable.”|
|Concept behind dementia613.ca initiative is good.||“Overall, this could be a good resource.”|
|Information on resource listings is well organized.||“The listing is good and makes finding things easier.”|
aDSORC: Dementia Society of Ottawa and Renfrew County.
|Useful||The full listing pages of resources does not provide enough information.||“I would want more detailed and specific information about this resource.”|
|Valuable||Some resource descriptions are not specific enough.||“Some of the descriptions for the resources are not helpful. They should say exactly what the resources are.”|
|Usable||Overall navigation needs improvement.||“Navigation should be made easier, without so much clicking.”|
|Accessible||Content legibility could be improved.||“Font size needs work in places.”|
|Findable||Content is poorly organized.||“Content is useful, but could be organized better.”|
Our evaluation showed that the idea behind dementia613.ca is strong and appeals to persons living with dementia and the stakeholders who serve them. Our evaluation found that although most users were new to the site, they were interested in using it again. Users saw the information on dementia613.ca as credible and felt that it provided them with relevant resources and information. Overall, the website reflects the basic ingredients needed to support a good user experience, aligning with Morville’s principles of “valuable” and “useful” [, ].
The evaluation also identified some areas where dementia613.ca could be improved, namely navigation and search, and clarity around the website’s purpose. The needed changes will align the website with the principles of “findable” and “usable” by Morville [, ]. We are hopeful that with these refinements, and as dementia613.ca becomes a more established and well-known resource, the site will better serve new and existing users.
Relevance to Community
Participants shared that dementia613.ca is a useful community resource. The development process we went through to create the website involved consulting relevant members of the community (persons living with dementia, care partners, businesses, and organizations). The website was shaped by an image of their needs and desires. Our success can be seen by users citing reasons that aligned with the motivation behind its creation, such as supporting the community and providing convenient resources when asked why they were using the website. Furthermore, in task analysis sessions, several participants indicated that they saw the website as desirable, as exemplified by the comment “Overall, this could be a good resource.” This shows dementia613.ca’s alignment with the principle of “valuable” by Morville [, ] by demonstrating that the website provides users with important information. Dementia613.ca is a useful resource that meets previously unmet local needs.
More specifically, dementia613.ca provided relevant resources and information to persons living with dementia and their care partners and to businesses and organizations that want to be dementia-inclusive. When asked, >60% (19/29, 66%) of persons living with dementia and their care partners and 70% (7/10) of businesses and organizations agreed that the website made it easier to find relevant dementia-inclusive resources. In open-ended responses, one participant said, “It has been very helpful for me. I have learned how to cope better. THANK YOU!” This shows dementia613.ca’s alignment with Morville’s principle of “useful” [, ], demonstrating that the website serves a purpose for our stakeholders. All of this illustrates that dementia613.ca is meeting local needs by providing relevant resources and information.
Dementia613.ca is aimed at care partners, who can be of a wide variety of ages, and persons with mild cognitive impairment. It was designed to meet the needs of these users; however, we recognize that it may also be used by people outside of those demographics. To ensure that the site would be as wide as possible, we were guided by Peter Morville’s User Experience Honeycomb framework [, ] and usability design best practices promoted by the Nielsen-Norman Group [ , ]. We also referred to Web Content Accessibility Guidelines 2.1 [ ] to develop the site. Because of the principles we followed during its creation, we feel that dementia613.ca supports accessibility and the inclusion of a variety of stakeholders, recognizing that there is always room for improvement as we learn about the site in use over time.
Areas for Improvement and Future Research
Participants provided constructive feedback that was used to produce further iterations of the design. Areas for improvement include the following:
- Refining how a website is organized as many users found it challenging to navigate. The most suggested improvement was organizing the service listings in an alphabetical order. Another suggestion is to narrow or further specify the categories provided or allow users to filter results within a category. The literature in this area aligns with these recommendations, finding that users appreciate well-designed tools to narrow search results [ ]. In the next round of design and development for dementia613.ca, we will work with stakeholders to develop a more usable navigation structure. This will increase the website’s alignment with Morville’s principles of “findable” and “usable” [ , ] by improving website navigation and making it easier for users to find information they seek.
- Adding a search feature, as many users noted that the map view was not sufficient for finding specific resources. Research in this area has found that website users are goal-driven and look only for the one thing they have in mind and often rely on searching in pursuit of their goal [ ]. This supports the findings of our task analysis and strengthens the case of adding a search feature. There are established best practices for developing strong on-site search engines that enable good user experience [ ]. However, best practices also caution against prioritizing searches at the expense of navigation [ ]. Thus, in the next round of design and development for dementia613.ca, we will work to develop both search and navigation and ensure alignment between these tools. This will further increase the website’s alignment with the principles of “findable” and “usable” [ , ].
- It is more explicit about who the website is for and what information it provides. Some users seemed unclear about the purpose of the website. In addition, a surprisingly high number of users from outside Ottawa have accessed this website. We are not sure why this occurred but it could be due to the site appearing in people’s search queries, word of mouth from people receiving the recruitment notice and passing the information onto community members outside the region, or other potential factors. This may mean that we need to clarify the purpose of the website and make it more obvious that the resources are for Ottawa and the surrounding areas.
The Google Analytics data we collected indicate that users, on average, did not spend a long time on individual pages of dementia613.ca. This could mean that individuals were looking for a specific resource or answering a specific question and were able to find the page they needed quickly. This interpretation aligns with the results from our questionnaire; approximately 60% (17/29, 59%) of the care partners and Persons Living With Dementia respondents strongly agreed or agreed that they were quickly able to find the information they were interested in on the website. However, short site visits could also indicate that individuals visiting the site only quickly browsed the resources before leaving, perhaps not finding useful information. Further studies are needed to understand this phenomenon. In addition, only 9% (310/3453, 8.98%) of the website visitors returned. This may be due to the timing of Google Analytics data collection, which was shortly after the launch of the website. This did not give the site much time to build a user base. Taken together, Google Analytics data brings up good points for future work to evaluate the website, which should aim to include a sample of previous website users. It should ask questions about their perceptions of the site’s utility, their primary uses for the site (using the website to find one specific resource vs using it to search for multiple different types of resources available in the area), and their reasons for returning to the site.
Relevance to the Development of eHealth Resources
Not all web-based health resources are created equally, and the best attempts at providing useful information can fall short. To address this, there is a growing body of research on the specific information needs of internet users and their requirements for web-based health information . There is also more specialized research on the barriers faced by users from marginalized populations [ , ]. A common thread among these studies is the importance of incorporating user feedback, priorities, and concerns into the design of web-based health resources. This information can be gathered by conducting evaluations with users, making it a critical step in creating useful web-based health resources.
Currently, there is a limited amount of literature evaluating public-facing health resource websites. Much of the existing research focuses on e-mental health resources [, - ]. These studies have found that the general public is interested in accessing quality-health resources on the web [ , ]. This aligns with our results and demonstrates the need for an increased number of reliable, public-facing health resource websites.
Other studies have found that there is a specific interest in accessing quality health resources on the web among traditionally underserved populations, including people in rural communities , persons with mild-intellectual disability [ , ], and adolescents and their parents [ , ]. Currently, there are scant quality web-based health resources for care partners of persons living with dementia; however, they are an overworked and overwhelmed group [ , ] that would greatly benefit from the support that this type of resource can provide. Persons living with dementia are a similarly underserved population. Dementia613.ca aims to serve both types of demographics.
Within the context of design, there is growing literature on best practices for evaluating websites in the health sector. Currently, work in this area often focuses on the websites of health institutions, such as hospitals and health departments [- ]. This literature emphasizes the importance of working with communities and using a mixed methods approach to ensure that feedback is well-rounded and relevant [ , ]. Although dementia613.ca is a website for community resources, not for a health institution, we feel that the broader principles advanced in this body of research are aligned with these principles and approaches.
This study has several limitations. First, web analytics were only collected for 9 months, which is a shorter time frame than has been used in comparable research projects where the analytics have captured 12 months of data . Furthermore, during the initial weeks in which web analytics were collected, final changes were made to the website design. In addition, during the data collection period, dementia613.ca was a newly launched website without an established user base. Thus, there is limited traffic to capture in the web analytics data. In future research, analytics should be conducted over a longer period. In addition, in any future evaluation, dementia613.ca will be more established, presumably with a larger flow of traffic.
The number of responses to the web-based questionnaires was small, leading to a limited sample size, especially for the Businesses and Organizations group. There were only 10 responses from this group, representing the health care and government sectors. This limited sample size could be viewed as homogenous. Future studies could benefit from purposive sampling to engage broader sector representation (ie, government organizations, community organizations, and private industry) to determine how well the site serves different needs and expectations among service providers.
The Care Partners and Persons Living With Dementia respondents skewed toward older adults and woman. Dementia613.ca was not specifically created for this demographic, suggesting that its responses may not be representative of all website users. Further research should gather feedback from a broader range of users; however, the testing done during phases 1 and 2 had responses from participants with a wide range of ages. Approximately 70% (34/46, 74%) of the responses to the phase 1 questionnaire were from individuals aged <65 years. Similarly, in phase 2, all 5 participants in the task analysis sessions were between the ages of 40 and 65 years. Thus, the site was developed using feedback from different age groups.
Furthermore, no one who self-identifies as an “individual living with memory difficulties or dementia” completed the web-based questionnaire. We promoted the study to this population through posters displayed at a Geriatric Day Hospital and Memory Clinic and in the Dementia Society Monthly Newsletter, but unfortunately did not receive any responses from this population. This limits the usability of the questionnaire results for understanding the population. However, 2 of the 5 participants in the task analysis sessions had mild cognitive impairment. Thus, the data from the task analysis sessions indicate that the initial design of dementia613.ca is appropriate for this stakeholder group. This information will be helpful to DSORC staff member when planning the future promotion of the website to both care partners and persons living with dementia.
Although Neilsen [, ] suggested that 5 users are a reasonable number to conduct an evaluation of a website, a limited number of task analysis sessions were conducted. It has also been suggested that for qualitative studies using nonprobability sampling techniques with a homogeneous population, saturation (where potentially similar patterns emerge and no new information or insight emerges) can occur with 4-12 participants [ , ]. The small number of participants could mean that the results of the task analysis were not representative of all website users. This could be addressed in future research by reaching out to more potential participants (eg, from each stakeholder group of persons living with dementia, care partners, and businesses or organizations) and having a longer testing period. Despite this, we believe that there were enough participants to achieve saturation for the initial design of dementia613.ca.
Taken together, the amount of data collected through web analytics, questionnaires, and task analysis sessions provided enough information for an initial evaluation of the utility and ease of use of the website. The data will be used as the basis for future work aimed at improving dementia613.ca.
Dementia613.ca was well received by most of the visitors. It will be a useful resource for the Ottawa area; according to the feedback we were able to collect from business owners and care partners. The participants indicated that they met a previously unfiled need in the area. Participants highlighted the utility of bringing local resources together in a web-based format in dementia613.ca. Both persons living with dementia and their care partners and businesses and organizations said that dementia613.ca provided reliable and relevant information. There is still room for improvement; participants pointed out that navigation and search features could be developed further. With further updates to the website, methods similar to those outlined in this paper will be used to obtain additional user feedback for further evaluation.
This website model can be used to inspire and guide the creation of dementia resource websites in other regions in Ontario and Canada. We believe that the framework behind dementia613.ca is generalizable and could be replicated in other regions to help care partners and persons living with dementia find local resources more easily.
This research was supported by a grant from the Centre for Aging and Brain Health Innovation Spark program (SPARK-4-00259). MT was supported by the Student Work Placement Program from BioTalent (Canada). The data sets generated and analyzed during this study are available from the corresponding author upon reasonable request.
Conflicts of Interest
DH is an employee of the Dementia Society of Ottawa and Renfrew County. Authors MT, CT, and NT have no conflicts of interest to declare.
Phase 1 survey: questions and responses.XLSX File (Microsoft Excel File), 19 KB
Complete the questionnaire in English. Care partners and persons living with dementia filled out questions 1-14, and businesses and organizations filled out questions 1 and 15-26.DOCX File , 80 KB
Complete questionnaire in French. Care partners and persons living with dementia filled out questions 1-14, and businesses and organizations filled out questions 1 and 15-26.DOCX File , 83 KB
Questionnaire responses in English and French.XLSX File (Microsoft Excel File), 19 KB
- The landmark study - report 1: navigating the path forward for dementia in Canada. Alzheimer Society Canada. 2022. URL: https://alzheimer.ca/sites/default/files/documents/Landmark-Study-Report-1-Path_Alzheimer-Society-Canada.pdf [accessed 2023-01-10]
- Census profile, population and dwelling counts: Canada and census subdivisions (municipalities). Statistics Canada. 2022 Feb. URL: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810000201 [accessed 2023-01-10]
- A dementia strategy for Canada: together we aspire. Public Health Agency of Canada. 2019. URL: https://www.canada.ca/en/public-health/services/publications/diseases-conditions/dementia-strategy.html [accessed 2021-07-01]
- Førsund LH, Grov EK, Helvik AS, Juvet LK, Skovdahl K, Eriksen S. The experience of lived space in persons with dementia: a systematic meta-synthesis. BMC Geriatr 2018 Feb 01;18(1):33 [FREE Full text] [CrossRef] [Medline]
- Dementia in home and community care. Canadian Institute for Health Information. URL: https://tinyurl.com/2xptvnwr [accessed 2021-07-10]
- Hadjri K, Rooney C, Faith V. Housing choices and care home design for people with dementia. HERD 2015;8(3):80-95 [FREE Full text] [CrossRef] [Medline]
- Dementia in Canada: summary. Canadian Institute for Health Information. 2018. URL: https://www.cihi.ca/en/dementia-in-canada/dementia-in-canada-summary [accessed 2021-07-10]
- 2nd annual spotlight on Ontario's caregivers. The Change Foundation: Exchange, Explore, Evolve. 2019. URL: https://changefoundation.ca/wp-content/uploads/2019/11/Spotlight-on-ontario-caregivers-2019_final.pdf [accessed 2021-07-10]
- Sajeev G, Weuve J, Jackson JW, VanderWeele TJ, Bennett DA, Grodstein F, et al. Late-life cognitive activity and dementia: a systematic review and bias analysis. Epidemiology 2016 Sep;27(5):732-742 [FREE Full text] [CrossRef] [Medline]
- Ducharme F, Kergoat MJ, Antoine P, Pasquier F, Coulombe R. The unique experience of spouses in early-onset dementia. Am J Alzheimers Dis Other Demen 2013 Sep;28(6):634-641 [FREE Full text] [CrossRef] [Medline]
- Dementia 2012: a national challenge. Alzheimer's Society. 2012. URL: https://www.alzheimers.org.uk/sites/default/files/migrate/downloads/alzheimers_society_dementia_2012-_full_report.pdf [accessed 2021-07-10]
- Elliot G. Banning the boredom. Montessori for Dementia and Ageing. 2017 Jul 24. URL: https://montessoridementia.org/blog/banning-boredom [accessed 2021-07-10]
- Toronto Dementia Network. 2016. URL: https://tdn.alz.to/ [accessed 2021-07-10]
- Boyd K, Doyle K, Borson S, Huang E, Lindau S, Nathan A, et al. A community resource referral intervention for informal caregivers of people with dementia. Innov Aging 2020;4(Suppl 1):148-149 [FREE Full text] [CrossRef]
- Venohr I, Fine R, Saunders V, Tenney E, Vahan V, Williams M. Improving dementia care through community linkages: a multi-site demonstration project. Home Health Care Serv Q 2001;19(4):51-63. [CrossRef] [Medline]
- Census profile, 2016 census. Statistics Canada. 2017. URL: https://tinyurl.com/57jnmb35 [accessed 2021-07-10]
- Clark DJ, Nicholas D, Jamali HR. Evaluating information seeking and use in the changing virtual world: the emerging role of Google Analytics. Learn Publ 2014 Jul 01;27(3):185-194 [FREE Full text] [CrossRef]
- Jeong D, Cheng M, St-Jean M, Jalali A. Evaluation of eMentalHealth.ca, a Canadian mental health website portal: mixed methods assessment. JMIR Ment Health 2019 Sep 06;6(9):e13639 [FREE Full text] [CrossRef] [Medline]
- Commission of the European Communities‚ Brussels. eEurope 2002: quality criteria for health related websites. J Med Internet Res 2002 Dec;4(3):E15 [FREE Full text] [CrossRef] [Medline]
- Morville P. User experience design. Semantic Studio. 2004. URL: https://semanticstudios.com/user_experience_design/ [accessed 2022-03-10]
- Macpherson E. The UX honeycomb: seven essential considerations for developers. Medium. 2019 Oct. URL: https://medium.com/mytake/the-ux-honeycomb-seven-essential-considerations-for-developers-accc372a398c [accessed 2021-01-10]
- Nielsen J. How many test users in a usability study? Nielsen Norman Group. 2012 Jun 03. URL: https://www.nngroup.com/articles/how-many-test-users/ [accessed 2022-02-10]
- Nielsen J. Why you only need to test with 5 users. Nielsen Norman Group. 2000 Mar 18. URL: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ [accessed 2022-02-10]
- Nielsen J. Usability engineering at a discount. In: Salvendy G, Smith MJ, editors. Proceedings of the Third International Conference on Human-Computer Interaction on Designing and Using Human-Computer Interfaces and Knowledge Based Systems. 2nd edition. Amsterdam, The Netherlands: Elsevier Science Publishers; 1989:394-401.
- Nielsen J. Parallel & Iterative Design + Competitive Testing = High Usability. Nielsen Norman Group. 2011 Jan 17. URL: https://www.nngroup.com/articles/parallel-and-iterative-design/ [accessed 2022-02-10]
- Henry SL. WCAG 2 overview. Web Accessibility Initiative. 2022. URL: https://www.w3.org/WAI/standards-guidelines/wcag/ [accessed 2022-01-10]
- Saldaña J. Chapter 3: First cycle coding methods. In: Saldaña J, editor. The Coding Manual for Qualitative Researchers. 3rd edition. Thousand Oaks, CA, USA: SAGE Publications; 2015.
- Morgan K. Defining helpful filter categories and values for better UX. Nielsen Norman Group. 2018 Jul 15. URL: https://www.nngroup.com/articles/filter-categories-values/?lm=navigation-menu-visibility&pt=youtubevideo [accessed 2022-03-10]
- Nielsen J. Is navigation useful? Nielsen Norman Group. 2000 Jan 08. URL: https://www.nngroup.com/articles/is-navigation-useful/ [accessed 2022-03-10]
- Laubheimer P. 7 ways to improve your website’s or intranet’s built-in search engine. Nielsen Norman Group. 2016 Mar 13. URL: https://www.nngroup.com/articles/internal-website-search/ [accessed 2022-03-10]
- Budiu R. Search is not enough: synergy between navigation and search. Nielsen Norman Group. 2014 Sep 07. URL: https://www.nngroup.com/articles/search-not-enough/ [accessed 2022-03-10]
- Wollmann K, der Keylen PV, Tomandl J, Meerpohl JJ, Sofroniou M, Maun A, et al. The information needs of internet users and their requirements for online health information-a scoping review of qualitative and quantitative studies. Patient Educ Couns 2021 Aug;104(8):1904-1932. [CrossRef] [Medline]
- Reen GK, Muirhead L, Langdon DW. Usability of health information websites designed for adolescents: systematic review, neurodevelopmental model, and design brief. J Med Internet Res 2019 Apr 23;21(4):e11584 [FREE Full text] [CrossRef] [Medline]
- Alshammari M, Doody O, Richardson I. Barriers to the access and use of health information by individuals with intellectual and developmental disability IDD: a review of the literature. In: Proceedings of the 2018 IEEE International Conference on Healthcare Informatics. 2018 Presented at: ICHI '18; June 4-7, 2018; New York, NY, USA p. 294-298 URL: https://ieeexplore.ieee.org/document/8419373 [CrossRef]
- Adams A, Davies V, Stubbs B. Producing a new website for a paediatric liaison mental health team with our service users. BMJ Open Qual 2020 Nov;9(4):e001128 [FREE Full text] [CrossRef] [Medline]
- Köhnen M, Dreier M, Freuck J, Härter M, Dirmaier J. Akzeptanz und Nutzung einer Website mit Gesundheitsinformationen zu psychischen Erkrankungen – www.psychenet.de. Psychiatr Prax 2022 May;49(4):205-212. [CrossRef] [Medline]
- Ogrodniczuk JS, Beharry J, Oliffe JL. An evaluation of 5-year web analytics for HeadsUpGuys: a men's depression e-mental health resource. Am J Mens Health 2021;15(6):15579883211063322 [FREE Full text] [CrossRef] [Medline]
- Watfern C, Heck C, Rule C, Baldwin P, Boydell KM. Feasibility and acceptability of a mental health website for adults with an intellectual disability: qualitative evaluation. JMIR Ment Health 2019 Mar 28;6(3):e12958 [FREE Full text] [CrossRef] [Medline]
- Sutherland S, Jeong D, Cheng M, St-Jean M, Jalali A. Perceptions of educational needs in an era of shifting mental health care to primary care: exploratory pilot study. JMIR Form Res 2022 Jan 07;6(1):e32422 [FREE Full text] [CrossRef] [Medline]
- Schoenberg N, Dunfee M, Yeager H, Rutledge M, Pfammatter A, Spring B. Rural residents' perspectives on an mHealth or personalized health coaching intervention: qualitative study with focus groups and key informant interviews. JMIR Form Res 2021 Feb 26;5(2):e18853 [FREE Full text] [CrossRef] [Medline]
- Oudshoorn CE, Frielink N, Nijs SL, Embregts PJ. eHealth in the support of people with mild intellectual disability in daily life: a systematic review. J Appl Res Intellect Disabil 2020 Nov;33(6):1166-1187 [FREE Full text] [CrossRef] [Medline]
- Ryan GS, Haroon M, Melvin G. Evaluation of an educational website for parents of children with ADHD. Int J Med Inform 2015 Nov;84(11):974-981. [CrossRef] [Medline]
- Puspitasari I, Cahyani D, Taufik. A user-centered design for redesigning E-government website in public health sector. In: Proceedings of the 3rd International Seminar on Application for Technology of Information and Communication. 2018 Presented at: iSemantic '18; September 21-22, 2018; Semarang, Indonesia p. 219-224. [CrossRef]
- Sarantis D, Soares D. Literature on website evaluation in health sector. In: Proceedings of the 10th IADIS International Conference Information Systems. 2017 Presented at: IICIS '17; April 10-12, 2017; Budapest, Hungary p. 93-100 URL: http://collections.unu.edu/view/UNU:7410
- Sarantis D, Soares DS, Carvalho J. HSWAI: a health sector website assessment instrument. In: Proceedings of the 13th International Conference on Theory and Practice of Electronic Governance. 2020 Presented at: ICEGOV '20; September 23-25, 2020; Athens, Greece p. 359-368 URL: https://dl.acm.org/doi/10.1145/3428502.3428551 [CrossRef]
- Sarantis D, Soares D, Carvalho J. Developing health sector websites assessment instrument: challenges and pitfalls. Eval Program Plann 2022 Jun;92:102065. [CrossRef] [Medline]
- Hignett S, McDermott H. Qualitative methodology. In: Wilson JR, Sharples S, editors. Evaluation of Human Work. 4th edition. Boca Raton, FL, USA: Taylor & Francis Group; 2015:119-138.
- Kuzel KJ. Sampling in qualitative inquiry. In: Crabtree BF, Miller WL, editors. Doing Qualitative Research. Thousand Oaks, CA, USA: Sage Publication; 2012:33-47.
|DSORC: Dementia Society of Ottawa and Renfrew County|
Edited by T Leung, Y Jiang; submitted 21.07.22; peer-reviewed by A Langener, A Toohey, V Provencher, B Davis; comments to author 12.01.23; revised version received 09.02.23; accepted 04.03.23; published 20.04.23Copyright
©Missy Thomas, Dean Henderson, Chantal Trudel, Neil Thomas. Originally published in JMIR Aging (https://aging.jmir.org), 20.04.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Aging, is properly cited. The complete bibliographic information, a link to the original publication on https://aging.jmir.org, as well as this copyright and license information must be included.