Object-Based Access: Enhancing Accessibility with Data-Driven Media

Authors

ACM International Conference on Interactive Media Experiences (IMX ’23), June 12–15, 2023, Nantes, France

DOI: https://doi.org/10.1145/3573381.3596500

Abstract

Audiovisual media is an integral part of many people’s everyday lives. People with accessibility needs, especially people with complex accessibility needs, however, may face challenges accessing this content. This doctoral work addresses this problem by investigating how complex accessibility needs can be met by content personalisation by leveraging data-driven methods. To this end, I will collaborate with people with aphasia, a complex language impairment, as an exemplar community of people with complex accessibility needs. To better understand the needs of people with aphasia, I will use collaborative design techniques to meet the needs of end users. This will involve them in the design, development and evaluation of systems that demonstrate the benefits of content personalisation as an accessibility intervention. This paper outlines the background and motivation to this PhD, the work that has already been completed, and current planned future work.

CCS Concepts

Human-centered computing: Accessibility

Human-centered computing: Accessibility systems and tools

Human-centered computing: Empirical studies in accessibility

Keywords

  • Accessibility
  • personalization
  • audiovisual media
  • aphasia
  • object based media

Introduction

Audiovisual media content is ubiquitous in everyday life. We enjoy watching TV broadcasts, films in cinemas, streaming online content, and much more. These, however, are not always accessible to all people. There are currently 14.4 million people in the UK who have a disability, with almost half being 65-years or older [17]. While there has been research focusing on making audiovisual media accessible to people with disabilities, this research has focused mostly on narrow user groups, such as people who are blind and visually impaired (BVI) or d/Deaf and hard of hearing (DHH). Existing accessibility intervention technologies that aim to help these communities have been extensively researched, such as subtitles [12] and audio description [4].

This focus, however, has left a gap in knowledge when it comes to making audiovisual media more accessible to people with complex needs, such as aphasia. For instance, due to the variable nature of complex communication impairments, symptoms and severity may vary significantly from person to person, meaning any “one size fits all” approach to audiovisual media accessibility interventions is unlikely to be successful. Furthermore, existing accessibility guidelines have a limited applicability when it comes to these communities [13]. Therefore, this PhD aims to explore this knowledge gap by working with an exemplar community with complex needs, people with aphasia (PWA), and investigating ways in which audiovisual media can be made more accessible to them.

Aphasia is an acquired communication disorder that is usually caused by damage to the brain due to a stroke or head injury. PWA can find reading, writing, listening, and/or speaking challenging, with different people experiencing aphasia differently, as well as experiencing distinct levels of challenge. Aphasia does not affect the persons intellect. There are currently over 367,000 PWA in the UK alone, with millions more in other countries [10]. Moreover, this is a population that is prone to increase in the coming years due to the odds of surviving a stroke increasing [6]. The motivation of this PhD, therefore, is to address this gap in knowledge by leveraging co-design techniques and novel technologies to explore how personalisation and customisation can be used to address accessibility challenges faced by PWA when consuming audiovisual media.

Research Questions

The scope of this PhD work would be set by the following research questions:

  1. What are the challenges faced by PWA when accessing digital, audiovisual media and how can we explore these through co-design practices?
  2. In what ways can personalisation be used to bridge accessibility gaps in digital, audiovisual media for PWA?
  3. How can we leverage data-driven methods to expand accessibility interventions into novel directions through customisation and personalisation?

Background

The field of audiovisual media has a long history of exploring accessibility challenges, from more conventional technologies, such as TV [14] and smartphones [32], to challenges with the interaction of the device [3]. These have focused on a wide range of communities, affecting the BVI [32] and DHH [7] communities, as well people cognitive impairments [31] and motor impairment [16], among others. The inherent complexity of having a constant stream of both audio and video information presented can introduce further cognitive [28] and language [5] challenges. Moreover, with the advancement of technology, accessibility research has shifted towards novel types of media and devices, such as virtual reality (VR) [21], augmented reality (AR) [23], and mixed reality (MR) [19]. With this shift in technologies focused, and the novel ways in which audiovisual media is both created and consumed, accessibility research needs to be at the core in order to not exclude users with access needs. For instance, a systematic literature review published by [30] on media accessibility research published at IMX/TVX shows that only 4.2% of papers addressed the needs of users with disabilities, focusing primarily on the BVI and DHH communities.

Recent developments in broadcast technologies, especially the emergence and development of object-based media, may offer a novel research direction through allowing more dynamic content personalisation. Object-based media (OBM) is a technology in which media content is created using media blocks that hold a wide range of data and are rendered into a cohesive structure based on that data. This has the potential for the same media to be presented in unique ways for different people at play time. OBM has been used to create interactive storytelling [29] or a personalised cooking show, in which the viewer controls various aspects of the presentation and pacing [8]. Moreover, the personalisation of content is something that has been used to extend accessibility capabilities [15]. An example of the use of increased accessibility offered by OBM personalisation is allowing hard of hearing people to personalise the audio during content playback, adjusting audio levels of different types of audio tracks to improve speech intelligibility [33, 34]. OBM, and the accessibility possibilities it offers, have been little explored, however, with most research focusing on established accessibility interventions [20].

Table 1: This table shows the proportion of papers in the SLR dataset that contained codes for the accessibility challenge explored, who the community of focus was, and what accessibility intervention was explored. For all three of these categories, papers could be coded with more than one code.
Accessibility challenge Papers w/ code Community of focus Papers w/ code Accessibility intervention Papers w/ code
Viewing video 43.1% BVI 45.9% Subtitles 48.1%
Hearing audio 12.2% DHH 53.6% Audio description 33.1%
Reading subtitles 42.5% Motor impairment 1.7% Tangible device 6.1%
Understanding speech 9.4% IDD 0.6% Audio/video manipulation 13.8%
Following narrative 19.9% Autism 0.0% Sign language 5.0%
Issues with image 5.5% Other cognitive 3.3% Content personalization 1.1%
Screen clutter 0.0% Older adults 4.4% Customisation 13.8%
Other 20.4% General disability 8.8% Second display 7.7%
Other 2.8% Other 34.8%

Completed Work

At this time, the completed work aimed to highlight current and past accessibility intervention research for digital, audiovisual media. Along with this, work has been done with the aim to start answering RQ1.

Systematic Review: Accessibility Research in Digital Audiovisual Media

In order to ground my research into the wider accessibility field and inform the future work done in my PhD, a systematic literature review (SLR) on accessibility interventions for digital audiovisual media was conducted. The SLR aimed to understand the challenges explored in the past, highlighting existing accessibility measures, and demonstrate any shortcomings or opportunities missed by the research community. This work modelled on prior SLRs of accessibility research [9, 18], using the PRISMA method [24] and following additional guidelines from [27], [26], and [36]. This work was also an important contribution on its own, as there has not been any systematic review of literature focusing on accessibility interventions for digital audiovisual media. Previous SLR have examined accessibility more generally, such as [18] examining accessibility research at the CHI and ASSETS conferences, or a review published by [30] on media accessibility research at the IMX/TVX conference. Other reviews have focused on specific disabled communities, such as [2] exploring assistive technologies for people with visual impairments, or have been overviews of specific accessibility interventions, such as collaborative technologies for children with special needs [1].

This SLR looked at 181 papers spanning a 26-year period (1996–2022), answering question focusing on the types of accessibility challenges researched, the communities this research focused on, and the accessibility interventions explored. This was achieved through the iterative creation of a codebook and manually coding all the papers, with a second coder validating the coding and a Fleiss’ Kappa inter-rater reliability (IRR) [11] calculation being performed. Through this work, we were able to answer the initial questions we had set. Most of the past researched had focused on challenges related to the viewing the visual aspect of the media and the accessibility of subtitles/closed captions, with relatively little research exploring other challenges, such as understanding speech or issues engaging with the narrative. We also see that the majority of research focuses on the BVI and the DHH communities, with significantly fewer papers exploring the challenges faced by people with intellectual and developmental disabilities (IDD), Autism, other cognitive disabilities, and older adults. We also see that people with disabilities are well represented in user studies, however other participant groups, such as specialists and caregivers, were less represented. Subtitles and audio description were by far the most common intervention methods examined. These results can be seen in Table 1.

Survey and focus group sessions with PWA

In order to answer RQ1, the next step of this PhD is a series of user studies with PWA, which will be achieved in two parts. First, an online survey with more general questions with the aims to gather a baseline understanding of the context in which PWA consume audiovisual media, the accessibility interventions they use, and surface level questions about accessibility challenges. The responses from this survey will inform the second user study, a set of two in-person focus group sessions with PWA and speech and language therapists (SLT). In these sessions, I will explore specific challenges faced by PWA when accessing digital, audiovisual media through a semi-structured interview question set and group discussion activities. In order to aid the discussion, both low-fidelity communication aids, such as the use of physical artefacts to facilitate communication [25], and high-fidelity interventions, such as the use of existing accessibility interventions and simulated examples of novel intervention ideas, will be used to replicate real life media consumption patterns. Previous research shows that PWA prefer using high-fidelity prototypes in order to reduce the amount of abstraction required and prevent cognitive fatigue [35]. Through this exploratory work, I will be able to better understand specific challenges faced by PWA and the way in which they use existing accessibility interventions. The focus group sessions will also allow me to explore potential ideas on ways to leverage personalisation, and to get the thoughts of end users on potential novel accessibility interventions prior to creating technical prototypes. As of writing this, however, this work is ongoing, and therefore does not have results to share.

Planned Future Work

Grounded in findings from the initial survey and in-person focus group session, I will start addressing RQ2 and RQ3. Specifically, the proposed accessibility intervention concepts will be explored initially with experts in content creation prior to system design and prototype development.

Exploring ideas with expert content creators

Following the initial idea gathering phase of the PhD, in which the scope of the interventions will be explored with PWA, I will conduct workshops with experts in the field of content creation. My current tentative plan for this phase is to organise these workshops with the BBC and other content creators at Media City UK and would include content creators and members of the BBC Research and Development team. In these workshops, the intervention ideas will be discussed to understand the aspects of the media content will be modified, the feasibility of these solutions within the production pipeline, and how such a technical intervention would be designed. These discussions would go towards answering RQ2 through building an understanding on how personalisation can be leveraged into an accessibility intervention. Additionally, there is a serious discussion to be had about the implications of using content personalisation as an accessibility intervention. For instance, an inherent tension exists between the artistic vision content creators may have for a specific piece of media they create and the altering of that media to accommodate people with accessibility needs. Moreover, the accessibility interventions may potentially use machine learning or artificial intelligence (ML/AI) tools, such as ChatGPT, to change the language in a piece of media to be more accessible for PWA, which could raise questions on the ownership of art.

System design and development

Informed by discussions with PWA and experts in the field of content creation, I will begin the design and implementation of the accessibility intervention prototypes. This will be done through an iterative co-design approach in which PWA will be involved at each stage of the process. For instance, a key aspect of such interventions is how the user can input their needs, which can be done in numerous ways, such as using abstractions (e.g., the use of personas that represent different aphasia needs [22]) or allowing the user to control a wide range of settings. The various design ideas will be initially demonstrated with high-fidelity prototypes and explored using Wizard of Oz style implementation during workshops. High-fidelity prototyping will be done using tools such as Figma1 to design and explore various user stories, and StoryFormer2, a BBC MakerBox3 tool for creating branching media content with conditional logic, to evaluate how changing various aspects of the content affects its accessibility. For the actual technical prototypes, it is currently unclear the exact format it will have, such as web-based platform, mobile or desktop application, therefore the exact technologies I will employ are not yet clear.

System evaluation

At this stage of the research, I will be performing evaluation of the designed systems. Similarly to the system development, the evaluation of systems created during this PhD will be done in collaboration with PWA. This will include evaluating specific aspects, such as the interaction and control of the personalisation, as well as whole system evaluations. Through initial sessions with PWA, I will collect feedback on existing accessibility interventions and identify the challenges and opportunities for implementing my novel technical solutions, as well as understand how such interactions should be modelled. After the development of early-stage prototypes, these will be introduced and explored in focus group sessions with PWA and SLT to evaluate the early design concepts, the interaction enabling personalisation and customisation, and the utility of these interventions. Following this iterative design approach, once a functioning accessibility intervention system is developed, usability testing will be conducted. It is not fully clear at this moment, however, what form this testing will take, since I do not yet know the needs of PWA, what the form of the accessibility intervention system will be, or whether it would make sense to compare this system with existing accessibility interventions. For instance, the system evaluation could take place in individual or group usability testing sessions in laboratory conditions, or it could be evaluated over a longer period of time by asking PWA to use it in their day-to-day life.

Contribution

Through this PhD research, I aim to make three main contributions. The first contribution comes through the qualitative research done to investigate the accessibility challenges PWA face when consuming digital, audiovisual media. Secondly, through an exploration of content personalisation and how this can be leveraged to improve the accessibility of media for PWA, I add to existing research on accessibility of audiovisual media and demonstrate how the variability of aphasia prevents any “one size fits all” approach from being effective. Thirdly, through the design and development of technical accessibility interventions, my work will contribute insights into how such accessibility interventions can be designed, implemented, and evaluated in an inclusive manner.

Acknowledgements

I would like to thank my two supervisors - Dr Timothy Neate and Professor Elena Simperl - for their assistance and guidance in these initial steps of my PhD journey. This PhD is funded by an EPSRC DTP studentship.

Notes

  1. https://www.figma.com/ Back to text
  2. https://www.bbc.co.uk/makerbox/tools/storyformer Back to text
  3. https://www.bbc.co.uk/makerbox/ Back to text

References

  1. Gökçe Elif Baykal, Maarten Van Mechelen, and Eva Eriksson. 2020. Collaborative Technologies for Children with Special Needs. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3313831.3376291
  2. Alexy Bhowmick and Shyamanta M. Hazarika. 2017. An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends. Journal on Multimodal User Interfaces 11, 2 (Jan. 2017), 149–172. https://doi.org/10.1007/s12193-016-0235-6
  3. Pradipta Biswas, Pat Langdon, Carlos Duarte, and José Coelho. 2011. Multimodal adaptation through simulation for digital TV interface. In Proceedings of the 9th European Conference on Interactive TV and Video. ACM. https://doi.org/10.1145/2000119.2000167
  4. Aditya Bodi, Pooyan Fazli, Shasta Ihorn, Yue-Ting Siu, Andrew T Scott, Lothar Narins, Yash Kant, Abhishek Das, and Ilmi Yoon. 2021. Automated Video Description for Blind and Low Vision Users. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411763.3451810
  5. Jade Cartwright and Kym A. E. Elliott. 2009. Promoting strategic television viewing in the context of progressive language impairment. Aphasiology (2009). https://doi.org/10.1080/02687030801942932
  6. Chris Code, Ilias Papathanasiou, Silvia Rubio-Bruno, María de la Paz Cabana, Maria Marta Villanueva, Line Haaland-Johansen, Tatjana Prizl-Jakovac, Ana Leko, Nada Zemva, Ruth Patterson, Richard Berry, Elizabeth Rochon, Carol Leonard, and Amelie Robert. 2015. International patterns of the public awareness of aphasia. International Journal of Language and Communication Disorders 51, 3 (Nov. 2015), 276–284. https://doi.org/10.1111/1460-6984.12204
  7. Enrique Costa-Montenegro, Fátima M. García-Doval, Jonathan Juncal-Martínez, and Belén Barragáns-Martínez. 2015. SubTitleMe, subtitles in cinemas in mobile devices. Universal Access in the Information Society 15, 3 (June 2015), 461–472. https://doi.org/10.1007/s10209-015-0420-5
  8. Jasmine Cox, Rhianne Jones, Chris Northwood, Jonathan Tutcher, and Ben Robinson. 2017. Object-Based Production. In Adjunct Publication of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video. ACM. https://doi.org/10.1145/3084289.3089912
  9. Humphrey Curtis, Timothy Neate, and Carlota Vazquez Gonzalez. 2022. State of the Art in AAC: A Systematic Review and Taxonomy. In The 24th International ACM SIGACCESS Conference on Computers and Accessibility. ACM. https://doi.org/10.1145/3517428.3544810
  10. Charles Ellis, Clara Dismuke, and Kathryn K. Edwards. 2010. Longitudinal trends in aphasia in the United States. NeuroRehabilitation 27, 4 (Dec. 2010), 327–333. https://doi.org/10.3233/nre-2010-0616
  11. Joseph L. Fleiss and Jacob Cohen. 1973. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement 33, 3 (Oct. 1973), 613–619. https://doi.org/10.1177/001316447303300309
  12. Benjamin M. Gorman, Michael Crabb, and Michael Armstrong. 2021. Adaptive Subtitles: Preferences and Trade-Offs in Real-Time Media Adaption. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411764.3445509
  13. Brian Grellmann, Timothy Neate, Abi Roper, Stephanie Wilson, and Jane Marshall. 2018. Investigating Mobile Accessibility Guidance for People with Aphasia. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’18). Association for Computing Machinery, New York, NY, USA, 410–413. https://doi.org/10.1145/3234695.3241011
  14. Zdenek Hanzlicek, Jindrich Matousek, and Daniel Tihelka. 2008. Towards automatic audio track generation for Czech TV broadcasting: Initial experiments with subtitles-to-speech synthesis. In 2008 9th International Conference on Signal Processing. IEEE. https://doi.org/10.1109/icosp.2008.4697710
  15. Elfed Howells and David Jackson. 2021. Object-Based Media Report. https://www.ofcom.org.uk/research-and-data/technology/general/object-based-media
  16. Claire Kearney-Volpe, Shannon Holloway, and Amy Hurst. 2019. Entertainment for All. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3290607.3312882
  17. Esme Kirk-Wade. 2022. UK disability statistics: Prevalence and life experiences. House of Commons Library (July 2022). https://commonslibrary.parliament.uk/research-briefings/cbp-9602/
  18. Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411764.3445412
  19. Mark McGill, John H. Williamson, and Stephen Brewster. 2016. Examining The Role of Smart TVs and VR HMDs in Synchronous At-a-Distance Media Consumption. ACM Transactions on Computer-Human Interaction 23, 5 (Nov. 2016), 1–57. https://doi.org/10.1145/2983530
  20. Mario Montagud, Jaume Segura-Garcia, J. Antonio De Rus, and Rafael Fayos Jordán. 2020. Towards an Immersive and Accessible Virtual Reconstruction of Theaters from the Early Modern. In ACM International Conference on Interactive Media Experiences. ACM. https://doi.org/10.1145/3391614.3399390
  21. Martez Mott, John Tang, Shaun Kane, Edward Cutrell, and Meredith Ringel Morris. 2020. “I just went into it assuming that I wouldnt be able to have the full experience”. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. ACM. https://doi.org/10.1145/3373625.3416998
  22. Timothy Neate, Aikaterini Bourazeri, Abi Roper, Simone Stumpf, and Stephanie Wilson. 2019. Co-Created Personas. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3290605.3300880
  23. Cristian Pamparău and Radu-Daniel Vatavu. 2022. The User Experience of Journeys in the Realm of Augmented Reality Television. In ACM International Conference on Interactive Media Experiences. ACM. https://doi.org/10.1145/3505284.3529969
  24. Melissa L. Rethlefsen, Shona Kirtley, Siw Waffenschmidt, Ana Patricia Ayala, David Moher, Matthew J. Page, and Jonathan B. Koffel. 2021. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews 10, 1 (Jan. 2021). https://doi.org/10.1186/s13643-020-01542-z
  25. Abi Roper, Ian Davey, Stephanie Wilson, Timothy Neate, Jane Marshall, and Brian Grellmann. 2018. Usability Testing - An Aphasia Perspective. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. ACM. https://doi.org/10.1145/3234695.3241481
  26. Andy P. Siddaway, Alex M. Wood, and Larry V. Hedges. 2019. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annual Review of Psychology 70, 1 (Jan. 2019), 747–770. https://doi.org/10.1146/annurev-psych-010418-102803
  27. Rodrigo L. S. Silva and Frâncila Weidt Neiva. 2016. Systematic Literature Review in Computer Science - A Practical Guide. (2016). https://doi.org/10.13140/RG.2.2.35453.87524
  28. Laurianne Sitbon, Ross Brown, and Lauren Fell. 2019. Turning Heads: Designing Engaging Immersive Video Experiences to Support People with Intellectual Disability when Learning Everyday Living Skills. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. ACM. https://doi.org/10.1145/3308561.3353787
  29. Marian Ursu, Davy Smith, Jonathan Hook, Shauna Concannon, and John Gray. 2020. Authoring Interactive Fictional Stories in Object-Based Media (OBM). In ACM International Conference on Interactive Media Experiences. ACM. https://doi.org/10.1145/3391614.3393654
  30. Radu-Daniel Vatavu. 2021. Accessibility of Interactive Television and Media Experiences: Users with Disabilities Have Been Little Voiced at IMX and TVX. In ACM International Conference on Interactive Media Experiences. ACM. https://doi.org/10.1145/3452918.3465485
  31. Johana María Rosas Villena, Bruno Costa Ramos, Renata Pontin M. Fortes, and Rudinei Goularte. 2014. Web Videos – Concerns About Accessibility based on User Centered Design. Procedia Computer Science 27 (2014), 481–490. https://doi.org/10.1016/j.procs.2014.02.052
  32. Agnieszka Walczak. 2017. Audio description on smartphones: making cinema accessible for visually impaired audiences. Universal Access in the Information Society 17, 4 (Aug. 2017), 833–840. https://doi.org/10.1007/s10209-017-0568-2
  33. Lauren Ward and Ben Shirley. 2019. Personalization in Object-based Audio for Accessibility: A Review of Advancements for Hearing Impaired Listeners. Journal of the Audio Engineering Society 67, 7/8 (Aug. 2019), 584–597. https://doi.org/10.17743/jaes.2019.0021
  34. Lauren Ward, Ben Shirley, and Jon Francombe. 2018. Accessible Object-based Audio Using Hierarchical Narrative Importance Metadata. Journal of the Audio Engineering Society (2018).
  35. Stephanie Wilson, Abi Roper, Jane Marshall, Julia Galliers, Niamh Devane, Tracey Booth, and Celia Woolf. 2015. Codesign for people with aphasia through tangible design languages. CoDesign 11, 1 (Jan. 2015), 21–34. https://doi.org/10.1080/15710882.2014.997744
  36. Claes Wohlin. 2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE ’14). Association for Computing Machinery, New York, NY, USA, Article 38, 10 pages. https://doi.org/10.1145/2601248.2601268