Mapping Movies: A Mind-Map Approach to Aphasia-Friendly Video

Introduction

Audiovisual media is an integral part of people’s lives, and is essential for accessing information and engaging with the wider world [29] across myriad social and civic domains [1, 18, 28]. However, despite its importance, access to audiovisual media remains unequal, with many people with disabilities, including those with complex communication needs (CCNs), facing persistent barriers [24]. Consequently, the research community, together with industry, has developed accessibility supports, such as subtitles and audio descriptions (AD) which are now commonplace. Despite significant progress, many accessibility challenges persist. In particular, accessibility of audiovisual content for individuals with CCNs, often related to language or cognitive impairments, remains limited [25]. The language disability aphasia is one such under-represented condition. People with aphasia experience difficulties with understanding, speaking, reading, and/or writing, with the nature and severity varying between individuals [4, 20]. These communication difficulties are manifested in the consumption of audiovisual media [25], yet little work has been done to explore how technology might be designed to support.

This paper investigates an exploration of how a common information presentation technique – the mind map – can support video comprehension. Mind maps, which are regularly used to aid users with comprehension difficulties, afford the clear presentation of narratives and offer flexibility to support user needs [19, 38]. We present a prototype which uses emerging AI capabilities to automatically create a mind map of any video, with a view of aiding comprehension within an aphasia-friendly interface. Our contribution is twofold:

1. The development of a mind map-based system for converting audiovisual content into interactive summarization
2. Initial formative feedback of VideoMind system through expert reviews with people with aphasia

Related Work

Aphasia and Audiovisual Media Barriers

Individuals with aphasia face several barriers when accessing audiovisual content, including linguistic challenges related to speech speed and complexity, limitations of subtitles, and visual overload caused by rapid or distracting visuals [24]. The vast majority of accessibility interventions are designed to support users with sensory disabilities, with wider populations significantly under-served [25] – including users with aphasia. While subtitles are widely used to support people who are d/Deaf or hard of hearing (DHH) [10], and audio description can enable access for individuals who are blind or low vision (BLV) [32], few approaches are designed for users with complex communication needs. In response to the lack of interventions designed with users with CCNs in mind, recent work has proposed accessibility interventions for people with aphasia, such as dynamic pacing and simplifying audiovisual content content [5, 22, 23].

Mind Maps and Aphasia

Mind maps allow for the organization of information into a simple visual form. In aphasia therapy, mind maps have been used to structure complex language tasks such as storytelling [38]. Moreover, mind mapping for narrative work in the speech and language therapy field has been identified as an evidence-based intervention that contributes to improving real-life communication [19]. In our prior work [5], users with aphasia built on their own personal experience as regular mind mappers to envision future technologies which support access to key topics, places and people in the audio content they were having difficulties with. For instance, one participant used intricate, hand-drawn mind maps to support her understanding of the 2024 UK Conservative party leadership race. Beyond aphasia, mind mapping has also demonstrated benefits for individuals with other CCNs, including autism spectrum disorder and cognitive-communication impairments, by supporting understanding, information retention, and greater communication autonomy [15, 39]. Inspired by this prior work, and by the design ideas seeded by co-designers in our prior aforementioned work [5], we explore dynamic mind map generation as a means to aid the comprehension of video.

The VideoMind Accessible Video Mind Mapping Tool

We present VideoMind – an automatic AI-supported mind map tool, which provides an aphasia-friendly interface to support the comprehension of key concepts in videos. A demonstration is provided as a Video Figure with this paper. The VideoMind tool generates a mind map, organized by topic and time. To enable this, the input video is divided into coherent segments and arranged chronologically. The central node represents the overall topic, while child nodes correspond to video segments, ordered by timestamp, and include sub-nodes with additional details derived from selected features.

VideoMind’s Simplification Methods

VideoMind uses a range of methods to extract and simplify content in a way that might support more aphasia-friendly presentation. Topic modelling is used to improve content understanding by enabling a structured breakdown, with a view to reduce cognitive load, enhance comprehension, recall and sense making for users with CCNs [12]. We also use keyword/key phrase extraction [26], so that we can emphasize main ideas, with similar aims. Such methods are implemented with bullet-wise summarization, which has been shown to aid comprehension for users with CCNs [3, 9]. To extract key people, places and dates, we used Named Entity Recognition – highlighting key grounding information such as these are important for supporting users with aphasia in comprehension [27].

VideoMind’s Presentation Methods, Interface and Technical Implementation

VideoMind combines multiple modalities and generated materials to aid comprehension. It uses text‑to‑speech summaries throughout, drawing on evidence that TTS supports users with aphasia [13] and has been effective in prior aphasia‑focused apps [5, 21]. Emojis are also used as visual representations of words and concepts throughout to support an additional modality; a method which has been shown to be useful for people with aphasia with comprehension challenges [17] and successfully employed in accessible media technologies for people with aphasia [5]. Similarly, to provide visual reference to the content being watched, key-frames are extracted at the specific times to link each mind map node to the relevant point in the video. To provide a textual representation, we also provide subtitles. Finally, the interface was iterated by the whole team based on prior experience of designing accessible technologies for users with aphasia, supported by aphasia-friendly guidance (e.g. [33, 34]), and the literature on aphasia and perception. This included reduced visual clutter [36], a stepped approach to revealing the interface [21] and consistent use of icons and text in tandem [33, 34].

In terms of technical implementation, VideoMind is a full-stack, responsive web-based system with both front-end and back-end components. The front end is built using TypeScript, HTML, and CSS, with React.js used to create an interactive, component-based interface. The back end is implemented in Python and generates mind map content using locally deployed, publicly available AI and NLP models. The Whisper base model [31] has been deployed locally for automatic speech recognition and subtitle generation. The Llama-3.1-8B-Instruct model [11] has been employed locally as a large language model for emoji prediction, text summarization, keyword extraction, and keyphrase extraction. A variant of Llama-3.2-1B has also been used for topic modelling. In addition, CLIP-ViT-B/32 [30], an image–text model, has been used for keyframe extraction based on segment summaries. Ultimately, uploaded videos are processed to generate a JSON-based mind map for visualization (see Figure 1).

VideoMind Interface

VideoMind and its views are shown in Figure 2. It consists of three views: Split View, Video Summary, and Interactive Mind Map. The Split view consists of three components: the Video Player (blue box), the Video Summary (yellow box), and the Interactive Mind Map (green box). The video player displays content and provides standard playback controls, along with adjustable generated subtitles. The next component is the Interactive Mind Map box. The Interactive Mind Map visualizes the video semantically and provides controls for resizing and resetting. It includes a central node representing the overall topic, with connected child nodes corresponding to individual segments of the audiovisual content. Each node is shown as a card displaying the segment’s timestamp, topic, and keywords, and includes Play and Listen buttons to jump to the segment or hear its audio summary. Each child node expands into three sub-nodes: a key frame with a brief summary and highlighted entities, a set of relevant emojis, and a list of keywords or key phrases. The mind map is fully interactive and synchronised to the video playback, enabling dynamic exploration. It also includes a collapsible Video Summary panel that shows keywords, a short summary, and the time frame for the current segment. The Video View, consists of the Video Player (blue box), the Dynamic Video Summary (yellow box), and a Live Transcription box. The Video Player and summary operate in the same way as in the Split View. The key difference is the Live Transcription box, which displays real-time subtitles along with the preceding and following sentences to provide additional context; prioritizing textual and audio information over visual elements such as the mind map. Finally, the Mind Map View displays the mind map on its own at a larger scale, providing users with more space and flexibility to interact with it more easily.

User Study: Recruitment and Results

Method

We recruited two people with aphasia who were experienced in co-designing technologies and giving feedback on research prototypes to a range of research groups for over 7 years. We prioritised depth of observation and critique over breadth, aiming to surface accessibility and interaction breakdowns through an early-stage, formative prototype study. Although both participants provided highly valuable critique grounded in extensive co-design experience, the small sample size means we treat the findings as indicative and plan broader evaluation in future work.

We conducted one-to-one sessions with each person in which we: introduced the technology; gave hands on time for them to explore its features; then allowed for a reflection. Mind maps were generated for three types of content: news, historical, and documentary, to ensure a diverse range of examples to reduce content bias. The mind maps were generated entirely by the system without human authoring. The participants were encouraged to freely interact with the system and engage with the prepared content using the mind map interface. At the end of the session, we administered a short questionnaire to capture feedback and improvement suggestions (see Appendix A) using it as a structured interview prompt to elicit the impressions.

Findings

Thematic analysis of participant feedback identified two themes defining VideoMind’s practical affordances and the interface conditions required for those benefits in people with aphasia.

Selective Navigation for Focused Engagement

The VideoMind operated as a meaning-first navigation layer that let participants triage content, skip fatigue-inducing segments, and return to relevant parts without relying on trial-and-error scrubbing.

Participants used VideoMind to jump directly to valued content rather than scrubbing a timeline or scanning chapters: “jump to the best part of the news, cut the crap out and just listen to what I want” (P1). Both participants appreciated a possibility to preview structure before committing attention, using labelled segments and short summaries/keywords as a quick decision aid “quickly look at the map” (P2). Chunking viewing across time was seen as important as it enabled them to treat segments as manageable units to revisit later: “Everything’s broken down so it is good. I can go back and see that part [now], then tomorrow I can watch the next part” (P1). This was especially valuable for news, where repetition and density were described as “boring” – not as a lack of interest, but as a signal of comprehension drop and overload risk. The mind map enabled content triaging (e.g. watching only new or relevant segments) rather than enduring long stretches that might become confusing or effortful.

Managing Overload through Simplified Interfaces and Controls

While the VideoMind increased control and context comprehension, participants emphasised that simultaneous presentation can negate these benefits; the interface must support on-demand access and reduce competing streams of attention.

Participants reported the split view could be “too much information” (P2), and “hard to focus” (P1), describing scattered attention when video and map were both visible “I’m looking at this, this, this, this” (P2).

Their instantaneous response was to propose design adjustments that preserve control while lowering cognitive load. In the light of imagining how this would appear on large TV screen P1 suggested on-demand overlay/toggle – “like a layer on top of the video when needed and otherwise stay hidden”. For example, both participants agree that pressing a button to briefly call up the map – akin to a “picture-in-picture” overlay would allow them to concentrate on the summary, and then disappear so they could resume watching without distraction. This preference was tied to screen size and readability. P1 having a large TV at home remarked that the side-by-side view shrank the video unnecessarily: “it’s taking up... making it [the video] go to a 32-inch [size on my 55-inch TV]. I don’t want that”. Both participants agreed that auto-focus on the active node would help to reduce manual zooming and scrolling: “As soon as you go to that, it expands” (P2).

Fast ‘lookup’ interaction was deemed important. P1 compared the ideal interaction to using “Shazam” – you trigger it and instantly get the info you need, then “it goes back to watching what you’re watching”, with minimal disruption.

Discussion

Our findings point to design implications for accessible video systems for people with aphasia. The first key implication suggests the main benefit is not simply “more summary”, but restricted control: an overview first way to decide what to watch, in what order, and in what dose.

Participants in our study did not passively consume videos from start to finish; instead, they adopted a strategic, non-linear viewing approach, using the mind map to navigate according to their interests and needs. This behaviour challenges the dominant timeline-first navigation model, where interfaces offer limited structure beyond scrubbing on a linear seek bar i.e., thumbnail previews [2], Scrubbing Wheel [37], RubySlippers for keyword navigation [7], LectureScape’s dynamic timeline[16], and Smart Jump for personalised jump-back [40], to name a few. This aligns with [5], where people with aphasia emphasised temporal navigation aids (segmentation, hierarchical summaries) to support comprehension. We extend this by showing an interactive AI-generated mind map as an operational navigation layer that lets users “skip the junk” with minimal effort, prioritise what matters, and mitigate aphasia-related audiovisual fatigue [6]. Participants used the map not only to locate relevant segments but to decide whether to watch a video at all and to partition viewing into manageable chunks, pointing to an overview-first paradigm: preview narrative structure, then selectively enter segments of interest. Designing for this behaviour suggests mainstream players could incorporate summarisation previews/overlays that restructure navigation around meaning rather than continuous playback, with particular benefit for users with cognitive or language impairments.

The second key insight of our study is the importance of interface adaptability and simplicity: the VideoMind’s value depends on adaptive disclosure and automation that keeps orientation without extra interaction cost.

While the mind map provided valuable context and control, participants made it clear that how this information is presented is critical. In fact, an overly complex or cluttered interface can negate the benefits of summarization by introducing new barriers through multi-modal cues (text, audio, visuals synchronously delivered). This insight extends prior work on aphasia-friendly design [35], which advocates for clean layouts, large print, and minimal distractions in textual materials. We found that these principles are equally crucial in interactive, multimodal interfaces. Our participants struggled when the video and mind map were shown side by side – a design that, while logical for concurrent viewing, proved cognitively taxing. By making the summary interface on-demand (e.g. through a temporary overlay or a second-screen option), designers can let users choose when to engage with detailed information and when to focus on the primary content. Current industry integrations (i.e. Amazon’s X-Ray) layers additional information on demand (actor bios, trivia) over the video – a mainstream proof-of-concept that viewers can access paratextual content when they choose. However, our approach goes further by focusing on cognitive accessibility: the mind map does not just add trivia, it reorganizes core content into a simpler visual-textual form. This is novel compared to existing media accessibility tools which typically do not integrate multimodal summaries directly with playback. By combining transcript keywords, images, and audio snippets, our system embodies the multimodality that prior studies have shown to aid memory and comprehension for aphasic individuals.

Finally, how we choose to represent structured information contained in the mind map is fundamentally determined by metadata design decisions. Prior work has shown that unreliable metadata (i.e. ill-defined or incomplete) can mislead systems and omit relevant context [14], while object-based media (OBM) research demonstrates that metadata models effectively govern what personalised renditions a user can receive [8]. In our setting, these choices shape which concepts are surfaced, how they are grouped and named, and what remains visible thereby directly affecting interpret ability and user’s agency. This makes metadata not just a technical substrate but an ethical boundary condition for AI-mediated accessibility, motivating further research on transparent, contestable, and harm-aware representations for people with complex communication needs.

Conclusion

Our work extends prior knowledge in several ways. First, we demonstrate the feasibility and value of an AI-driven interactive mind map for video – moving beyond the static and linear media solutions. The expert participants’ experiences reveal that people with aphasia can actively use such a tool to shape their media consumption, from skimming news to revisiting specific scenes in entertainment media. This interactive summarization challenges the status quo of accessibility features by showing that supporting understanding is not just about add-ons or linear explanations, but about re-structuring content into more digestible forms. Second, we identify new design opportunities: for instance, the “Shazam-like” quick info lookup suggests a direction for seamless context-aware assistance, where a viewer could speak or click to get an instant explanation of a confusing segment and then continue watching. Finally, we contribute to the broader discourse on cognitive load in media accessibility. Our study provides initial qualitative evidence that chunking information (via a mind map) can reduce overload and increase engagement for viewers with cognitive-language impairments.