ConAn: Measuring and Evaluating User Confidence in Visual Data Analysis Under Uncertainty

Information

• Publication Type: Journal Paper (without talk)
• Workgroup(s)/Project(s): not specified
• Date: 2024
• DOI: 10.1111/cgf.15272
• ISSN: 1467-8659
• Journal: Computer Graphics Forum
• Pages: 18
• Publisher: WILEY
• Keywords: decision making, uncertainty, user confidence, visual analytics, guided visual data analysis

Abstract

User confidence plays an important role in guided visual data analysis scenarios, especially when uncertainty is involved in the analytical process. However, measuring confidence in practical scenarios remains an open challenge, as previous work relies primarily on self-reporting methods. In this work, we propose a quantitative approach to measure user confidence—as opposed to trust—in an analytical scenario. We do so by exploiting the respective user interaction provenance graph and examining the impact of guidance using a set of network metrics. We assess the usefulness of our proposed metrics through a user study that correlates results obtained from self-reported confidence assessments and our metrics—both with and without guidance. The results suggest that our metrics improve the evaluation of user confidence compared to available approaches. In particular, we found a correlation between self-reported confidence and some of the proposed provenance network metrics. The quantitative results, though, do not show a statistically significant impact of the guidance on user confidence. An additional descriptive analysis suggests that guidance could impact users' confidence and that the qualitative analysis of the provenance network topology can provide a comprehensive view of changes in user confidence. Our results indicate that our proposed metrics and the provenance network graph representation support the evaluation of user confidence and, subsequently, the effective development of guidance in VA.

Additional Files and Images

Weblinks

• Entry in reposiTUm (TU Wien Publication Database)
• DOI: 10.1111/cgf.15272

BibTeX

@article{musleh-2024-conan,
  title =      "ConAn: Measuring and Evaluating User Confidence in Visual
               Data Analysis Under Uncertainty",
  author =     "Maath Musleh and Davide Ceneda and Henry Ehlers and Renata
               Raidou",
  year =       "2024",
  abstract =   "User confidence plays an important role in guided visual
               data analysis scenarios, especially when uncertainty is
               involved in the analytical process. However, measuring
               confidence in practical scenarios remains an open challenge,
               as previous work relies primarily on self-reporting methods.
               In this work, we propose a quantitative approach to measure
               user confidence—as opposed to trust—in an analytical
               scenario. We do so by exploiting the respective user
               interaction provenance graph and examining the impact of
               guidance using a set of network metrics. We assess the
               usefulness of our proposed metrics through a user study that
               correlates results obtained from self-reported confidence
               assessments and our metrics—both with and without
               guidance. The results suggest that our metrics improve the
               evaluation of user confidence compared to available
               approaches. In particular, we found a correlation between
               self-reported confidence and some of the proposed provenance
               network metrics. The quantitative results, though, do not
               show a statistically significant impact of the guidance on
               user confidence. An additional descriptive analysis suggests
               that guidance could impact users' confidence and that the
               qualitative analysis of the provenance network topology can
               provide a comprehensive view of changes in user confidence.
               Our results indicate that our proposed metrics and the
               provenance network graph representation support the
               evaluation of user confidence and, subsequently, the
               effective development of guidance in VA.",
  doi =        "10.1111/cgf.15272",
  issn =       "1467-8659",
  journal =    "Computer Graphics Forum",
  pages =      "18",
  publisher =  "WILEY",
  keywords =   "decision making, uncertainty, user confidence, visual
               analytics, guided visual data analysis",
  URL =        "https://www.cg.tuwien.ac.at/research/publications/2024/musleh-2024-conan/",
}