Overview / Usage

RE-AIS refers to humans' semantic knowledge of domain concepts to establish a benchmark for assessing the quality of AIS randomly-collected training datasets. Within this framework, a process is implemented to extract common social specifications of primary concepts in a domain, which in turn, derives useful knowledge to be passed to AIS, such that domain specifications are well-met within the intelligent model. RE-AIS automatically creates a set of domain-specific benchmarks, which are further referred to for the purpose of relative evaluation of the correctness and completeness of AIS visual perception. In consequence, AIS data-driven perception, gained through AIS inductive nature, is enhanced with knowledge-based domain analysis. The improvement occurs through the incorporation of inferred knowledge into the AIS training process, exploiting a semantic layout to compensate for the dataset missing variation of a targeted domain concept, and later, to repair AIS misconception of the concept.

Methodology / Approach

For a fair evaluation of a software product or process, a reference point allows to relatively qualify the comparison, yet, none of the said approaches provided an oracle. Without a reliable point of reference for assessment purposes, the proposal of dataset quality metrics sounds rather abstract, since even assuming that the expected metrics are satisfied (e.g., the dataset is complete), no one could confirm. We aim to use visual representations and techniques to effectively communicate, analyze, and comprehend the requirements and expectations for AI-enabled software. The goal is to connect all components through interactive visualization methods, to ensure the requirements are clear, thorough, properly defined, and aligned with the desired outcomes and constraints of the AI system.

Technologies Used

Our objective in this project phase is to arrange the information gathered from encyclopedias in a manner that allows us to compare the knowledge derived from this source with that of the image dataset. To achieve this, we have implemented topic modeling to the gathered data, with the aim of uncovering relationships, patterns, and trends within the specifications. By creating a hierarchical structure, we can more accurately represent the connections between the specifications, which will help to clarify any ambiguity and refine the specifications, particularly for complex or difficult-to-specify concepts. To derive a hierarchy of relationships, we utilized a pre-trained transformer-based neural network, BERTopic, to extract higher-level topics and their relationships among related specifications.

Repository

https://observablehq.com/d/98faa4a83915aedb