Fraunhofer USA Center Mid-Atlantic CMA
Team communication and coordination is of critical importance for intelligence gathering by the military and government security agencies, with weaknesses in gathering and processing information often associated with shift handovers, resulting in team cognition challenges. These challenges include inaccuracy blindness, group sharing and storing of knowledge, known as transactive memory systems, and shared mental models. Artificial intelligence (AI) has often been proposed as a possible solution to these problems, since, for example, AI can support teaming by augmenting individuals’ production capabilities, summarize machine read documents and convert them to summary output text, and organize intelligence analysis around entities, such as people and places, rather than freeform text. However, it is not clear how to best align rapidly developing AI technologies with intelligence analysis work. Engineers at Fraunhofer USA CMA have worked on a project with colleagues at the University of Maryland and Duquesne University for the United States Army Research Office to assess the application of AI and machine language analysis to mitigate team communication and coordination problems such as information overload, ignoring potentially relevant data and erosion of trust between team members. The goal of the project was to provide much needed insight into how human teams can work together with AI, especially AI that provides sensemaking support, to improve outcomes in intelligence analysis and avoid exacerbating team interactions. Based on insights gathered from interviews with intelligence analysts, the team developed a software platform and an experimental infrastructure testbed to experimentally study the role of different types of AI during intelligence analyst shift handovers. They also conducted controlled immersive behavioral experiments to test the effect of AI manipulations on sensemaking, problem solving, workload, and transactive memory systems. The testbed consisted of task-relevant input materials, such as mission descriptions and source documents, simulated team members, activity recording tools, such as search tools and scratchpads, experimental monitoring capabilities, such as recording and survey systems, and AI support tools for human analysts, such as AI that can summarize large quantities of information by, for example, constructing topic models. The experiments simulated the 5Vs challenges associated with big data: a high volume of material, a wide variety of material sources, a rapid velocity of information accrual, questionable veracity of some sources, and extractable value being dependent on linking information from multiple sources. The testbed was most recently applied to analyze interactive shifthandovers, comparing relatively simple AI tools with an entity-based AI drawing on developments with ChatGPT and theories on information science and intelligence analysis. The approach shows great promise for assessing AI tools being applied with the goal of improving the efficacy of intelligence analysis.