MIT researchers developed and studied a customized AI training program for users with varied backgrounds, which could be delivered across large organizations.
Adam Zewe | MIT News Office
A new academic program developed at MIT aims to teach U.S. Air and Space Forces personnel to understand and utilize artificial intelligence technologies. In a recent peer-reviewed study, the program researchers found that this approach was effective and well-received by employees with diverse backgrounds and professional roles.
The project, which was funded by the Department of the Air Force–MIT Artificial Intelligence Accelerator, seeks to contribute to AI educational research, specifically regarding ways to maximize learning outcomes at scale for people from a variety of educational backgrounds.
Experts in MIT Open Learning built a curriculum for three general types of military personnel — leaders, developers, and users — utilizing existing MIT educational materials and resources. They also created new, more experimental courses that were targeted at Air and Space Forces leaders.
Then, MIT scientists led a research study to analyze the content, evaluate the experiences and outcomes of individual learners during the 18-month pilot, and propose innovations and insights that would enable the program to eventually scale up.
They used interviews and several questionnaires, offered to both program learners and staff, to evaluate how 230 Air and Space Forces personnel interacted with the course material. They also collaborated with MIT faculty to conduct a content gap analysis and identify how the curriculum could be further improved to address the desired skills, knowledge, and mindsets.
Ultimately, the researchers found that the military personnel responded positively to hands-on learning; appreciated asynchronous, time-efficient learning experiences to fit in their busy schedules; and strongly valued a team-based, learning-through-making experience but sought content that included more professional and soft skills. Learners also wanted to see how AI directly applied to their day-to-day work and the broader mission of the Air and Space Forces. They were also interested in more opportunities to engage with others, including their peers, instructors, and AI experts.
Based on these findings, which the program researchers recently shared at the IEEE Frontiers in Education Conference, the team is augmenting the educational content and adding new technical features to the portal for the next iteration of the study, which is currently underway and will extend through 2023.
“We are digging deeper into expanding what we think the opportunities for learning are, that are driven by our research questions but also from understanding the science of learning about this kind of scale and complexity of a project. But ultimately we are also trying to deliver some real translational value to the Air Force and the Department of Defense. This work is leading to a real-world impact for them, and that is really exciting,” says principal investigator Cynthia Breazeal, who is MIT’s dean for digital learning, director of MIT RAISE (Responsible AI for Social Empowerment and Education), and head of the Media Lab’s Personal Robots research group.
Building learning journeys
At the outset of the project, the Air Force gave the program team a set of profiles that captured educational backgrounds and job functions of six basic categories of Air Force personnel. The team then created three archetypes it used to build “learning journeys” — a series of training programs designed to impart a set of AI skills for each profile.
The Lead-Drive archetype is an individual who is making strategic decisions; the Create-Embed archetype is a technical worker who is implementing AI solutions; and the Facilitate-Employ archetype is an end-user of AI-augmented tools.
It was a priority to convince the Lead-Drive archetype of the importance of this program, says lead author Andrés Felipe Salazar-Gomez, a research scientist at MIT Open Learning.
“Even inside the Department of Defense, leaders were questioning if training in AI is worth it or not,” he explains. “We first needed to change the mindset of the leaders so they would allow the other learners, developers, and users to go through this training. At the end of the pilot we found they embraced this training. They had a different mindset.”
The three learning journeys, which ranged from six to 12 months, included a combination of existing AI courses and materials from MIT Horizon, MIT Lincoln Laboratory, MIT Sloan School of Management, the Computer Science and Artificial Intelligence Laboratory (CSAIL), the Media Lab, and MITx MicroMasters programs. Most educational modules were offered entirely online, either synchronously or asynchronously.
Each learning journey included different content and formats based on the needs of users. For instance, the Create-Embed journey included a five-day, in-person, hands-on course taught by a Lincoln Laboratory research scientist that offered a deep dive into technical AI material, while the Facilitate-Employ journey comprised self-paced, asynchronous learning experiences, primarily drawing on MIT Horizon materials that are designed for a more general audience.
The researchers also created two new courses for the Lead-Drive cohort. One, a synchronous online course called The Future of Leadership: Human and AI Collaboration in the Workforce, developed in collaboration with Esme Learning, was based on the leaders’ desire for more training around ethics and human-centered AI design and more content on human-AI collaboration in the workforce. The researchers also crafted an experimental, three-day, in-person course called Learning Machines: Computation, Ethics, and Policy that immersed leaders in a constructionist-style learning experience where teams worked together on a series of hands-on activities with autonomous robots that culminated in an escape-room style capstone competition that brought everything together.
The Learning Machines course was wildly successful, Breazeal says.
“At MIT, we learn by making and through teamwork. We thought, what if we let executives learn about AI this way?” she explains. “We found that the engagement is much deeper, and they gained stronger intuitions about what makes these technologies work and what it takes to implement them responsibly and robustly. I think this is going to deeply inform how we think about executive education for these kinds of disruptive technologies in the future.”
Gathering feedback, enhancing content
Throughout the study, the MIT researchers checked in with the learners using questionnaires to obtain their feedback on the content, pedagogies, and technologies used. They also had MIT faculty analyze each learning journey to identify educational gaps.
Overall, the researchers found that the learners wanted more opportunities to engage, either with their peers through team-based activities or with faculty and experts through synchronous components of online courses. And while most personnel found the content to be interesting, they wanted to see more examples that were directly applicable to their day-to-day work.
Now in the second iteration of the study, researchers are using that feedback to enhance the learning journeys. They are designing knowledge checks that will be a part of the self-paced, asynchronous courses to help learners engage with the content. They are also adding new tools to support live Q&A events with AI experts and help build more community among learners.
The team is also looking to add specific Department of Defense examples throughout the educational modules, and include a scenario-based workshop.
“How do you upskill a workforce of 680,000 across diverse work roles, all echelons, and at scale? This is an MIT-sized problem, and we are tapping into the world-class work that MIT Open Learning has been doing since 2013 — democratizing education on a global scale,” says Maj. John Radovan, deputy director of the DAF-MIT AI Accelerator. “By leveraging our research partnership with MIT, we are able to research the optimal pedagogy of our workforce through focused pilots. We are then able to quickly double down on unexpected positive results and pivot on lessons learned. This is how you accelerate positive change for our airmen and guardians.”
As the study progresses, the program team is sharpening their focus on how they can enable this training program to reach a larger scale.
“The U.S. Department of Defense is the largest employer in the world. When it comes to AI, it is really important that their employees are all speaking the same language,” says Kathleen Kennedy, senior director of MIT Horizon and executive director of the MIT Center for Collective Intelligence. “But the challenge now is scaling this so that learners who are individual people get what they need and stay engaged. And this will certainly help inform how different MIT platforms can be used with other types of large groups.”
