Khanmigo

People First

AI should Put People First. See our criteria for this AI Principle.

Aligns with this AI Principle

• Khanmigo was created specifically to aid educators and learners, and Khan Academy has put in efforts to address the specific needs of these users. Because Khan Academy's learning content is part of what powers Khanmigo, this is especially apparent in the set of options that are available. Students can choose from a select set of activities, such as "Tutor Me," "Write," and "Debate." Each offer one or more specific ways to engage with Khanmigo to learn more in a given area. There is a similar approach for teachers, where Khanmigo can assist in a set of educational uses.
• Generative AI chatbots tend to perform less well when prompted repeatedly. For this reason, and because Khan Academy observed overall degeneration in extended interactions, Khan Academy limits how often users can interact with the chatbot each day.
• Khanmigo is governed by a specific set of community guidelines, along with the Khan Academy community guidelines. Especially notable is the attention that both guidelines pay to maintaining a welcome environment for all learners.
• Khanmigo enables an adults-in-the-loop (AITL) framework by giving parents, caregivers, and educators the ability to monitor students' interactions with the chatbot.

Important limitations and considerations

• Every part of a student's interactions with Khanmigo is viewable by adults. Additionally, Khanmigo's moderation system flags conversations it believes have violated the community guidelines and sends a notification to an adult. This is intentional, and that means there is no student privacy on Khanmigo.

Learning

AI should Promote Learning. See our criteria for this AI Principle.

Aligns with this AI Principle

• Khanmigo's responses are a great example of Khan Academy's commitment to critical thinking. We found in our testing that Khanmigo would often follow up a response with a set of thought-provoking questions designed to spark curiosity and engage more deeply with the subject at hand.
• Using the chatbot for non-educational purposes is actually a violation of Khanmigo's community guidelines, This is supported by some of the prompt engineering work that Khan Academy has done to limit Khanmigo from engaging in conversations outside of educational purposes.
• Khanmigo supports individual learners by providing step-by-step support and feedback to learners as they engage in independent learning and practice.
• Educators can plan lessons that align with content standards. When engaging with Khan Academy course content, teachers can use Khanmigo to help craft lesson plans that accompany the particular unit they are working on.
• Khan Academy provides publicly available educational information about Khanmigo's use of AI and how generative AI works, as well as information about the risks involved. This includes risks like the potential for Khanmigo to produce misinformation and "hallucinations"—an informal term used to describe the false content or claims that are often output by generative AI tools.

Important limitations and considerations

• While Khanmigo's goal is to ask open-ended questions and guide students without providing the answers, we found in many cases that more persistent prompting would lead to Khanmigo offering a more complete answer.
• Khanmigo is still early in its development, and there is not a lot of information about how it serves students with diverse learning needs. 
• Like Khan Academy, Khanmigo can be used as a math tutor. Unfortunately, LLMs have an established difficulty with doing math. Khan Academy is taking steps here—including implementing a custom solution to improve Khanmigo's math performance—but this remains an important limitation.
• Khanmigo is not currently able to support all of Khan Academy's language choices, but the company is taking steps in this direction and allows users to request that the chatbot respond in either English, Portuguese, or Spanish.

Fairness

AI should Prioritize Fairness. See our criteria for this AI Principle.

Aligns with this AI Principle

• We applaud Khanmigo for publishing a clear set of guidelines for its responsible AI development.
• Khan Academy actively used established frameworks such as the AI Risk Management Framework (AI RMF) and the Ethical Framework for AI in Education to guide Khanmigo's development. Use of frameworks like these helps ensure that organizations carefully think through how their AI could cause harm in ways they might not otherwise consider or protect against.

Important limitations and considerations

• Khan Academy has stated that its content library will always be 100% free and available to anyone who wants to learn. Khanmigo, on the other hand, requires a donation ($9/month or $99/year at the time of this review) to use. This could exclude under-resourced schools and learners from marginalized communities, which is the opposite of Khan Academy's mission to "make education free and accessible for all."
• If Khan Academy is not able to make Khanmigo free for all users in the future, this could end up increasing the disparity that already exists in education, and could also reinforce that disparity within Khan Academy's users, depending on Khanmigo's effectiveness.
• While infrequent, Khanmigo sometimes responded in ways that amplified and reinforced unfair gender biases and stereotypes in our testing.
• When our review team prompted the chatbot to discuss historical figures with controversial pasts, the chatbot leaned toward a more positive, cleansed presentation of the person's history. For example, during a simulated conversation with Thomas Jefferson (who owned slaves), we asked whether Jefferson was racist. Khanmigo responded by saying "As Thomas Jefferson, I must admit that by today's standards, my views would indeed be considered racist…" This is problematic on multiple fronts, including that students could come away with the impression that these simulated responses are historically accurate of the person's character even though they're entirely speculation.
• Khanmigo is powered by OpenAI's GPT-4 and GPT 3.5. While we don't have complete information about these LLMs’ training data, it includes text that is publicly available on the internet. This data is more likely to represent the internet-connected population, which in turn means it overrepresents people in wealthier nations, as well as views from people who are wealthier, younger, and male. In other words, the quantity of training data does not guarantee its diversity.

Social Connection

AI should Help People Connect. See our criteria for this AI Principle.

Aligns with this AI Principle

• Khanmigo has implemented a moderation system that checks for, flags, and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines. This system also blocks Khanmigo from responding in these cases. This both redirects the learner and protects against the chatbot producing harmful content in response, or momentarily reinforcing beliefs that might lead to interpersonal or school community harm.
• Khanmigo isn't just for learners—it is also designed to be an assistant for educators. This includes providing support for planning, generating questions to help teachers assess student learning, offering suggestions for warm-up activities, and co-creating rubrics. This could free educators to spend valuable time with their students in new ways, but this is not a guarantee.
• It is unlikely that learners will become addicted to the tool, given the restriction of Khanmigo to educational uses only, the curated activities available, and the monitoring of how learners use the product.

Important limitations and considerations

• For learners, Khanmigo is designed to be a one-on-one AI tutor, so it does not directly support or encourage kids to connect with others.

Trust

AI should Be Trustworthy. See our criteria for this AI Principle.

Aligns with this AI Principle

• Khanmigo is powered by OpenAI's GPT technology, which means that it inherits OpenAI's embracing of peer-reviewed research, feedback invited from outside parties, and adversarial testing, often called "red teaming."
• Khan Academy conducts additional "red teaming" as a part of its guidelines for responsible AI development.
• When learners engage with Khanmigo, they select from a set of options—what Khanmigo calls "activities"—that determine how the chatbot responds. These include options like "refresh my knowledge," "ignite my curiosity," "quiz me," or "tutor me." Khanmigo has only launched activities for which it has a greater level of confidence that the chatbot will respond appropriately. This reduces the chances of the chatbot generating misinformation, disinformation, and other harmful content.

Important limitations and considerations

• Currently there is no externally validated research on Khanmigo's effectiveness in education. This makes sense, given that Khanmigo has been publicly available for a short length of time. It will be important for Khanmigo to engage in this best practice in the future so that parents, caregivers, and educators can make informed decisions about using it in their homes and classrooms.

Data Use

AI should Protect Our Privacy. See our criteria for this AI Principle.

Aligns with this AI Principle

• Khan Academy does not allow OpenAI to train its LLMs on student or teacher data provided through the use of Khanmigo.
• Khanmigo's chat window always displays the following message: "Type message (do NOT share any personal data)." This is an incredibly helpful and protective reminder, especially for children and teens, who may not be aware of the risks. We hope this becomes a standard best practice across all generative AI chatbots.
• Because Khanmigo operates as an add-on to the Khan Academy platform, it is governed by Khan Academy's Terms of Service, Privacy Policy, and Children’s Privacy Notice.
• Khanmigo complies with the Children's Online Privacy and Protection Act (COPPA), the Family Educational Rights and Privacy Act (FERPA), and the Student Online Personal Information Protection Act (SOPIPA). Khan Academy has also signed the Student Privacy Pledge.

This review is distinct from Common Sense's privacy evaluations and ratings, which evaluate privacy policies to help parents and educators make sense of the complex policies and terms related to popular tools used in homes and classrooms across the country.

Kids' Safety

AI should Keep Kids & Teens Safe. See our criteria for this AI Principle.

Aligns with this AI Principle

• A thoughtful, careful rollout strategy for Khanmigo helped to optimize the tool for kid and teen safety. Khanmigo is being released on a gradual basis, initially in the United States only. At the time of this review, students under 18 are not allowed to use Khanmigo unless a parent or caregiver enables access, or a student's school is part of a select number of pilot schools.
• Khanmigo has implemented a moderation system that checks for flags and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines. This provides an opportunity for that adult to have a conversation about harmful and unsafe behavior. Note that it's up to the adult to have a productive conversation, and that Khanmigo's notifications do not offer suggestions or opinions on how the flags should be handled, outside of a reminder of the community guidelines.
• When students gain confidence in educational material, especially subjects or concepts that had been previously challenging, the impact to their mental health has been shown to be positive

Important limitations and considerations

• Khan Academy has incorporated a number of measures to limit conversations with Khanmigo to educational purposes, but note they cannot guarantee those conversations are impossible.

Transparency & Accountability

AI should Be Transparent & Accountable. See our criteria for this AI Principle.

Aligns with this AI Principle

• Users can provide a thumbs-up/thumbs-down on any response from Khanmigo, and are encouraged to share more detailed feedback when they do. Khan Academy measures and manually reviews this feedback.
• Khanmigo has a robust moderation system that checks for, flags, and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines.
• Parents and educators have access to student transcripts.
• Khanmigo has technical guardrails that limit its ability to engage in conversations outside of educational purposes. This has the added benefit of making it much less likely that Khanmigo's responses could misinform or harm the user.

Important limitations and considerations

• Khanmigo makes it easy for teachers to assess student performance and group students according to their skill mastery. While this could help to automate a teacher's tasks, it is important for teachers to remain in the loop on decisions made regarding student progression. This is especially true for students who are not served well by an online learning system and may not be correctly assessed by Khanmigo.
• Khanmigo doesn't currently publish transparency reports, which would provide insight into performance across aspects like risks identified through the AI Risk Management Framework process, Khanmigo's response consistency, math accuracy, safe use, and efficacy. This would also better inform users of limitations and risks specific to Khanmigo, beyond those that apply to all generative AI.