TMK Evaluation Metric

In the world of online education, getting human-centric feedback is key to making sure these virtual agents really enhance the learning experience.

From an interesting study by Qiaosi Wang et al., from CHI 2020, they introduced a virtual agent called Jill Watson (same DILab), designed to help students connect by matching them based on shared interests like location and hobbies. They gathered feedback through a few short surveys to see how well Jill Watson SA was doing and how it could be improved. This study highlighted how important it is to consider user feedback and community building when developing educational AI tools.

When it comes to evaluating Ivy’s text-based responses, we’ve followed a similar human-centric approach. Here’s a quick rundown of the metrics we’re using:

• Correctness: Does the response provide accurate and relevant information?
• Completeness: Is the response thorough and does it cover all aspects of the question?
• Confidence: How confident does the model appear in its response?
• Comprehensibility: Is the response clear and easy to understand?
• Compactness: Is the response concise and to the point?

These metrics help ensure that Ivy not only gives accurate and complete answers but also communicates them in a way that’s clear and confident. Plus, they help me refine the model to make sure it’s hitting all the right notes for users.

TMK Evaluation

I formulated 12 verification questions to thoroughly assess Ivy (List_of_questions_for_MCM_eval txt). To make sure we’re getting a well-rounded view, I had three different reviewers involved:

• A Data Scientist: They crafted three questions from a student’s perspective to ensure we’re considering how Ivy might be perceived in a real-world educational setting.
• An Ivy Developer: Someone not directly involved in TMK development brought an unbiased perspective, focusing on the technical aspects of Ivy’s responses.
• The TMK Modeler (That’s Me!): I provided my own ratings and insights to ensure we’re looking at Ivy’s performance from all angles.
• How We Do It: The 5-Point Scale

For the evaluation, I used a 5-point scale to get nuanced opinions on Ivy’s responses. Here’s a quick rundown of what that looks like:

• 1: Strongly Disagree
• 2: Disagree
• 3: Neutral
• 4: Agree
• 5: Strongly Agree

This scale helps capture a range of feedback, so we don’t just get yes or no answers, but rather a spectrum of opinions on how well Ivy performs. We considered whether mean (average) or modal (most common) values were more suitable for analyzing the feedback. By incorporating perspectives from different stakeholders—like the TMK modeler, Ivy developer, and students—I aimed to get a comprehensive understanding of Ivy’s performance.

The evaluation involved 15 question-answer pairs, with scores for each metric averaged to derive an overall score. This approach ensures that every aspect of Ivy’s responses is considered, from accuracy to clarity.

Evaluation Data Analysis

To get a clear picture of how my TMK is performing, I used a few different types of visualizations:

• 1. Bar Charts: I created bar charts to visualize the average scores for different question types. This was super helpful in seeing which categories were doing well and which ones needed improvement. For example, the “Knowledge” and “Task” question types scored the highest, while “Method” and “Student Questions” lagged behind. You can see this in Figure 2 in the paper, where the varying performance levels across these question types are clearly laid out.
• 1. Box Plots: I used box plots to dive deeper into the distribution of scores. This type of chart helps us understand the range and variation in scores for each question type. Figure 3 in the paper shows that while the “Knowledge” category generally performed well, other areas like “Task” had a wider spread in scores. This means that while some responses were great, others didn’t quite hit the mark. Box plots also highlighted that the “Method” and “Student Questions” categories consistently scored lower, which suggests room for improvement.
• 1. Heatmaps: To explore the relationships between different metrics, I used a heatmap. This visualizes how various scoring categories, like confidence and correctness, are related to each other. Figure 4 in the paper reveals some interesting patterns. For example, “Correctness” and “Completeness” are strongly correlated, meaning that answers that are correct are also usually complete. On the other hand, “Confidence” and “Compactness” showed moderate correlations, indicating that while more confident answers are often more compact, this isn’t always the case.

Incorporating Results to Paper

I summarized the main findings from the bar charts, box plots, and heatmaps. For instance, I pointed out that Ivy excels in “Knowledge” and “Task” questions but struggles with “Method” and “Student Questions.” This provided a clear picture of where Ivy performs well and where it needs more work. Then, the box plots revealed significant variability in scores, especially for the “Task” category. I discussed how this variability indicates inconsistent performance and suggested areas where targeted improvements could enhance Ivy’s overall effectiveness. Then finally, the heatmap was crucial for understanding the relationships between different scoring metrics. I highlighted how some metrics, like “Correctness” and “Completeness,” are closely related, while others, like “Confidence,” have weaker correlations with the rest. This insight helped refine the scoring system and suggested potential simplifications.

Finishing the Final Paper and Website

The abstract is like the movie trailer for my paper—short and sweet, but packed with essential information. It needed to capture the essence of my research in a few concise paragraphs. I started by outlining the core objectives: enhancing autonomous coaching agents in online education by integrating Generative AI with learning theory. I highlighted the problem of current evaluation methods and how my Task-Model-Knowledge (TMK) model aims to bridge this gap. I wrapped it up by mentioning the evaluation framework and the key findings, including the model’s strengths and areas for improvement. The goal was to provide a snapshot that grabs attention and invites readers to dive deeper. Meanwhile, the introduction sets the stage for the entire paper. I began by discussing the importance of effective feedback in online education and the limitations of traditional methods. To wrap it up, I outlined the significance of this research and the expected impact on educational technology. The big part that took a lot of time: In the results section, I presented the findings from my evaluation of Ivy using various metrics and question types. I included visualizations like bar charts, box plots, and heatmaps to illustrate the performance across different categories. Each visualization was paired with a clear explanation to make the data understandable. In the discussion, I reflected on the broader impact of these findings on educational technology and outlined potential future directions for research.

With the paper drafted, it was time to update the research blog to reflect the latest findings and developments. Updating the blog was about ensuring that all the information was current. I aimed to make the blog a comprehensive resource that complements the paper and offers additional context and insights.

Next Steps and Future Work

I’m thrilled to share final updates and future directions for the TMK model and the IVY project. I’ve learned a lot from my recent analyses, and pinpointed some key areas that will really take our model to the next level.

First, I’ve noticed that the model can struggle with complex and open-ended questions. This is definitely something I’m focusing on improving. My goal is to make the model more robust and adaptable when faced with diverse and tricky questions. I might need to tweak the model’s architecture or bring in some advanced techniques to handle these questions better. It’s all about making sure the model can tackle any challenge that comes its way. Next, I found that the metrics for Correctness and Completeness are quite closely linked. This overlap means that the current metrics might not be giving me the full picture. So, I’m planning to suggest to the IVY team to develop new, more distinct metrics that will provide unique insights into different aspects of the model’s performance. This way, we’ll be able to evaluate the model more comprehensively and get a clearer understanding of how it’s doing.

On a bigger scale, the IVY project is making great progress. I’m on track to finish system testing by mid-October, which is super exciting. After that, I’m gearing up to deploy IVY in the Knowledge Based AI (KBAI) OMSCS class at Georgia Tech in Spring 2025. This will be a fantastic opportunity to get real-world feedback and see how the model performs in an actual academic setting.