TMK Modeling Progress

In the latest update, the state transition diagram has been further refined. Specifically, the sub-level FSM for Generating New Move has been broken down in more detail. This sub-level now includes steps for:

• Checking the top block
• Removing the top block if necessary
• Placing the top block in a new position
• Validating the move immediately after placement

This breakdown enhances the granularity of the FSM, ensuring each action is clearly defined and can be individually validated for correctness. This can be illustrated in the state diagram below:

Implementing the configuration for block arrangements has been organized more systematically. Here’s a detailed look at the conceptual and instance definitions:

Concept(position, {Property(block: block), Property(on: block|table)});

Concept(block_arrangement, {Property(name: string), Property(configuration: [position]), Property(delta_name, delta: string, integer});
Instance(block_arrangement, ‘GoalArrangement’, 
     configuration: [
         {block: BlockA, on: BlockB}, 
         {block: BlockB, on: BlockC}, 
         {block: BlockC, on: BlockD}, 
         {block: BlockD, on: Table1}
     ],
     {delta_goal, 0}
   );

This structured approach defines each block’s position and its relationships, facilitating easier manipulation and validation of different block configurations.

Previously, the block arrangement configuration was described as follows:

block arrangement description:
Concept(block_arrangement, block_arrangement_name, [block instances], table_instance, [block_table_triples]);

Relations and Triples:
on description:
Relation(on, block, block);
Relation(on, block, table);

goal state description:
Instance(block_arrangement, Goal, [BlockA, BlockB, BlockC, BlockD], Table1, 
	[on(BlockA, BlockB), on(BlockB, BlockC), on(BlockC, BlockD), on(BlockD, Table)])

Additionally, operations for generating the move list have been defined to streamline the problem-solving process. These operations ensure that each action taken is logical and valid within the Blocks World context:

Operation(Check_Top_Block(target_block: block), 
    true, 
    target_block is free or has blocks on top);

Operation(Remove_Top_Block(target_block: block), 
    target_block has blocks on top, 
    top block removed and placed on the table);

Operation(Place_Block_on_Another_Block_or_Table(target_block: block, target_position: position), 
    target_block is free, 
    target_block placed on another block or table);

These operations ensure that blocks are only moved when they are free and that each move results in a valid block configuration.

Overall, these refinements and structured definitions are significant steps forward in the TMK modeling process, providing a clearer and more robust framework for solving the Blocks World problem.

DILab Cross Project Meeting

The main goal of the meeting was to know (atleast for a newbie like me) and reinforce our understanding across teams on how explanations are being generated using our own internal MCM repo. Specifically, Rahul explains how a user’s question is classified into one of four categories using a GPT API call. We discussed challenges in question classification and explainability in AI, particularly in the context of generative and cognitive AI. With this, it was emphasized how our project framework for with the integration of generative AI and cognitive AI, needs to be encompassed through the lens of theory of mind.

Across different projects in DI Lab, there were inconsistencies in TMK models across three teams, with missing values and inconsistent formatting. Additionally, metrics for evaluation were not uniform – SAMI uses correctness, completeness, and confidence, VERA uses recall, precision and accuracy while IVY uses recall, correctness, completeness and precision. Spencer shares a link to a Wikipedia article on GQM (goals, quality, and metrics) to help frame the discussion on metrics for AI explanations. I was able to quickly find a paper that discusses its approach in UMD’s public web domain and quickly review it for my own understanding.

DILab IVY Meeting

The IVY Team discussed tasks for a pilot study to integrate IVY into EdStem, including creating an IRB protocol, merging TMK models, and evaluating explanations produced. We also exchanged ideas on improving Ivy’s performance in Ed Stem, tracking user questions and answers, fine-tuning the Open AI GPT model, and prioritizing tasks for the pilot study. Additionally, they discussed the incremental progress of a pilot study on automating TMK model creation with natural language processing, explored various approaches to TMK modeling, and discussed automation TMK models.

Literature Review: The Goal Question Metric Approach

From the paper titled “The Goal Question Metric Approach by “ Basili, Caldiera and Rombach. GQM asserts that projects must specify the goals for itself, trace its goals to the data intended to define the goals, then provide a framework for interpreting the data with respect to the goals. Using the GQM should result to a measurement system that targets a set of issues and rules for interpretation. G stands for Goal, Q for Question and M for Metric. G is at the conceptual level where the goal is defined for an object of measurement such as products, processes and resources. Q is at the operational level where a set of questions is used to characterize the assessment of a specific goal, Finally M is at the quantitative level where a set of data is associated with every question to answer it in a quantitative way. To me, it sounds like GQM is a hierarchical structure that defines Goals with questions relevant to the goal and metrics which answers each questions quantitatively. A successful implementation of GQM model and has methodological steps that a project can follow. The main takeaway for me is that measurement must be defined in a top-down fashion and that it must be focused, based on goals and models.