TMK Modeling Progress

Re-iterating on TMK v3 on Blocks World from peer feedback, I added the initial validation in the beginning of problem solving to check whether any given block arrangement already matches the goal block arrangement and then followed with validating if the given block arrangement is legal. Adding a ‘while loop condition-like” statement feedback came from Erin, so I added these to the FSM transition.

      "transitions": {
        "begin problem solving": {
          "type": "validation",
          "source subtask to target subtask": [["Solve using means ends analysis", "Validate Block Arrangement"]],
          "validation condition": "None of the given block arrangement completely match the goal block arrangement."
        },
        "given arrangement matches the goal": {
            "type": "validation",
            "source subtask to target subtask": [["Solve using means ends analysis", "Success"]],
            "validation condition": "Else"
        },
        "validate block arrangement": {
          "type": "validation",
          "source subtask to target subtask": [["Validate Block Arrangement", "Individual Position Extraction"]],
          "validation condition": ["Each block must occupy a unique position.", 
          "If a block is on the table, it can have at most one block directly on top of it.", 
          "If a block is on top of another block, it should not have more than one block directly above it.",
          "A block can only be moved if it is free, no other blocks are stacked on it."
          ]
        },

Other things done are changing “datacondition” to simply “condition”, changing some nuianced terminologies like “select answer” to “select move” and defining legalities in “validate block arrangement”. The inputs and outputs within the method model were made less granular since those were explicit in the task model in order to follow the given condition and the post condition of the sub-task within the transitions.

• Task Model v4 json file
• Method Model v4 json file
• Knowledge Model v4 json file

I tried to do sanity check with TMK v4 model with both GPT-4o and Gemini 1.5. In the first test with Chat GPT (GPT-4o) using prompt-chaining, the initial prompt yielded a very detailed response, but subsequent answers were less detailed. In the second test, Chat GPT responded with detailed and correct answers without needing prompt-chaining. Gemini (Gemini1.5), when using prompt-chaining, used unfamiliar terminology and showed uncertainty, suggesting it relied more on its training data than the provided json file. This was further evident in the second prompt, where Gemini expressed uncertainty and did not utilize the TMK files effectively.

LLM Testing Results for TMK v4

Next, I reviewed the WIP Standardization document and tried to modify the Method model again order to make use of ‘makes’ and ‘requires’. Also from this document, I created the text file that describes the TMK similar to Figure 3 in the TMK Standardization file – with v1 before and v2 after the first peer feedback. The comment blocks on v2 will be the point of modification for v3 after the second peer feedback.

• Block World TMK v1 txt file
• Block World TMK v2 txt file

Later in the week, I was able to get some comments and reviews on v4 of the Method model. Another restructuring has occured to make the flow better.

The top level transition will be: Compare Current to Goal -> Generate Possible Block Moves -> Select Next Block Move -> Update Current Arrangement.

Top Level FSM:

The next level within Generate Possible Block moves will be Generate Move List -> Apply and Validate Move -> Count Delta Difference. In the first iteration we have possible block moves given, but in the next iterations it will generate the next possible move, validate the legality of the move and count the delta difference for that move.

Last week, I started translating Block World methods into applicable terms for re-use in Ravens Progressive Matrices (RPM) problem. While that was a nice mental exercise, I got the clarification from our post doc/project lead that I should look into Semantic Networks skill for Guards and Prisoners (GPP) problem next and explore to what extent can a TMK model capturing the skill of semantic network for GPP can be applied to RPM. I plan to pick this up once I wrap up on the BW Means End Analysis TMK modeling.

Literature Review on Interactive Task Learning

Two weeks ago, Sashank has presented the paper, “VAL: Interactive Task Learning with GPT Dialog Parsing” by Lawley and MacLellan (acccess here). This paper discussed enabling AI systems to learn tasks through iterative natural language interactions. We reviewed this paper since it implements HTN as its knowledge representation, which is also what we are doing with DILAB’s IVY Agent. I wanted to go in detail now with the methodology and evaluation.

From the review, I’ve learned that the GPT subroutines are not involved in the overall task learning algorithm and when users want to audit the knowledge base. The GPT subroutines are mostly used in translating user input and symbolic knowledge structures. VAL was intended for LLMs to capitalize on the large language fluency on extracting symbolic information, rather than using it to directly learn the task itself. The authors noted that this can be considered as a hybrid neuro-symbolic systems.

In the evaluation, the researchers implemented a user study to evaluate the user experience and the objective performance of VAL’s components. I found it interesting how the opinions on sessions for each user dictated the success rate of the GPT subroutine, which is pretty much central to any HCI research paper. Other details on this section points to VAL’s usability – we aim to do the same thing for IVY. In the discussion section of the paper, the section on Theory of the Mind really piqued my interest. It emphasized the importance of knowing the intent of AI agents for humans to trust it as much as how the AI agent needs to know what task the human user intends for it to learn.

Other Reading

Earlier in the week, I finished reading Animal Farm by George Orwell. There is so much to unpack here because the main themes of the book are relevant to our current sociopolitical landscape. Other than the last and most provocative modifications of the Animal Farm constitution (“All animals are equal, but some animals are more equal than others”), the one that stuck with me most is this quote:

“Several of them would have protested if they could have found the right arguments.”

To me, this seems like a bad headspace to be in. You know something fundamentally wrong has taken place but you can’t seem to state why it is so. Being able to express what is wrong and the capacity to make logical reasoning from point A to point B should be straightforward. This just means that one needs to make logical leaps to understand why something is flawed and that somewhere in the logical flow of thoughts and rationalization has gone awry. Was some deduction accepted so easily that went uninspected and neglected? When this happens to me, I try to figure out why I get stuck and work backwards and investigate the logical flow of my arguments. Zooming out, this book really emphasized how one can become complacent with the current power structure and the overall inability to obtain and retain knowledge as opressed members of society. Literacy is a powerful tool that can be used for the good or for bringing out the worst of their kind.

Again, I’ve been trotting around Northern Spain and I’ve come from Galicia (A coruna and Lugo) to Leon this week. Because of the long bus rides, I have completed Chapter 7 and 8 in How to Read a Book: The Classic Guide to Intelligent Reading. Chapter 7 emhasized the “bones” and Chapter 8 “flesh”, as the author has been comparing writing to covering up the skeleton with flesh and reading to uncovering the skeleton from seeing through the flesh. An analytic reader should be able to state the unity of the book, know the major parts and how it is organized and ordered to form the unity of the whole. This corresponds to the skeleton or structure of the book. Chapter 8 goes through the process of examining the flesh as it relates to the bones. It was a great revelation for me that the most important words are usually the ones that give me a hard time. This goes hand in hand with the an important rule for analytic reading: Find the important words, and through them come to terms with the author. The author explains what it means to “come to terms” with an author, it is to properly understand the context and the meaning of a specific word through how it is specially used. Because the word is not being used ordinarily, the word becomes technical and sometimes may be hard for the reader to really understand. From here on, I will try to apply the things I have learned from this book with my literature reviews.