TMK Modeling Progress

I initiated setting up the date and time for my first peer review of my Means End Analysis TMK. Following the standardization document, there needs to be a TMK text file that will be the basis of the TMK json files as well as a UML Class diagram to supplement the text file.

For the text file:

• Block World TMK Txt File

Resolving the stative verbs I chose as indicated in TMKL2 Step 5 page 8, has other blocks on top, free, I added the concept movable to the block class like this:

block description:
Concept(movable) //Boolean property of a block, related to stative verbs
Concept(block {Property(name), Property(movable});
Instance(block, BlockA, True);
Instance(block, BlockB), True;
Instance(block, BlockC), True;
Instance(block, BlockD), True;

For the objects/classes in the Block World Problem, here is the corresponding UML Class diagram (first draft) incorporated in the Knowledge base:

For the method base, the organizer is divided into two FSM, the sub-level being a child of the top level FSM.

Organizer/FSM:
Top Level Transition:
- Compare Current to Goal -> Generate Valid Block Moves* -> Select Next Block Move -> Update Current Arrangement.

Sub Level Transition(Generate Valid Block Moves*):
- Generate Move List -> Apply and Validate Move -> Count Delta Difference

It should be able to follow this state transitons:

Initial
  |
  V
begin_problem_solving
  |
  V
Compare_Current_to_Goal
  |--(if matches)--> Success
  |--(if differs)--> Generate_Valid_Block_Moves*
                           |
                           V
      +-----------------Sub-Level FSM-----------------+
      | Generate_Move_List                            |
      |   |                                           |
      |   V                                           |
      | Apply_and_Validate_Move                       |
      |   |                                           |
      |   V                                           |
      | Count_Delta_Difference                        |
      +-----------------------------------------------+
                           |
                           V
Select_Next_Block_Move
  |
  V
Update_Current_Arrangement
  |
  V
Compare_Current_to_Goal (loop back)

Lastly, I was able to develop a list of Task, Method, Knowledge, and Can’t answer questions for Block World TMK for verification as recommended by the Project Lead.

• List_of_questions_for_MCM_eval txt

DILab Meeting and DILab Summer Reading

In our IVY Project meeting, we started our agenda with the NSF Feedback on our IVY Project. NSF evaluators praised IVY project’s AI and human AI interaction, with recommendations focused on education and learning. Dr. Goal suggests understanding educational goals, learning needs, pedagogical strategies, and instructional design to improve AI-assisted learning. Next, it was discussed that the IVY project needs to be reorganized due to Rochan’s departure and Jeanette’s potential inability to join as a full-time research scientist. In designing interactive exercises for IVY, we look to the perspective of a teacher, they may ask student to provide solution to problem and critique it, they may also identify common misconceptions and test the students on it. Dr. Rugaber highlights importance of distinguishing misconceptions from missteps in teaching. Our post doc Rahul adds that the approach is not limited to tackling student mistakes, but also includes answering a range of questions students might have about a particular skill. These were identified in order for us to better create a learning platform that are aligned with well known instructional designs in education. The rest of the time were spent on evaluating TMK models using various metrics and standardizing the process. Finally, the group plans to reorganize the IVY branches in MCM to simplify the infrastructure and improve the development process.

One of my labmates Samanita presented the paper “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” by the Google Deepmind Team and Jason Wei et al. The main idea is that using chain of thought prompting improves language model performance in solving complex problems by mimicking human reasoning. Moreover, the paper shows that chain of thought prompting can significantly improve language model performance, especially for more complicated problems. With CoT, the model can break down complex problems into steps and arrive at an answer using natural language prompts. For example, CoT prompting outperformed an unaided human sports enthusiast on sports-related questions, demonstrating the potential value of this technique for improving language model performance. Samanita also presented that CoT can be effective for complex problems with multi-step reasoning, but model scale and prompt quality matter. During the discussion, we quickly touched upon transfer learning in machine learning and cognitive science, with a focus on the idea of derivational trace. Dr. Ashok highlights the importance of transferring knowledge from one problem to another and mentions derivational trace. Derivational trace, an idea from the 1980s, is mentioned as a way to make transfer work effectively, involving the steps taken to solve a problem. At the end, it was interesting to note that CoT and prompt engineering did not really help planning tasks. It is because of the nature of planning, which emphasized the importance of understanding relationships between different goals and the interdependence between sub-goals.

DREAM Cohort Meet-Up and Presentation

This week, I got to meet with the DREAM Cohort Team where we discussed the steps to pursuing a PhD. We discussed the importance of contacting the faculty that we are interested to join to gauge interest and availability. This can help potential students avoid unnecessary application expenses if the faculty is not accepting new students. Then, we discussed the financial challenges faced by prospective students, such as application fees and other costs. Naturally, the conversation covered the importance of exploring funding options and grants to support these expenses. Finally, finding the right research fit was highlighted and the importance of aligning interests with potential advisors. At the end, Kathleen mentioned plans for another session in late July or early August, focusing on industry versus academia research. I am thankful for these discussions because these are the things that I also take into consideration. Starting a PhD journey takes a lot of time, effort and dare I say, financial risks. The opportunity costs are blatant especially in the field of Computer Science, where one can attain a bachelors degree (in our case, Masters) and bag a six figure income, then begin saving for retirement and live with financial security. I think that there are many highly driven and very intelligent students that are dissuaded from this path because of this exact predicament. However today, I rest my case.

Literature Review on Chain of Thought

Interestingly, during the Summer Reading discussion session, Prof. Rugaber has raised the question of asking, “What is a chain of thought?”. In my head, I just recalled that the paper literally said that it is ‘a series of intermediate steps’. When thinking about it carefully, every word in this definition is more nuanced than I initially thought. Series means that there is an order to each chain and implies it has linear qualities to it. Intermediate means that there are levels of progression between related things and steps means there could be stages to the progression. In the context of reasoning, this definition really means each ordered step deliberately reinforces a ‘thought’ or set of ideas and meanings (in my opinion).

Piggybacking on the Summer DILab discussion, I read the paper further and with a focus on common sense reasoning. First, what is ‘common sense reasoning’? Common means that some reasoning is shared and reoccuring to many entities. It is an important benchmark for artificial intelligence because humans interact with the physical world while using and developing common sense knowledge. If artificial intelligence has to interact with humans and its physical world, then it also has to understand ‘common sense’.

Results showed that larger models like PaLM 540B benefited significantly from chain-of-thought prompting in common sense reasoning, outperforming previous state-of-the-art models on StrategyQA and demonstrating superior performance on sports understanding. The authors also emphasized the linguistic nature of chain of thought reasoning makes it an ideal way to solve common sense reasoning, which mimics human thought process. However, the study acknowledges that chain-of-thought prompting does not guarantee correct reasoning paths, which can result in both correct and incorrect answers. This is in contrast with humans being able to use a teacher, reference and peers to correct its own reasoning path. Improving factual accuracy in language models remains a key area for future research. I wonder if we are able to correct reasoning path (similar to how humans do it) using reinforcement learning, we would be able to achieve an even better results and findings.