“HeyGitHub!”, Thoughts on Agency and Conversational Copilot
CoT (Chain of Thought) for Goal-Directed Code-Generation
In my last two more-technical articles for this publication, I let myself go on a rather untethered envisioning of adding “Sherlockean” deductive thinking into the #HeyGitHub voice-coding dialogue between #DisabledDevelopers and the brilliant but savant-like “Rainman” GitHub Copilot code-generator. While writing these articles, I began a search-quest for methods and publicly available implementations of advances in the Machine-Learning research community that could provide the kind of support needed to implement these more thoughtful code-generating conversations.
Happily I found that there are a number of Machine-Learning research sub-communities working on these challenges to support more “human-like” thinking processes, using context and prior knowledge, as a complement to the brute-force pattern recognition of LLM (Large Language Model) performance. I learned about #CoT (Chain of Thought), prompt-sequencing, model-switching, and agents among other areas of exploration and development in the Machine-Learning domain. I’m beginning to get firsthand experience using the exciting LangChain Python library from Harrison Chase and the LangChainAI developers. Many of these breadcrumb trails I followed started by my reading the insightful overview article, “The Near Future of AI is Action-Driven”, by John McDonnell.
I also learned about a massive, hyper-funded agenda underway at Microsoft — Project Bonsai — that is bringing AI/ML solutions to the challenges of robotics automation and industrial process control. In this physical-world problem space, simulation and human-in-the-loop coaching are just as important, if not more so, as the emphasis on data-modeling and data-science methods in the more general Machine-Learning application domain.
My focus in this article is not to delve more deeply into these SOTA advancements, but rather to speculate a bit about the role of role-based agency as it may be applied to the #HeyGitHub/Copilot platform in support of software development not just by #DisabledDevelopers and #DisabledSTEMstudents, but by all prospective developers using #HeyGitHub/Copilot as a development productivity multiplier.
A Wayback Reflection on Agency in Executable Business Models
In my Metamodel Subgraph article in this #HeyGitHub series, I gave a brief overview of my involvement in a skunkworks developing a Smalltalk-based framework supporting the dynamic composition and application generation of EBMs, Executable Business Models.
The early-mid 1990s saw a range of model-driven methods for both software development and business process engineering. One strong thread in this era was the BPR, or Business Process Reengineering, movement. Within IBM’s consulting services, a popular method used at the time was called LOVEM, the Line Of Visibility Enterprise Model or sometimes, Line Of Visibility Engineering Method. Central to this method was a diagram format using “swim lanes” for business process models.
LOVEM’s swim lanes represented roles that could be played by either humans or machines (AKA software processes). Inspired by David Gelernter’s 1993 book, “Mirror Worlds: or the Day Software Puts the Universe in a Shoebox…How It Will Happen and What It Will Mean.”, my colleagues and I in IBM’s Object Technology Practice envisioned a Smalltalk object framework that could be used to compose and execute these role-based LOVEM models. Our EBM, Executable Business Model, framework was based on a metamodel “construction kit” of object classes that looked very much like this UML Class model from early activity in my post-cancer rebirth as a Digital Humanities Citizen Scientist:
The aspect of our EBM model I want to highlight in this article is the cluster of classes seen in the red box in the lower-left corner of this diagram. These three classes — Persons or Agents as Actors — participate in Goal-driven, Role-based processes. In this way our EBM framework implemented the concept of Agency essential to the LOVEM business process method. Our metamodel’s implementation of Agency was, as you might expect, driven by our Mirror Worlds inspiration.
In his book, Gelernter took readers on a thought experiment that said, paraphrasing here, “Imagine if we built detailed software simulations of our complex systems at work in the Real World. Once we have these simulations humming along, imagine what would happen if we took all the input and output feeds of our simulations and hooked them up to the actual feeds in these Real World systems. At that point, our software models would stop being simulations and start being heads-up displays, or dashboard views, on the Real World…” or what Gelernter called Mirror Worlds.
We implemented this approach to Agency in our EBM metamodel-based framework so Agent objects could be implemented as software proxies for Persons as Actors of Roles in executable LOVEM models. This design gave us two important features consistent with our Mirror Worlds inspiration. First, during consulting engagement sessions with customer executives, we could have SMEs, customers’ subject matter experts, sit at networked laptops in a conference room to interact with and validate our evolving simulation models. During these sessions, we could populate an executing model instance with sim-actor Agent software proxies to perform all the other Roles in a modeled business process.
Once the customers’ SMEs were convinced we had nailed these business processes in our EBM simulation we could — in true Mirror Worlds-inspired fashion — simply hook these models up as a deployed system in the actual customer business. During this hook-up process we would determine which Roles were to be played by Persons — using dynamically-generated EBM views on the model seen by users as typical software applications — or by Roles to be played by software proxy Agent Actors in the customers’ business processes.
Reflecting on that time nearly thirty years ago, my days spent working as a thought-leader/developer in this EBM skunkworks were some of the most exhilarating experiences in my working career.
A Forward-looking Speculation About Agency in the #HeyGitHub/Copilot Platform
So, how might the concept of Agency contribute to our leveraging the capabilities of such approaches as Chain of Thought, prompt-sequencing, and model-switching as part of the evolving stack of technologies contributing to the implementation of GitHub’s #HeyGitHub/Copilot service?
With no intention to rigor or non-trivial example, we might apply this concept of Role-based Agency from my EBM experience to the solution design of the #HeyGitHub/Copilot service as in this simple swim lane diagram:
What we see in this overly simple diagram is a cluster of four swim lanes in the upper layers that actively interact as a Human and three software Agents whose conversation leads to the eventual generation in the “dumb”/data transcriptionist-like Roles of the IDE app and source file (memory). This simple diagram makes clear that the tremendous challenge we have going forward is how to inject “Sherlockean smarts” into that Agent-based conversation before making best use of our savant-like Rainman-Agent Copilot LLM.
Imagine our design requirement is, for example, creating a high-functioning telephone Call Center bot Agent. In this case, a flexible repertoire of template-based prompts that can be dynamically selected for a goal-oriented result might be sufficient to go to deployment. But successfully acting as a non-human Pair Programmer (Agent “buddy”), as GitHub often asserts in describing its Copilot service, the #HeyGitHub Listener and CoT Dialoguer Agents will need to implement a far more complex intelligence than a scriptable Call Center bot conversation.
Let’s take, for example, the challenge for #HeyGitHub/Copilot helping a Developer to create the UI/UX — user interface and interactive experience — for an application. There are a decent number of excellent UI/UX frameworks the Developer might use. Our Rainman-savant Copilot has already seen countless examples of these frameworks in the Copilot’s foundation model training to refine its code-generating talent. But being able to helpfully provide code suggestions based on pattern-recognition drawn from its training experience does not afford Copilot the underlying understanding of the framework architecture, best practices, and user interface/interaction guidelines that a competent human Developer brings to the role of participating in a Programming Pair partnership.
Consider the example of the wxPython wrapper on the venerable C++ wxWidgets framework. No amount of random walks through the myriad of publicly accessible code repositories will reveal the extent of the knowledge imparted by taking a deep dive into the excellent online documentation website at https://docs.wxpython.org/. No amount of NLP decomposition, topic-modeling/summation rumination on the content of this website will be sufficient to impart human-developer knowledge into the #HeyGitHub/Copilot<>Developer conversation.
So Where Do We Go From Here?
I wish I had domain knowledge enough to suggest a specific solution design and development path ahead to produce the technologies needed to take the performance of today’s code-generating LLMs to that proverbial “next level.” While the full roadmap ahead is yet to be written, there are some assumptions we can make about next steps. We can anticipate that it will be important for the #HeyGitHub Listener agent to be able to distinguish between Developer voice inputs that require a hand-off to either the CoT Dialoguer for further thoughtful interaction, or passed through in textual form to the Copilot LLM for generation of code suggestions.
The gnarly challenge ahead of us is to encapsulate the information structures and semantic meaning of the art and science of software engineering such that the CoT Dialoguer can perform credibly as a true Programming Pair partner. The technologies for meeting this challenge will likely come from the leading edge of research activity around Chain of Thought, prompt-selection/sequencing, model-switching, and human-in-the-loop reinforcement learning/coaching.
Some of the breakthroughs in this “Sherlockean smarts” solution space will involve brilliant innovative coding by Machine Learning research engineers. I also believe, however, that a non-trivial contribution to this challenge will come from the design and development of effective Ground Truth reference model datasets that will serve as training curricular materials for domain-specific models to work hand-in-hand in a model-switching context with the ever-evolving foundational Copilot Large Language Model. An example of these emerging reference datasets is the Metamodel Subgraph Ground Truth Storage design I have been pursuing through my Digital Humanities research and envisioned here in my #HeyGitHub series of articles.
But developing a standard Agent-oriented Ground Truth reference dataset format will not be sufficient to evoke the Sherlockean dialogues I envision between a human Developer and an ML-based Copilot Pair Programming partner. Rather these Ground Truth reference datasets of domain knowledge will need to serve as instructional material in model-training simulated user-interactive scenarios that refine the CoT Dialoguer agent’s ability to engage in collaborative conversation with its human Developer partner. These simulation sessions can time-compress and tirelessly pursue refining the CoT Dialoguer’s effective use of domain knowledge.
Just as we now use data-modeling best practices to generate training datasets for current data-intensive ML applications, so too will we be able to generate Agent-based simulation experiences that allow our knowledge-driven applications to train and refine models like the imagined future #HeyGitHub/Copilot service. As part of this Mirror Worlds-like simulation training environment, conversational exchanges that stymie the in-training #HeyGitHub CoT Dialoguer model will pop up to the human-in-the-loop Supervisor whose response coaching will encourage and reinforce appropriate model behavior.
A Couple Bees in the Bonnets at GitHub and Microsoft
At this point my flights of imagining the way forward are little more than the fever dreams of an indie Digital Humanities Citizen Scientist and Disabled Developer. But if I were to dust off my old IBM consultant hat to put a couple of bees in the bonnets at GitHub and Microsoft, I know what I would suggest…
First, I can’t help but say, if I were part of GitHub management I would offer me a job as a member of the GitHub Next research team. In that position I would work diligently on the ideas I have written about in this series of #HeyGitHub articles*.
Next, once we have Proof of Concept for the Metamodel Subgraph Ground Truth domain knowledge model-training datasets, I would lobby to dedicate a portion of the recently announced GitHub Accelerator or the M12 GitHub Fund to provide stipends or project startup funding to high-performance, expert Open Source developers to create a collection of these domain-specific Agent-oriented model-training datasets. A particular emphasis should be considered for developing UI/UX framework-specific metamodel-based training datasets.
Why UI/UX framework-specific domain knowledge datasets? Because these “beasts” are often large, complex architectures with parameter-wise excessively detailed customization. And for most development project scenarios, crafting an application UI and its user interactions are necessary activities that take time and effort away from coding the solution to the project’s problem space that the application UI/UX simple makes available to the project’s end-user. Providing strong UI/UX framework code-generation by voice input in #HeyGitHub/Copilot is then comparable to having a UI/UX framework having a powerful WYSIWYG interface builder application.
As a more far-ranging and strategic recommendation, I would encourage thought-leading contacts from GitHub Next and Microsoft’s Project Bonsai to form a Collaboration Committee. The purpose of this group would be to explore ways that GitHub could leverage the best practices and experience of the extensive commitment Microsoft has put into the robotic and industrial process automation markets.
And Hopefully Lurking Just Over the Horizon
Looking even further down the Innovation Highway while putting on my rose-colored glasses and optimistic slogan tee-shirt, where might this research agenda lead? If, after creating a limited number of domain-specific Metamodel Subgraph reference Ground Truth datasets, and taking the time and effort to train a Machine Learning model to play the CoT Dialoguer agent role in a number of domains, we may someday see the Dialoguer system demonstrate emergent behavior.
By this I mean, the CoT Dialoguer Agent model may understand the generalized structure and use of domain knowledge so well that it does not have to be preloaded with a new domain-specific reference model to become useful in participating in a new context. At this level of function, the far-future #HeyGitHub/Copilot multi-model system may simply engage in a self-directed learning conversation with its Developer Pair Programming partner to create and validate its own new domain knowledge model. Through this process the #HeyGitHub/Copilot service may become a truly versatile peer-level collaborator in the design and development of essential software systems.
In Closing
After surviving my cancer battle and learning to cope with the dexterity and mobility challenges of my spinal cord injury, I would love nothing more than to have a chance to Beat the Reaper by contributing to the exciting coming wave of remarkable innovations in the design and development of software as envisioned in my #HeyGitHub series of articles. 🤓👍🤞✌️
Happy Healthy Vibes from Colorado USA,
-: Jim :-
* P.S. To be clear, in addition to my interest to pursue the research agenda envisioned in this #HeyGitHub article series, I remain passionately committed to implementation of the agenda of Copilot as #AssistiveTechnology through the research study and support program for #DisabledDevelopers and #DisabledSTEMstudents described in the bulk of articles in this Medium publication.
Jim Salmons is a seventy-one year old post-cancer Digital Humanities Citizen Scientist. His primary research is focused on the development of a Ground Truth Storage format providing an integrated complex document structure and content depiction model for the study of digitized collections of print era magazines and newspapers. A July 2020 fall at home resulted in a severe spinal cord injury that has dramatically compromised his manual dexterity and mobility.
Jim was fortunate to be provided access to the GitHub Copilot Technology Early Access Community during his initial efforts to get back to work on the Python-based tool development activities of his primary research interest. Upon experiencing the dramatic positive impact of GitHub Copilot on his own development productivity, he became passionately interested in designing a research and support program to investigate and document the use of this innovative programming assistive technology for use by disabled developers.
