Asynchronous Pair Programming and Personal Learning Network “Distillation”
Getting the Most Out of “GitHub Copilot for Disabled Developers”
I read an insightful article by fast.ai’s Co-founding Researcher Jeremy Howard that set me back on my heels. His article drove me to think more deeply about the human element in my proposed GitHub Copilot for Disabled Developers research study and support program. The blog post on the fast.ai website is titled, “Is GitHub Copilot a blessing, or a curse?” and I encourage you to read it before continuing with this article.
Having read Jeremy’s article, I am sure that my proposed Copilot study and support program design will be better for my having to consider his insights about the potential strengths and non-trivial challenges in the development of AI/ML assistive coding technologies. You see, Jeremy approaches the subject of assessing GitHub Copilot from the perspective of a brilliant (self-taught, learn-by-doing) experienced computer scientist/developer. I, however, experienced Copilot from the point of view as a recently severely disabled developer. With my significantly limited dexterity and mobility, Copilot has been a life-changer for me.
Beauty (and Warts) in the Eye of the Beholder
Copilot enabled me to get back to work on my Digital Humanities research developing text-mining tools and data analysis programs in Python. Copilot met my code development needs with the playing-field-leveling equivalent of voice recognition we use and often take for granted to do routine text writing on today’s smartphones, tablets, and computers. For whatever Jeremy’s well-reasoned limits are of Copilot — as proxy for ML-powered coding assistive technologies in general — just being able to get new code developed has been a blessing to me unimaginable even just a few years ago.
My purpose here is not to enumerate and summarize the many insightful points and cogent examples presented in Jeremy’s blessing-or-curse article. Rather I hope to cite a few of his points that stimulated new thoughts about how to envision the GitHub Copilot for Disabled Developers research study and support program to be more effective for program participants and, potentially, to provide useful feedback to GitHub’s Copilot developers to help improve their technology.
First off, it is important to note that Jeremy is not outright disparaging ML-powered code-generating assistive technology as a Pollyanna-pushed empty promise. Rather he uses his deep personal experience in software development and AI/Machine-Learning to insightfully present the inherent challenges of creating and using this technology. His well-reasoned cautions can be grouped into two broad categories. The challenges for Copilot and similar code-generating assistive technologies are:
- Bad code generation due to “Garbage in, Garbage out”, and
- We humans can be our own worst enemies due to our cognitive shortcuts
I don’t mean to trivialize Jeremy’s insights here. I am just trying to summarize in a way that captures his excellent points by categorizing them using some widely shared colloquial sayings.
The Bad Code Generation Problem
One problem with training Deep Neural Network Language Models for programming language code generation is the result of the intersection of two statistical trends in the body of code on which the model is trained. The Copilot model was trained on the vast storehouse of code repositories at GitHub. As you might expect, the quality of that code varies along the bell-shaped curve that is representative of the skills and knowledge of the coders who wrote it. This demographic of human nature means that most code will be of middling quality written by middling-skilled coders. Add to that some amount of truly bad code by incompetent folks, and thankfully for the users of Open Source software, a good bit of code written by truly gifted and knowledgeable developers.
Copilot and other Deep Neural Network Language Model systems don’t truly learn and reason about writing code the same way our brains do. The capability these systems develop is more akin to a remarkably effective complex and flexible pattern matching mechanism. Having looked at a vast quantity of code, Copilot is able to make a highly educated guess as to what code you might be wanting to write given the context of the code you are editing at this moment.
As Jeremy effectively shows through examples in his article, Copilot’s suggestions inevitably reflect the character of its training data. Since that codebase had lots of so-so code, Copilot’s suggestions — no matter how helpful they are contextualized for your current use — will often be inefficient and verbose in the same way that it has learned from its middling training data. On top of that, since code-generation of these systems do not truly have an underlying deep knowledge of how program languages are executed, generated code suggestions can include coding errors that will need to be identified and fixed.
To add to this bell-curve of the code quality spectrum of Open Source repositories used for code-generation model training, we have to understand an additional complication that results from the long-tail of the accumulation of this code over time. Since programming languages themselves are evolving systems, the code that runs at one point in time may not run, or may be supplanted by new language features that overcome deficiencies of earlier versions of the language. Think, for example, how different a Python script written under version 2.7 looks compared to one written under today’s version 3.10.
The problem is that our training codebase accumulates over time. There will always be lots more of the old and crusty versus the new and improved. This means that the bulk of this training data will be old and potentially inefficient, at best, and outright broken by today’s standards at worst.
While pointing these challenges out with regard to the task of developing code-generation technologies, Jeremy does not leave us with the sense that code-generation is forever a fool’s errand. Rather, these challenges require us to have reasonable skepticism in their use at the current level of maturity of these systems. And, although we don’t yet know the upper bounds of how good such systems can be, we can surely expect the state of the art to improve going forward.
Our Cognitive Shortcuts
All the challenges we face with developing and using Copilot and related code-generation technologies cannot be laid at the virtual feet of those technologies themselves. We humans, with our ever-clever but so often poorly understood cognitive processes, are part of the devil in the details of making best use of code-generation technology. In particular, Jeremy cites two sources of how we think that have a way of biting us in the backside as we try to make effective use of ML-powered code-generation technologies.
The first aspect of our cognitive challenges is automation bias. As cited by Jeremy in his blessing-or-curse article, Wikipedia defines automation bias as:
“…the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct”
With but just a small percentage of us who are relatively immune to this cognitive trait, we tend to be easy marks for the quick answers and suggestions of automated technologies that we take to be imbued with encapsulated knowledge superior to our own. It’s a kind of leftover behavior from our Shaman-worshipping era when we felt that some among us had a superior grasp on the Unknown. We then accept the Shaman’s expert opinion rather than trust our own judgement.
I’m particularly susceptible to this dynamic because I am not an expert developer despite the decades of my involvement in the software industry. I’ve worked with those superhuman coders who are valuable contributors to a development team, and I am in awe of their capabilities. The way my brain works has relegated me to the role of prototyping thought leader and project/team promoter. I’m very good at finding the unexpected Road Less Traveled. I’m just not good at paving it and making it safe for everyday use.
So, like Jeremy points out in his blessing-or-curse article, non-expert developers like me don’t have the knowledge to spot the weaknesses of Copilot and similar code-generation suggestions. We may get a job done but the quality of the finished product may suffer for our acceptance of poor or even broken coding suggestions.
The second of our human foibles that impact our ability to make effective use of Copilot and similar code-generating assistive technologies is anchoring bias. As Jeremy cites for us, The Decision Lab describes anchoring bias as:
“a cognitive bias that causes us to rely too heavily on the first piece of information we are given about a topic.”
Most of us are familiar with this behavior as we fall for it so easily in the Internet era when we are faced with the need to find an answer to a question for which we don’t know an answer. We Google or Bing it, whatever our favorite search engine is. Before Copilot my coding day was spent step-by-step hunting down snippets of code that would do whatever was needed in the next bit of code under development. And sure enough, it was often the case where the first solution I found that made sense to me was The One to be used as other subsequent examples tried but failed to usurp the throne.
Code-generating technologies tend to simply short circuit this process of cognitive bias by popping inline with that first-best solution without our having to take those detours out to StackOverflow and similar code-sharing and advice-offering websites.
So, What’s To Be Done?
At first my inclination was to feel disheartened at my exuberant impression of the value I felt from using Copilot as a recently disabled developer. Jeremy’s sound insights of the “warts” on my Mona Lisa reminded me of my own weaknesses in software development. Perhaps my inexpert judgement of the potential value of Copilot as a life-changer for disabled developers was undermined by my inability to see these weaknesses and challenges in the development and use of Copilot and similar code-generating assistive technologies. Such cognitive dissonant thoughts have plagued me my entire life.
But just as I have tended to do with other bouts of dissonant thinking, after I slept on it I thought through the insightful points of Jeremy’s blessing-or-curse article. I realized that his article had shifted my thinking from the data model and research design aspects that I have recently written about. I now had a rush of thoughts about how the Pair Programming dynamic and cohort-wide Personal Learning Network aspect of my proposed GitHub Copilot for Disabled Developers study and support program could mitigate, and potentially overcome, the challenges of developing and using Copilot as a life-changer for disabled developers.
Asynchronous Pair Programming
Earlier in this series of articles envisioning the GitHub Copilot for Disabled Developers program, I reflected on how I had first experienced Pair Programming as a member of the Smalltalk developer community. In the late 1980s and early 90s, the Internet was young, personal computers were weak, and broadband communication was a pipe-dream as slow telephonic modems still ruled. Back then, Pair Programming was done face-to-face in Real Time at one computer with shared keyboard, mouse, and monitor.
But today, so much has changed. Given the technological and societal changes we’ve experienced, it is not only possible but is likely probable that this research study and support program idea will be realized as a global and distributed network of enthusiastic participants. Today’s technology makes Real Time distant/remote Pair Programming not just possible but increasingly likely to be used within our program.
Sure, there are some rough edges with regard to incorporating Copilot into remote Pair Programming. For example, at present Copilot suggestions are shown only to the Pair member adding code to a file being actively edited. The observing member only sees the code inserted by the “driver” after a Copilot suggestion is accepted. This is an aspect of the current VSCode LiveShare extension that perhaps will be addressed in a future update of the IDE’s UI/UX experience. But even if it is not addressed, there are ways to develop Pair interactive behavior to account for this limitation. Interactive chatting and powerful undo/redo editing features allow Copilot to be a “third party” contributor to the Real Time Remote Pair Programming experience.
As I thought through the Pair Programming use case scenarios and considered the potentially geographically distributed nature of the GitHub Copilot for Disabled Developers participant community, I realized that there was a good possibility that time zones and differential personal schedules could result in Pairs needing to adopt an asynchronous mode of working together.
Rather than coding interactively in Real Time, we need to refine our abilities to creatively interact as Pair members over time and distance. In this scenario it is most likely that the novice Pair member will initiate a next-step session, leaving a trail of new code with Copilot-supported input which the expert Pair member reviews and accepts or improves with feedback to support the novice’s learning experience.
This fine-grained incorporation of code review into the remote Pair Programming experience can contribute to overcoming the challenges of time zone and personal schedule differences in our project community. In addition, this code-reviewing step in the process can help address Jeremy’s noted challenge of bad code contributed by Copilot to the work produced by the Pairs in our project.
To make this review and improve-or-correct step even more valuable, we can develop a feedback loop to provide user experience data to GitHub’s Copilot development team. This feedback can then be used to refine the Copilot model to improve its suggestions and interactive experience. Valuable feedback to the researchers and support program managers can help improve and document the project as well.
Personal Learning Network “Distillation”
Having thought through this idea of Asynchronous Pair Programming as a use case, I expanded my thinking to explore the value and behaviors of the Personal Learning Network (PLN) level of the proposed GitHub Copilot for Disabled Developers research study and support program. As envisioned, the program would run as a series of scheduled cohorts throughout the calendar year. Participants would apply and be admitted to the program and enter a cohort on a rolling admission basis, much like an academic calendar year. At minimum, a cohort would consist of ten differentially-composed Programming Pairs. The first cohort would be the minimum set of 20 participants to satisfy the ten Pair compositions as described in earlier posts to this publication. With program delivery experience and participant interest, subsequent cohorts could be enlarged by increments of 20 paired participants.
To support the work of individual Pairs, the cohort will meet virtually as an extended PLN twice a month. During these group meetings, each Pair will provide an experience report by the Novice Pair member. These reports would be followed by the cohort’s responding to Pair requests for help or review of specific coding issues. In follow-up to each cohort-wide meeting, participants will fill in a short experience report survey with comments to provide the study’s researchers and program managers valuable data about the value and effectiveness of the program.
This cohort-wide PLN layer of the program will further improve individual Pair’s code contribution to their affiliated Open Source, Digital Humanities, or Citizen Science projects. In this way the PLN layer of the program design will provide a “distillation” filter that further addresses code quality concerns highlighted by Jeremy in his blessings-or-curse article.
And although not related to the Copilot code quality improvement and feedback to the program managers and GitHub AI development team, the PLN level activity of our program will contribute to the social contact and self confidence-building aspects for the program’s disabled developer participants.
Going Forward
Before thinking about the next steps in envisioning the GitHub Copilot for Disabled Developers study and support program, I want to take a moment to thank Jeremy Howard. The insights and examples in his blessing-or-curse article about the challenges facing the development and use of Copilot and similar code-generation assistive technologies helped me to think creatively about the design and management of my proposed program.
As to what’s next, I will explore ideas about the Pair Programming Platform to be used to capture and facilitate participant interaction and feedback to the program managers and Copilot developers. This platform will be a combination of hardware and software to enable both active and passive data collection.
Until my next article, thank you for reading this one. If you are interested in participating as a Pair participant, disability researcher, or seed funder, do not hesitate to contact me through this Medium publication’s communication features or feel free to contact me by tweet or DM on Twitter at @Jim_Salmons. In the meantime…
Happy Healthy Vibes from Colorado USA,
-: Jim :-
Jim Salmons is a seventy-one year old post-cancer Digital Humanities Citizen Scientist. His primary research is focused on the development of a Ground Truth Storage format providing an integrated complex document structure and content depiction model for the study of digitized collections of print era magazines and newspapers. A July 2020 fall at home resulted in a severe spinal cord injury that has dramatically compromised his manual dexterity and mobility.
Jim was fortunate to be provided access to the GitHub Copilot Technology Early Access Community during his initial efforts to get back to work on the Python-based tool development activities of his primary research interest. Upon experiencing the dramatic positive impact of GitHub Copilot on his own development productivity, he became passionately interested in designing a research and support program to investigate and document the use of this innovative programming assistive technology for use by disabled developers.
