Zero cost software engineering. What is it? There are a few of different contexts. Let’s use this one. It’s soup-to-nuts software you build by instructing an agent. No need for a team of software engineers to turn your wishes, ideas, and requirements into runnable code. You do it yourself with a virtual software engineer.
Think you can’t do it? Why? Are you detailed oriented, great at writing and itemizing instructions, understand how things can fail, and were pretty good figuring out those truth tables in 9th grade math? Then you are half way there. Just trust me.
Think it won’t happen? It will. Give this WealthTrack episode a listen from T. Rowe Price’s Global Technology Fund manager Dom Rizzo [1]. Software development is going to become a zero cost game. It simply must – it’s too easy for it not to! This post explains how it might unfold.
Take One: Adaptive Script Execution
I first toyed with the idea of zero cost software engineering in the spring and summer of 2023. The appearance of large language models (LLMs) in the Fall of 2022 motivated the idea. I and two of my student researchers, Andrew and Ellie, worked on a first cut. You can read all about it in this arXiv article here [2]. Figure One shows the intuition:
Figure One: Initial Concept of LLM Backed Zero Cost Engineering
It was a neat idea. We had a text editor where the business logic of some buttons was served up by an LLM on demand. You could define new buttons and menu options in real time as well. The compiled Python code would be added to the executable and run as the user defined it. This allowed for some customization of the UI’s behavior, though not all behavior. However, it had some drawbacks:
- LLMs cost money – We needed an API key from ChatGPT to get it to generate our code. Each request cost money.
- Messy requirements – It’s not entirely advisable for users to issue purely natural language directives for how they want their UI to behave. Some structure and itemized requirements are necessary. We needed a way to precisely specify the behavior and failure cases to best guide the virtual software engineer.
Lessons were learned and the project was set aside. Onto teaching for two semesters.
Take Two: Containerization
I rekindled the project this summer. So much has changed since last year! Everything I write about you can find here, on GitHub [3]. This time around, the idea of zero cost needed to have some teeth. A real path to being used in Industry or the research community. Welcome Docker containers. Welcome Ollama and AI democratization. The pieces are starting to fall in place.
Take a peek at Figure Two which encapsulates what the GitHub repo holds. It’s a component diagram mashed up with a data flow. Here are the main components:
Figure Two: The Zero Cost Components
1. Client UI – This is where the user enters what they want the program to do. It’s a basic implementation of what we do with LLMs all the time (if you are the programming type); “Go and make me this code and have it do this.” Except for one thing – once you click submit, the Zero Cost system is going to compile and run your code for you as well. And persist it. And (eventually) allow you to extend it.
2. Zero Cost Coordinator – It’s a real legit Flask Server. Ollama and Mistral helped me to create it – for free. The server:
- Receives the users request from the Client UI,
- Sends out the request to create the runnable code,
- Compiles and runs it,
- Sends back the result to the Client UI. Sometimes it works, sometimes it fails (remember – it’s a prototype!) relaying the failure message.
Many moving parts here! My wee little flask server has to get the code back from Ollama, ‘compile it’, load it in real time, and then run it. Note, I went down a path with the code compiler and runner not being the Flask Server itself but an entirely separate Docker container. Every LLM I consulted to help me create Python code that would create and spawn Docker containers said “A Docker container that creates other containers! Hold on!” However, it is indeed the way go. We need to make the runnable code a micro container that can be reused. It’s a future task; integrating microcontainers into the architecture.
3. Virtual Engineer – The Docker version of Ollama! A high quality LLM engine running locally on my (or your) laptop. Remember that issue with LLMs costing money from the summer of 2023 – problem solved! It’s basically a virtual engineer – but it needs some guidance.
Data Flow: Here is how this works.
- docker-compose.yml – This fires up our system. It creates the containers for both the Coordinator and Ollama. We have a virtual software engineer and runnable system with just these two components.
- Client UI – Go to the URL http://localhost:8013/ and you have available a simple screen to instruct our virtual engineer (don’t worry – Cut Three will be all about streamlining this UI – its the key to everything!) You enter a set of instructions and the input the program requires if it needs it. I know – it’s very basic (remember – it’s a prototype!)
- Submit – Send your request to the Coordinator! There, it unpacks your JSON message and gets to work. There is an inherent prompt that instructs Ollama and whatever model (codellama, mistral, etc.) to generate pure code – no flowery wrapper descriptions of what the code does, please.
- Compile & Run – The Coordinator gets Ollama’s reply. At this point, it literally compiles this code and adds it to the running executable so that it may be called. Either the code runs successfully, or it fails. I toyed with a very primitive self-healing feedback loop that tried five times to solve any missing library or minor compile issues.
- Output – If the code runs without error, it returns what the instructions asked for. It will work for basic examples. In Figure Three I ask for a program that will go to a URL, download a zip file, and extract the first two lines from a CSV and return them. I gave it an example URL – one that had Airbnb data. Guess what it returned. Exactly what I wanted – the first two lines from the file. That’s the goal of Zero Cost – you instruct the computer on exactly what you want it to do.
Figure Three: Zero Cost Getting the Job Done
Take Three: Coming Fall of 2024!
Remember back to adaptive script execution? It was like rolling the dice, you never quite knew if the code would work as envisioned. The longer the sequential set of instructions the more likely it was to not work. Zero cost needs two items to overcome this:
- Workflows: Natural language instructions need to be encapsulated in a workflow with decision points for when things go wrong. I have looked at NiFi and Airflow, these are probably overkill as an initial starting point. PyFlow is probably a good starting point.
- Controlled Language: It’s probably time to pick up something like Attempo [4]. This would provide a more structured language for people to enter instructions for the code they want.
Take Three will start this Fall. More to come on that!
Conclusion
There are certainly good arguments for why auto generated code is not quite ready for application to all programming tasks. See this article, The Future of Software Engineering and Acquisition with Generative AI, from Carnegie Mellon SEI [4]. It is a great analysis of when to use generative AI to create code and when you may want to cut bait (see Figure 3 of the article – what a great graphic!)
However, think of it like a baseball game. Zero cost engineering has just completed its warmups and taken the field. Nine innings to play – lots of improvements to be made – and lots of things to learn along the way. It’s going to be wild game. It might just end up with a new tool that fundamentally changes how people interact with computers.
Footnotes:
[1] Here is a link to a WealthTrack episode about many things AI related. A discussion on zero cost software development is in there – but you should listen to the whole thing, its that good: https://wealthtrack.com/ai-could-be-the-biggest-productivity-enhancer-since-electricity-says-t-rowe-prices-dominic-rizzo/
[2] Link to arXiv article: https://arxiv.org/abs/2408.11060
[3] Link to the Zero Cost GitHub Project: https://github.com/justinGmail/ZeroCostEngineering
[4] Link to Attempo project: http://attempto.ifi.uzh.ch/site/description/
[5] Crosstalk: https://community.apan.org/wg/crosstalk/m/documents/464157