The Generative AI (Gen AI) wave is becoming more powerful every day. This is because as Gen AI engines ingest more data, their ability to produce better output gets a boost. Gen AI can see, hear, and read, making its way through data easily and quickly. But data is not limitless, although often it seems as if it is infinite. The implication is simple: Gen AI will eventually begin to plateau. The question to explore is: How and when will Gen AI plateau, and what can we do to delay this eventuality?
Take the case of a pharma company. It would not want to invest time and money in feeding raw data to its Gen AI system irrelevant to its goals. Data engineers would, therefore, point it to a relevant but finite universe of training data. In other words, the input data would be limited.
Doubtless, as time goes by, based on changing business requirements, the size and scope of this “universe of data” used by data engineers will expand marginally. However, there will be no significant jump in training data volumes; at least, we should not expect that to happen too often. This logic will remain valid for any industry, from retail to manufacturing, education, entertainment, etc.
When the Gen AI system has ingested all (or most data), it will add to that data universe with its output. Soon, it will begin to use the work it has created to coach itself.
At some point, say when 30 to 40 percent of the total volume of data has its origins in the Gen AI system, the lack of diversity will have a visible impact. The Gen AI system will now begin to regurgitate data with a sameness. To many, it will sound predictable. Nothing will be new about the “viewpoints” and “ideas” these systems throw up because they lack access to fresh and diverse input. Mark this carefully: the data produced will be accurate and valuable but no longer feel fresh, different, or innovative.
The obvious answer to this problem is to give the Gen AI system new data. The other solution is to give the Gen AI system more autonomy. This is an argument that often finds instant and considerable resistance. This is as it should be. We should worry about — and be wary of — autonomous systems. We must insist on placing great emphasis on how the behavior of these autonomous systems is determined, monitored, controlled, and regulated to achieve the highest levels of safety before they are deployed. But the truth is rather obvious and cannot be scoffed at. We want to make these systems as human as possible — and humans crave more autonomy and interactivity, not less. Achieving a balance between autonomy and compliance for Gen AI systems may or may not be difficult. Still, it seems to be the logical way forward, and more autonomy and interactivity are inevitable. We will probably let Gen AI systems requisition — or request for, from their human supervisors — other applications and relevant databases as and when the need arises to make them more effective.
Given our current failures with online regulations, providing more autonomy to machines will sound like unfettered techno-optimism. But progress is a constant trade-off between differences, conflicts, fairness, needs, and benefits.
Given these uncomfortable issues, Gen AI will use other hacks and solutions to ensure the output quality does not dip significantly. It is only human for us to consider providing more autonomy to machines as a “last resort.”
What could the other solutions be to extract more out of Gen AI without having to deal with the “autonomy” challenge?
One area to examine with greater thoroughness would be prompt engineering. Microsoft tells us, “Prompt engineering, also known as prompt design, is an emerging field that requires creativity and attention to detail. It involves selecting the right words, phrases, symbols, and formats that guide the model in generating high-quality and relevant texts.” (The output need not be limited to “text” — the model can respond with images, video, music, code, and actions.) Prompt engineers need to have a mastery over large language models (LLMs), deep learning, NLP, programming, applications, ethical challenges, model testing, and business needs. They can then use an iterative method — a Chain-of-Thought process — to improve the output.
The role of a prompt engineer is already becoming infinitely more sophisticated than we had first imagined. Today, new techniques have emerged that help Gen AI systems deliver better results. One is called Promptbreeder, a self-referential self-improvement process that evolves, mutates, and adapts prompts.
Promptbreeder outperforms state-of-the-art prompt strategies, say researchers at Cornell University in a paper on Promptbreeder. “Crucially,” says their report, “The mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutation-prompts that improve these task-prompts.”
Much work needs to go into processes that extract more from Gen AI. In the coming months, this is where the focus will shift to.
