AI

Data to the fore as Snowflake and Nvidia report

Data architects will have an increasing role in AI discussions, and may offer their usual cautionary perspective. When people ask how soon GenAI will take hold, they might keep this in mind.

So far, the major gauges for GenAI progress have been the hyperscalers’ capex accounting and Nvidia’s quarterly results. The latter came in last week, and the general picture appeared lackluster to muddy. Only Nvidia’s tremendous three-year upsurge can explain the use of the ‘lackluster’ term; its sales jumped +56% in the quarter. [See footnote to this story.]

What’s happening with the Generative AI juggernaut may hinge a lot on results of firms like Snowflake, makers of the noted cloud-based data platform.

A quick view on that came via Mandeep Singh, Global Head for Technology Research at Bloomberg. He spoke on Bloomberg Podcasts just as both Nvidia and Snowflake numbers came in. The context was today’s general search for a defined timeline for GenAI deployment and monetization.

What’s going on now, Singh says, is that companies are plying through their own enterprise data. They do this in order to create specialized AI agents that bring the power of LLMs to desktops.

“That’s where someone like Snowflake or MongoDB benefits, because your training is getting a lot more specialized depending on the use case,” he said. “It’s not just about the compute cluster anymore, but also about the data. I think that points to why Snowflake is doing well.”

In fact, the company beat Wall Street expectations for both earnings per share (EPS) and revenue. Quarterly revenue grew 32% Y2Y, to $1.14 billion. The company cited adoption of AI tools as a growth driver.

The importance of cloud and on-premises databases and data warehouses in this regard would be emphasized this week as OpenAI bought Statsig and its “data warehouse native” platform for, of course, something in the neighborhood of $1 Billion. -JV

~~~~~~~~~~~~~~~~~~~~

Footnote – Nvidia leader Jensen Huang and his company are caught right now between China leader Xi Ping and US leader Donald Trump, both pursuing new forms of State Capitalism or Capitalist Statism. The Era is still searching for a name.

Speaking of Agentic AI data architecture

Harvard Stadium

Connecting-to and building-out Generative AI processing in day-to-day business is not the easy task it may seem, given the colorful renderings of the future that ChatGPT is known for.

Sometimes the AI train seems stuck at the HyperScaler depot. Recent news shows that AI leader Nvidia tech, while firmly established in the top rank HyperScaler houses – is still edging out, and not yet hurtling forward in the wider space.

Another view of data technology’s potential role in changing that scenario is seen in a recent Diversity.net blog post by Mohan Varthakavi, Vice President of Software Development, AI and Edge, Couchbase. “Reimagining Data Architecture for Agentic AI” addresses the data challenge from an overarching perspective, and it is an interesting take.

Varthakavi said traditional data architectures are insufficient for the demands of Agentic AI, that being the medium generally expected to bring Generative AI into wide use.

Here he refers to Generative AI’s need to work on unstructured data – that is, something outside the realm of SQL and often taking the form of human speech. Diligently prepping such data for GenAI, and finding you may not have enough to feed the AI beast, is a common problem today. And, as Agentic AI is currently envisioned, it means feeding numerous agents collaborating around a host of tasks.

Still, as Varthakavi, writes: “Advanced unstructured data processing is quickly emerging as the defining differentiator between AI leaders and followers.” It’s not that different than it was in the Big Data era. One of Big Data’s hallmark “V’ traits – Variety – still looms larger than ever.

If Agentic AI is going to work, a fundamental departure from traditional data frameworks is called for, he said. Going forward, the back-and-forth music of conversations, the probabilistic nature of machine learning, and the deep complexity of human language will call the shots, and shift the approaches used by those who apply system architecture.

The shift to agentic AI … marks a migration from traditional rule-based logic toward architectures centered around language understanding. This isn’t as simple as swapping one model for another; it requires a rethinking of how systems are composed. Large language models can provide powerful general capabilities, but they are not equipped to answer every question pertaining to a company’s specific business domain.

Also noted by Varthakavi is the growing need for architects to be aware of new types of data dependency risks. These appear when agents that use LLMs may receive corrupted or insufficiently updated data. Latency pressures also confront the designers, as workloads and user-machine dialogs become more complex.

Just as important in the Brave New World are conversation storage systems that provide persistent memory, vector search engines that quickly contextualize customer queries in the data format that machine learning likes best, and dynamic memory management to support real-time context updates.

Few people arrive at work in the morning with a complete mastery of these kinds of requirements. But these underlie the elements implied in the futuristic advertisements for Future AI.

Enterprise AI attention turns to data architecture

Getting Generative AI apps up and running is the first problem that enterprise teams encounter. Then comes maintaining those systems. As Sanjay Mohan points out in a recent blog post on data quality and AI projects, this makes for a moving target, as data flowing into the AI engines must be continually monitored. Constant changes are inherent in active computer data.

Once again, it is meta data – dull as the day is long – that is a key factor in managing data for these new AI platforms. But there is more. In this thoughtful blog, industry analyst Mohan enumerates a host of quality, reliability, access control, lineage, and observability processes that organizations must master in order to turn AI dreams into reality.

Why is the need for good governance more important now? Mohan writes:

Because more people are accessing more data for more business use cases than ever before. Without trusted and reliable data for AI and analytics, the outcomes will be poor, time and money will be wasted, and business leadership will lose enthusiasm for and confidence in AI and analytics.

He goes on to emphasize structured approaches to governing data for AI.

Mohan is spot on here. He and other go-to data experts have been on the case since about Day One of the great Gen AI rising. They warn these data pipelines must be continuously adapted to deal with the volatility of data in the real world. It was a lesson revisited during the Big Data days – but as Gen AI took shape it’s been one PT Barnum’s of high tech have found convenient to gloss over.

It took two-plus years for such for “data caution” to gain much of a hearing amid the AI swarm. Like summer football camps, data governance for AI requires a continual focus on blocking and tackling. It’s important although it comes without the excitement of the long passes that will be remembered from the Sunday games. [Ed.Note: The Correspondent promises no more sports analogies for 12 months.]

Nvidia and Cloudflare Deal; NYT on Human Feedback Loop

September 29, 2023 By Jack Vaughan

Cloudflare Powers Hyper-local AI inference with Nvidia – The triumph that is Nvidia these days can’t be overstated – although the wolves on Wall St. have sometimes tried. Still, Nvidia is a hardware company. Ok, let’s say Nvidia is still arguably a hardware company. Its chops are considerable, but essentially it’s all about the GPU.

Nvidia is ready to take on the very top ranks in computing. But, to do so, it needs more feet on the street. So, it is on the trail of such, as seen in a steady stream of alliances, partners and showcase customers.

That’s a backdrop to this week’s announcement that Cloudflare Inc will deploy Nvidia GPUs along with Nvidia Ethernet switch ASICs at the Internet’s edge. The purpose is to enable AI inferencing, which is the runtime task that follows AI model training.

“AI inference on a network is going to be the sweet spot for many businesses,” Matthew Prince, CEO and co-founder, Cloudflare, said in a company release concerning the Cloudflare/Nvidia deal. Cloudflare said NVIDIA GPUs will be available for inference tasks in over 100 cities (or hyper-localities) by the end of 2023, and “nearly everywhere Cloudflare’s network extends by the end of 2024.”

CloudFlare has found expansive use in Web application acceleration, and could help Nvidia in its efforts to capitalize on GPU technology’s use in the amber fields of next-wave generative AI applications.

With such alliances, all Nvidia has to do is keep punching out those GPUs – and development tools for model building.

*** *** *** ***

NYT on Human Feedback – The dirty little secret in the rise of machine learning was labeling. Labeling can be human-labor intensive, time-consuming and expensive. It harkens back to the days when ‘computer’ was a human job title, and filing index cards was a way of life.

Amazon’s Mechanical Turk – a crowdsourcing marketplace amusedly named after the 18^th Century chess-playing machine “automaton” that was actually powered by a chess master hidden inside the apparatus — is still a very common way to label machine learning data.

Labeling doesn’t go away as Generative AI happens. As the world delves into what Generative AI is, it turns out that human labelers are a pretty significant part.

That was borne out by some of the research I did in the summer for “LLMs, generative AI loom large for MLOps practices” for SDxCentral.com. Sources for the story also discussed how “reinforcement learning through human feedback” was needed for the Large Language Models underpinning Generative AI.

The cost of reinforcement learning, which makes sure things are working, is more than a small part of the sticker shock C-suite execs are experiencing with Generative AI.

Like everything, improvement may come to the process. Sources suggest retrieval augmented generation (RAG) is generally less labor intensive than data labeling. RAG retrieves info from an external database and provides it to the model “as-is.”

RAG is meant to address one of ChatGPT’s and Generative AI’s most disturbing traits: It can make a false claim with amazingly smug confidence. Humans have to keep a check on it.

But the build out of RAG requires some super smarts. As we have come to see, many of today’s headline AI acquisitions are as much about gaining personnel with advanced know-how as they are about gaining some software code or tool. This type of smarts comes at a high price, just as the world’s most powerful GPUs do.

This train of thought is impelled by a recent piece by Cade Metz for the New York Times. “The Secret Ingredient of ChatGPT Is Human Advice” considers reinforcement learning from human feedback, which is said to drive much of the development of artificial intelligence across the industry. “More than any other advance, it has transformed chatbots from a curiosity into mainstream technology,” Metz writes of human feedback aligned with Generative AI.

Metz’s capable piece discusses the role that expert humans are playing in making Generative AI work, with some implication that we should get non-experts involved, too. In response to the story, one Twitter wag suggested that Expert Systems are making a comeback. If so, guess we will have to make-do with more expert humans until “The Real AI” comes along! – Jack Vaughan