Progressive Gauge

Media and Research

Computing

Nvidia and Cloudflare Deal; NYT on Human Feedback Loop

By

Cloudflare Powers Hyper-local AI inference with Nvidia – The triumph that is Nvidia these days can’t be overstated – although the wolves on Wall St. have sometimes tried. Still, Nvidia is a hardware company. Ok, let’s say Nvidia is still arguably a hardware company. Its chops are considerable, but essentially it’s all about the GPU.

Nvidia is ready to take on the very top ranks in computing. But, to do so, it needs more feet on the street. So, it is on the trail of such, as seen in a steady stream of alliances, partners and showcase customers.

That’s a backdrop to this week’s announcement that Cloudflare Inc will deploy Nvidia GPUs along with Nvidia Ethernet switch ASICs  at the Internet’s edge. The purpose is to enable AI inferencing, which is the runtime task that follows AI model training.

“AI inference on a network is going to be the sweet spot for many businesses,” Matthew Prince, CEO and co-founder, Cloudflare, said in a company release concerning the Cloudflare/Nvidia deal. Cloudflare said NVIDIA GPUs will be available for inference tasks in over 100 cities (or hyper-localities) by the end of 2023, and “nearly everywhere Cloudflare’s network extends by the end of 2024.”

CloudFlare has found expansive use in Web application acceleration, and could help Nvidia in its efforts to capitalize on GPU technology’s use in the amber fields of next-wave generative AI applications.

With such alliances, all Nvidia has to do is keep punching out those GPUs – and development tools for model building.

***  ***  ***  ***

NYT on Human Feedback – The dirty little secret in the rise of machine learning was labeling. Labeling can be human-labor intensive, time-consuming and expensive. It harkens back to the days when ‘computer’ was a human job title, and filing index cards was a way of life.

Amazon’s Mechanical Turk – a crowdsourcing marketplace amusedly named after the 18th Century chess-playing machine “automaton” that was actually powered by a chess master hidden inside the apparatus — is still a very common way to label machine learning data.

Labeling doesn’t go away as Generative AI happens. As the world delves into what Generative AI is, it turns out that human labelers are a pretty significant part.

That was borne out by some of the research I did in the summer for “LLMs, generative AI loom large for MLOps practices” for SDxCentral.com. Sources for the story also discussed how “reinforcement learning through human feedback” was needed for the Large Language Models underpinning Generative AI.

The cost of reinforcement learning, which makes sure things are working, is more than a small part of the sticker shock C-suite execs are experiencing with Generative AI.

Like everything, improvement may come to the process. Sources suggest retrieval augmented generation (RAG) is generally less labor intensive than data labeling. RAG retrieves info from an external database and provides it to the model “as-is.”

RAG is meant to address one of ChatGPT’s and Generative AI’s most disturbing traits: It can make a false claim with amazingly smug confidence. Humans have to keep a check on it.

But the build out of RAG requires some super smarts.  As we have come to see, many of today’s headline AI acquisitions are as much about gaining personnel with advanced know-how as they are about gaining some software code or tool. This type of smarts comes at a high price, just as the world’s most powerful GPUs do.

This train of thought is impelled by a recent piece by Cade Metz for the New York Times. “The Secret Ingredient of ChatGPT Is Human Advice” considers reinforcement learning from human feedback, which is said to drive much of the development of artificial intelligence across the industry. “More than any other advance, it has transformed chatbots from a curiosity into mainstream technology,” Metz writes of human feedback aligned with Generative AI.

Metz’s capable piece discusses the role that expert humans are playing in making Generative AI work, with some implication that we should get non-experts involved, too.  In response to the story, one Twitter wag suggested that Expert Systems are making a comeback. If so, guess we will have to make-do with more expert humans until “The Real AI” comes along! – Jack Vaughan

 

Deconstructing and re-combining on the way to Turing: On Metcalfe

By

i. Taking a little time to review some writing from earlier this year… One especially memorable for me. For it was a welcome opportunity to interview networking pioneer Bob Metcalfe as he was named recipient for the 2022 ACM A.M. Turing Award for the co-invention, standardization and commercialization of Ethernet.

I won’t reconjure the full story but something I didn’t run in the final draft is something I’d like to share here. It’s a bit about his youth and early interest in technology. I find the pre-histories can be very telling, or just simply interesting. In the run-up to the typical story, the origin questions aren’t a fit. But let’s go back to those thrilling days of yesteryear.

Like ham radios, model trainsets were a path to engineering in Metcalfe’s youth in the 1950s. The model railroad arose in Metcalfe response to my origin question. But in a roundabout, and perhaps surprising way. A trainset was gifted and then built as a father-son bonding experience , and then was atomized – its relays, lights and toggle switches removed from casings and repurposed into something else entirely. As Metcalfe tells it:

My father was a gyroscope technician. And we built a railroad set on a four by eight piece of plywood that we painted green, and then I took the pieces of the control system for the railroad and I built what my eighth-grade teacher called a computer. It could add any number between one, two and three to any other number between one, two and three, and show the result by turning on a light between two and six. So those were the beginnings of my computer days. I never went back to the model railroad. Eventually I started programming computers.

It would be an overstatement to suggest this was the spark that launched Metcalfe’s life-long study in communications technologies and applications. But it has a place in his memory, and it discloses one of the engineer’s methods on the road to invention –  what G. Polya described as ‘decomposing and recombining’ important operations in the mind. Decomposed elements are combined in a new manner. Such possibly can send a lad or lass to a science fair exhibition, and on to the institute. It’s likely essential to most problem solving.

ii. Some more. Now, in the popular mind Ethernet has taken a decided backseat to Internet, as I noted in the VentureBeat piece. Still, Metcalfe rightly boasted that Ethernet deserves a lot of credit for proving incredibly resilient and potent. As he has said:

Ethernet was media-independent from the beginning. In retrospect, that was an important insight because Ethernet has been on every medium since.

That’s’ borne out today when you look at the ChatGPT cauldron. The AI asks for GPUs, but to scale them you have to connect them. Right now, Nvidia holds the winning hand with Infiniband. But its not alone. An Ethernet update running under the banner RoCE (RDMA over Converged Ethernet, pronounced “Rocky”) is out there punching today, taking some – but not  all – parts of the latest data center mission  — to move LLM vectors around in generative ML models.

Is it faster than Nvidia’s favored Infiniband? No. Is it a brewing de jure standard? Yes.

Worth recalling: De jure Ethernet coming out of the work of Metcalfe, co-inventor David Boggs, and others helped advance networked PCs. Another force was government-influenced second-sourcing of key semiconductors. Not exactly the fashion of the moment, but things change.

Some background on Me and Ethernet: When I was in grad school I did my thesis on Local Area Networks – it was a last minute arbitrary selection done at the suggestion of my wonderful classmate Richard Mack. I did the work of the technology assessment under the direction of my tech guru, brilliant beyond words Prof. Kirt Olsen. Cut my teeth! Catching time over more than a year, while working full-time night job at BU Mugar Library, I came up with the estimate that in five years LANs would be a $1B industry. I couldn’t believe it – but it happened. What an honor it was for me to interview Bob Metcalfe, the most pivotal and compelling individual figure in those developments! He has a great spirit still today, with enthusiasm and curiosity. Why I mention curiosity: He was a publisher and pundit in his time, too, you know – and asked me how what formerly was called the trade press was doing these days. – Jack Vaughan

Note: “How to Solve It” Cover by George Giusti/Typography by Edward Gorey – Book PDF Download

Also shown: Bob Metcalfe on a Zoom call with VentureBeat contributor Jack Vaughan.

 

 

Kreps, Dorsey, riff on ChatGPT

By

Stagg Field Nuclear Pile [fragment]

[Boston — May 2023] — “It’s a law that any conversation around technology has to come back to AI within five minutes.”

Well put, Jay Kreps, co-founder and CEO for real-time streaming juggernaut Confluent. Speaking at J.P. Morgan’s Boston Tech Investor event, Kreps knew this was coming. ChatGPT rules the news these days.

Given the daily pounding of 1,000 reporters’ laptops, given Nvidia’s vault into the highest clouds of valuation, it is no surprise that ChatGPT generative AI is the recurring topic. It will impede all other discussion, just as expected by tech stalwarts at J.P. Morgan’s and others’ tech events.

It’s the 600-lb. ChatBot in the room, and it is bigger than big.

Confluent chief on Chatbot interaction

Back in the nascent days of social media, the founders of Confluent, then working at LinkedIn, created a distributed commit log that stored streams of records. They called it Kafka and grew it out into a fuller data stream processing system. It’s intent is to bring to the broader enterprise real-time messaging capabilities akin to that of the Flash Boys of Wall Street.

The company is still in “spend a buck to make a buck” mode. For the quarter ending March 31, Confluent revenues increased 38% to $174.3M, while net jumped 35% to $152.6M. Customers include Dominos, Humana, Lowes, Michelin and others. In January it purchased would-be competitor, Immerok, a leading contributor to the Apache Flink stream processing project.

What’s the significance of real-time streaming in “the age of AI,” Kreps is asked at the Boston event. He says:

It’s really about how a company can take something like a large language model that has a very general model of the world and combine it with information about that company, and about customers, and be able to put those things together to do something for the business.

He gives an example: A large travel company wants to have an interactive chatbot for customers. Seems the barrier ChatGPT faces there for improvements is not so high. As Kreps said: “The chatbots were always pretty bad. It’s like interacting with like the stupidest person that you’ve ever talked to.”

Improvements needed for chatbots include a real-time view of all the information the company holds about customers and operations.

What do you need to make that work? Well, you need to have the real-time view of all the information about them, their flights, their bookings, their hotel, are they going to make their connection, etcetera. And you need a large language model which can take that information and answer arbitrary questions that the customer might ask. So the architecture for them is actually very simple. They need to put together this real time view of their customers, what’s happening, where the flights are, what’s delayed what’s going on. And then they need to be able to call out to a service for the generative AI stuff, feed it this data, feed it the questions from customers, and … integrate that into their service, which is very significant. This is a whole new way of interacting with their customers. And I think that that pattern is very generalizable.

Popping the question: Dorsey

For Jack Dorsey, the question “What about ChatGTP?” is raw meat. He melded SMS and the Web to create Twitter, and now with a nod to bitcoin and block chain has built Block, nee Square. The financial services and digital payments company posted revenue results for the three months ended April 1 that increased 26% to $4.99B, while net loss decreased a significant 92% to $16.8M. The good news was based on increased use of its Cash App product.

At the J.P. Morgan tech investor conference, Dorsey told the people, while hype obviously abounds, true progress rides on use cases.

There’s a ton of hype right now. And I think there’s a lot of companies being started that are going to fail because of that hype. I think the technology industry is very trendy, and very fashionable and jumps from one thing to the next, to the next, to the next. It wasn’t so long ago that we were only talking about Bored Apes and Crypto and NFTs and now we’re talking only about AI and how it’s going to kill us.

There’s always some truth in all these things. I just would caution any company that’s approaching it from a technology perspective, [to] instead use a use case perspective. What is the use case you’re trying to solve? And what technologies can you use to solve it more creatively?

THAT’S THE WAY IT IS — Clearly, panelists and podiumists are preparing to take on ChatGPT questions. At the same time, the clamor of the now will shift to prioritizing generative AI strategically within a host of technology initiatives. ChatGPT may be generalizable — but the proof will not appear overnight. The proof is in the business use case.

Reporter’s Notebook – At MIT Tech Review Future Compute 2023: Navigating the straits of semis

By

[May 9, 2023 ] – When the US last year announced new export rules on advanced chips,  the role of semiconductors in modern foreign affairs reached a new zenith. The chips have assumed the stature of oil in today’s geopolitics and depriving China of the chips now seems a strategic objective.

Unease has only grown with the appearance of the ChatGPT AI Large Language Model, which is a chip-hungry, power-guzzling presence ready to take over the world, to hear networks of experts and Cassandras tell it. Just as unsettling are Chinese maneuvers around Taiwan, a crucial center of global chip production.

Such activity formed a partial backdrop for the MIT Technology Review’s recent Future Compute 2023 conference at the Cambridge, Mass. Campus. Semiconductor issues were probed in a Q&A session featuring Chris Miller, Tufts University lecturer and author.

Miller said the semiconductor has taken on an outsized role in strategizing on China, and that the focus now is both on economics and defense.

”China spends as much money importing chips each year as importing oil,” he said. “You can’t understand the structure of the world economy without putting semiconductors at the center of your analysis.”

This is increasingly true for economic issues, Miller continued. Semiconductors that drive computers and embedded systems are top of mind when defense ministries and intelligence agencies think about future procurements.

“What they know is that over the past half century one of the key forces that’s transformed the way militaries fight has been computing power,” according to Miller, who traced the developments leading to the present predicament in “Chip War: The Fight for the World’s Most Critical Technology,” a recent noteworthy [Financial Times Book of the Year 2022] look at semiconductor industry history and its ever-shifting role in the larger body politic.

“Chip War” is described by a New York Times reviewer as something of a nonfiction thriller in which ‘pocket-protector men’ at Fairchild Semiconductor and Intel  tamed the raw transistor, fashioned the Integrated Circuit, outdid the Soviet Union, and left a war weary Europe in the dust as they formed what’s now Silicon Valley. Many of those developments bear review as governments’ and companies’ take on present complexities.

The complexities include more seemingly modest products than high-end processors, Miller indicated. Simpler chips that complement the hot processers grow in importance as well.

“The entire electronics supply chain is actually beginning to shift. It’s not only at the chip level, it’s also electronics assembly and simpler components,” Miller said, adding that a reduction in China’s level of server assembly has led to a major increase in Mexico’s market share in that field.

The also point emergence of new market dynamics as large companies take on design of their own chips, which could be spurred for a wider range of companies as US Chips Act R&D funding addresses the need for less expensive chip design processes.

A qubit for your thoughts

Infant quantum computing looms as an adjacent technology where geopolitical ambitions may play out.

China, the US the EU, and countries such as Australia, Singapore, and Canada now devote research monies to pursue such quantum efforts. They stir this new ground at the same time they test the limits of Moore’s Law – the perceived dead end for further large-scale silicon chip integration, which Tuft’s Miller cites as a fundamental challenge facing the chip industry.

However, quantum technology is still-raw technology – the quantum researchers on the main are still found toiling at the qubit level with lab rigs and signal scopes – that is, the quantum equivalent of the lone transistor work that preceded development of the Integrated Circuit.

A high-point of the Future Compute 2023 agenda for me was a visit to MIT’s Engineering Quantum Systems Group’s labs. Smart people are working hard on this frontier technology. And, with notable exceptions, there is knowledge sharing going on.

But, in a conference panel on quantum at the event, the impression emerged that quantum computing needed a large-scale working version of a quantum computer before the international competition for quantum computing would reach a less-sanguine stage akin to that the advanced CPU, GPU, NPU and network processing chips now experience.

For his part, at Future Compute, Chris Miller hesitated somewhat in responding to an audience question on quantum computing.

“I struggle to say anything that intelligent on quantum computing, both because I’m really not an expert in computing, but also because there’s a chip industry that I can study and I know how to talk about, whereas quantum computing is still a prospective industry,” he said. “We all hope it will materialize but it hasn’t materialized in a practical form.”

My take

Global chip wars must be viewed in the context of a real war underway in Ukraine. It has exposed the pivotal role of new technology in the exercise of war, as well as the vulnerability of the supply chains that feed modern commerce. It’s also pushed diplomacy to the sidelines, narrowing the opportunity for maneuver in the semiconductor straits.

“Mythical Man-Month” Author Frederick Brooks, at 91

By

[Dec 20, 2022] – Noting here the passing at 91 last month of Frederick Brooks, director of some of IBM’s most important mainframe-era programming projects. He was a key figure in establishing the idea that software projects should be intelligently engineered and organized.

He helped as much as anyone to move the mysterious art of tinkering with computer code toward a profession capable of repeatable results. “The Mythical Man-Month,” his 1975 distillation of years of development management, became a common reference work in many a developer’s desk library.

La Brea Tar Pits – Huntington Library.

Working at IBM in the 1950s and 1960s, and spearheading development of the vaunted IBM/360, Brooks gave a lie to notions that were bedrock in hardware-software projects, and came up with a few notable inventions as well.

Especially, he is credited with IBM’s decision to settle on an eight-bit byte. This allowed the systems to handle both text and numerals. Strange to think there was a time when machines were dedicated either to text handling or numerical calculation, but it was so!

He oversaw the development of systems that could be offered with an expandable range of processor and memory equipment at different price points, thus entering the development era of “platform” over “product.”

Brooks studied and found some surprising truths about complex software and projects – the most telling: That projects slow down at a greater rate when leaders add people as a project gets closer to completion.

He also saw the dangerous lure technology offers in the form of “The Silver Bullet” that promises a sudden tech- or organizational-style breakthrough.

With these and other observations, Brooks help build a philosophical underpinning for structured analysis, a school of thinking that held sway for software during an era of big projects marked especially by NASA’s Apollo program.

Ed Yourdon, Ivar Jacobsen, Tom DeMarco and others would take Brooks’ work into the 1990s. Like him, they realized it’s not just about “the code” – that the culture of the organization can play a more dominant role.

Brooks paved the path forward with emphasis on requirements gathering. But he foresaw the tarpit that beckoned with any search for the greatest schema perfection ahead of actually getting the project going.

He conveyed this without embracing the extreme that says “Fail Fast,” and he did it as always with a measure of humor. It’s right there in the title for one of “The Mythical Man-Month” chapters: “Plan to Throw One Away.”

Some of Brooks’ musings do echo another era. I don’t think we have Man Months anymore – mythical or other. Unquestionably too, team programmers have gained much more responsibility over the years, so Brooks’ emphasis on the manager wears thin [Heck, a whole era of development sprang from gritty JavaScript developers who found a way around obstacles their managers took for granted as their ‘lot in life.’]

But managers, at the end of the day, bear the greatest responsibility for the software project. Technology acumen is just a table stake. Their communications and organization skills must be stellar, as Brooks indicates when he writes:

The Tower of Babel was perhaps the first engineering fiasco, but it was not the last. Communication and its consequent, organization, are critical for success. The techniques of communication and organization demand from the manager much thought and as much experienced confidence as the software technology itself. – From The Mythical Man-Month.

Is software engineering really a profession? The question will continue to be asked, and Brooks work will likely ever be part of that discussion.

Coda: My days as a software project manager were brief – about a year. [After which my colleagues welcomed me back to editorial and told me they thought I’d been crazy ever to leave.] What I learned building web sites was that, no matter what you think your problem is, it is probably a project management problem. I owe that to Brooks. As expressed in his thoughtful and very often bemused writings, Brooks’ thinking on the topic informed my and many others’ efforts to ‘ship the darn thing’. – Jack Vaughan, 2022