State of the AI Industry

Current stance (June 2026): The AI industry is best understood as a stack of six layers, each with different economics, different defensibility, and different implications for someone who works at the business layer. The bottom of the stack (silicon, cloud) is where value is captured today. The top (applications, business domains) is where value will accrue tomorrow — but only for those who understand that distribution and domain expertise matter more than model quality. This document is my attempt to build a mental model of the whole thing, layer by layer, so I can think clearly about where I sit and what comes next.

Why this exists

I’m a Data Scientist. My job is to help organizations make better decisions using data and, increasingly, AI. That puts me in a particular part of the AI ecosystem — the business domain layer — where the raw capabilities of models get translated into actual outcomes: better forecasts, smarter resource allocation, faster processes, more profitable strategies.

Over the last few years, I’ve watched the industry evolve at a pace that makes it hard to keep a coherent mental model. Every week there’s a new model, a new company, a new funding round, a new regulatory scare. The signal-to-noise ratio is terrible. This piece is my attempt to fix that for myself — to build a layered model of the AI industry that lets me:

1. Understand what’s happening at each layer
2. See how changes in one layer propagate to others
3. Decide where to pay attention and where to ignore the noise
4. Anticipate how my own layer will evolve

The framing I use is simple: six layers, from the physical foundation to the business surface. For each layer I ask: What’s the economics? Who wins? What’s changing? And what does this mean for someone in the business domain layer?

Layer 1: The Foundation — Silicon & Physical Infrastructure

What it is

This layer is the physical substrate of everything else. It includes:

• Raw materials — rare earth elements, silicon wafers, specialty chemicals, copper, lithium for the power infrastructure
• Chip design — the architecture of GPUs, TPUs, ASICs, and accelerators (NVIDIA, AMD, Intel, Broadcom, startups like Groq and Cerebras)
• Chip fabrication — the manufacturing process that turns designs into physical silicon (TSMC, Samsung, Intel Foundry)
• Energy and cooling — the power and thermal infrastructure required to run data centers at scale
• Data center construction — the physical buildings, networking, and logistics of standing up compute capacity

The economics

This is the highest-margin layer in the entire stack. NVIDIA operates at 75–80% gross margins. TSMC runs at 50–55%. The moats are structural: chip design takes years and billions of dollars; fabrication takes even longer and requires capital expenditure that most companies cannot stomach.

The numbers are staggering. Deloitte estimates that generative AI chips will approach $500 billion in revenue in 2026 — roughly half of all global chip sales. AMD CEO Lisa Su has raised her TAM estimate for AI accelerator chips to $1 trillion by 2030. And yet, for all that revenue, AI chips represent less than 0.2% of total chip volume. The economics are absurdly concentrated.

What’s changing

Several things worth watching:

The NVIDIA moat is real but not unassailable. NVIDIA dominates this layer with something like 80–90% of the AI training market. But the challengers are credible: AMD’s MI300X and MI325X now appear in about 25% of new deployments. Custom silicon from hyperscalers (AWS Trainium, Google TPU, Microsoft’s Maia) holds 15–20% of inference traffic. The question is not whether NVIDIA gets disrupted — it’s whether the disruption comes from above (hyperscaler vertical integration) or from the side (architectural shifts like analog or neuromorphic computing).

The fabrication bottleneck is geopolitical. TSMC makes the world’s most advanced chips in Taiwan. That’s a single point of failure that everyone in the industry pretends isn’t as scary as it is. The CHIPS Act, Intel’s foundry push, and Japan’s Rapidus project are all attempts to diversify. They will take a decade to change the picture materially.

Energy is the hidden ceiling. Training a single frontier model consumes as much electricity as a small town. Inference at scale (think: every ChatGPT query) adds orders of magnitude. Data center power demand is growing so fast that utilities in Northern Virginia, Ireland, and Singapore are struggling to keep up. This will constrain the industry long before model architecture or data quality does.

What it means for my layer

At the business domain level, the silicon layer matters because it determines the cost curve. Every improvement in chip efficiency or manufacturing yield drops the price of intelligence. For a Data Scientist, that means the models I can afford to run today will be cheaper and more capable next year, and the year after that. The trend is my friend — but it also means that any capability I build that depends on proprietary hardware access is a temporary advantage at best.

The energy constraint is the more interesting signal. If compute becomes the binding constraint on AI progress (and there are strong arguments that it already is), then access to compute becomes a competitive moat. The companies that can afford to train and deploy at scale will pull ahead. Everyone else consumes what the market offers.

Layer 2: The Platform — Cloud & Compute

What it is

If Layer 1 is the factory, Layer 2 is the wholesale distributor. This layer rents compute to everyone who needs it:

• Hyperscalers — AWS, Microsoft Azure, Google Cloud (the incumbents, each spending $50B+/year on AI capex)
• Neoclouds — CoreWeave, Lambda, Crusoe, Vultr (cloud-native AI infrastructure providers, often leaner and cheaper)
• Sovereign cloud providers — regional players in Europe, India, the Gulf, and Japan offering AI compute within local jurisdictions
• Data infrastructure — Snowflake, Databricks, MongoDB, Confluent, Pinecone (the data platforms that feed AI workloads)

The economics

Cloud compute margins run 60–65% for hyperscalers, though the AI segment is lower because GPU instances are capital-intensive and competition is fierce. The neoclouds undercut hyperscalers by 25–40% on GPU pricing but have thinner margins and less ecosystem lock-in.

The real economic story here is the capex arms race. Microsoft, Amazon, and Google are collectively spending well over $150B per year on AI infrastructure. This is not a bet — it’s a forced move. If any of them stops investing, they lose the next decade. The market is pricing this as rational, but the scale is historically unprecedented.

What’s changing

The neocloud squeeze. Hyperscalers dropped H100 on-demand prices by 40% in 2025. The neoclouds still undercut, but the margin gap is narrowing. The neocloud thesis was “cheaper GPUs without the lock-in.” As hyperscalers compete on price and add AI-specific services (Bedrock, Vertex AI, Azure AI), the neocloud advantage narrows to commodity access.

GPU fleet diversity. In 2024, everything ran on H100s. In 2026, deployments run three generations of GPU simultaneously — A100s for small models and dev workloads, H100s for mainstream inference, B200/B300s for frontier training and large-model inference. Platform teams now route workloads to the cheapest suitable GPU rather than running everything on one type.

Data gravity. The real moat for hyperscalers is not compute — it’s data. Companies store their data in S3, ADLS, or GCS. They run their ETL in the same cloud. Once the data lives in a cloud, moving AI workloads to a different cloud is expensive and painful. This lock-in is the most defensible position in this layer.

What it means for my layer

For a Data Scientist working in a business context, the cloud layer is where most of my practical decisions live: which platform to build on, how to manage costs, how to think about vendor lock-in.

The practical reality is that most organizations will pick one hyperscaler and stay there. The multicloud dream is dead for most companies — it adds complexity without enough benefit. The question is not “which cloud” but “how much of my AI budget goes to inference vs. training vs. data engineering.”

The neocloud disruption is relevant if I’m doing heavy training work. But for business-domain AI — where the work is more about retrieval, fine-tuning, and orchestration than training from scratch — the hyperscalers are fine. Their managed AI services are increasingly good enough that the build-vs-buy calculus tilts toward buying.

Layer 3: The Intelligence — Foundation Models

What it is

This is the layer everyone talks about: the models themselves. The frontier labs and the open-source ecosystem:

• Closed frontier — OpenAI (GPT-5, o-series), Anthropic (Claude 4), Google DeepMind (Gemini 2.5 Ultra)
• Open-weight — Meta (Llama 4), Mistral, Qwen, DeepSeek
• Niche leaders — Cohere (enterprise), Midjourney (image), ElevenLabs (voice), Suno (music)

The economics

Foundation model margins are 50–60% and falling fast. Training costs are enormous — $500M–$1B for a single frontier run — but inference costs are dropping 10x per year. The market is $30B+ and growing at 50% CAGR, but almost all the revenue goes to two companies: OpenAI ($10B+) and Anthropic ($3B+). Everyone else is either giving models away (Meta, Mistral) or struggling to monetize.

The structural problem for this layer is that model quality is commoditizing. The gap between GPT-5 and Llama 4 405B is real but small. The gap between GPT-5 and Claude 4 is negligible for most tasks. When models are roughly interchangeable, the pricing power shifts to the layer above.

What’s changing

The commoditization cascade. Open-source models now close the gap with proprietary ones within 6–12 months of each frontier release. This means that any application built on a proprietary model has a limited window before an open-source alternative offers similar quality at near-zero marginal cost. The implication: building your business on top of a specific model is a bad bet. Build on top of the model abstraction — the layer that lets you swap models as the market evolves.

The inference economics inversion. Training costs dominated the narrative in 2023–2024. In 2026, inference costs dominate. A single frontier model might cost $500M to train but $5B+ to serve over its lifetime. This is driving the shift toward smaller, more efficient models — and toward architectures that make inference cheaper (quantization, pruning, speculative decoding, mixture of experts).

Agentic reasoning as the new frontier. The next competitive vector for foundation models is not raw knowledge or language ability — it’s reasoning, tool use, and long-horizon task completion. This is where the frontier labs are competing in 2026: who can build a model that reliably executes multi-step tasks without hallucinating, forgetting context, or getting stuck in loops.

What it means for my layer

For a Data Scientist, the foundation model layer is where the raw material comes from. The headline is simple: the models are getting better and cheaper faster than most organizations can adapt.

The practical implications:

• Don’t build on a single model. Any system I build should be model-agnostic. The model that’s best today won’t be best in six months.
• Fine-tuning matters less than the hype suggests. For most business applications, retrieval-augmented generation (RAG) and prompt engineering outperform fine-tuning at a fraction of the cost. Fine-tuning is valuable only when you need to teach the model a specific behavior or domain that’s not in its training data.
• The real bottleneck is not model capability — it’s integration. The hardest part of deploying AI in a business context is not choosing the model. It’s connecting the model to the data, the workflows, the governance, and the people who need to use it.

Layer 4: The Stack — Middleware, Tooling & Serving

What it is

This is the layer that makes models usable in production. It’s less visible than the model layer but arguably more important for anyone actually shipping AI:

• Inference engines — vLLM, SGLang, TensorRT-LLM (the software that runs models efficiently on GPUs)
• Orchestration frameworks — LangChain/LangGraph, CrewAI, AutoGen, Bedrock Agents, Anthropic Agent SDK
• Vector databases — Pinecone, Qdrant, Weaviate, Milvus, pgvector
• Gateways — LiteLLM, Portkey, Kong AI Gateway (routing, cost management, failover)
• Observability — Langfuse, Weights & Biases, OpenTelemetry (tracing, evals, cost attribution)
• Prompt management — tools for versioning, testing, and deploying prompts
• Agent infrastructure — the new category that emerged in 2025–2026 for managing multi-agent systems

The economics

Middleware operates at 65–75% gross margins — higher than the model layer. The market is smaller (~$25B) but growing fast. Margins are high because these are software products with low marginal cost and significant switching costs once integrated.

The interesting dynamic: open-source dominates this layer (LangChain, vLLM, LiteLLM, Qdrant all have strong open-source offerings), but commercial wrappers grow on top. The classic open-core business model is alive and well here.

What’s changing

Orchestration is the new battleground. In 2024, orchestration meant “a LangChain pipeline with some tool calls.” In 2026, it means managing fleets of agents with persistent memory, tool registries, and inter-agent communication. The orchestration layer is where the most interesting infrastructure innovation is happening.

The gateway layer plays an outsized role. LiteLLM and Portkey have established the gateway as the single most valuable piece of AI infrastructure to add early. It handles routing, failover, cost tracking, guardrails, and semantic caching. If you only add one piece of AI infrastructure before building applications, add a gateway.

Context engineering has become a discipline. Managing what an agent “knows” — short-term memory (context window), working memory (active session state), and long-term memory (vector stores + graph databases) — is now a recognized engineering specialty. The tools are maturing fast.

What it means for my layer

For a Data Scientist shipping AI in a business context, the middleware layer is where most of my time goes. Not training models — connecting them.

The practical pattern that works: gateway + vector store + orchestration + observability. LiteLLM (or equivalent) for routing and cost control. Pinecone or pgvector for retrieval. LangGraph or a similar framework for the application logic. Langfuse for monitoring and evals. This stack handles 80% of business use cases.

The danger is over-investing in infrastructure before you have product-market fit. Most organizations I see spend too much time building elegant AI infrastructure and too little time putting something in front of users. The infra should be good enough, not perfect.

Layer 5: The Surface — Applications & Agents

What it is

This is where users interact with AI. The products that translate model capability into user value:

• Consumer AI products — ChatGPT, Claude, Gemini, Perplexity, Midjourney
• Developer tools — Cursor, Copilot, Replit Agent, Codeium
• Vertical SaaS — Harvey (legal), Glean (enterprise search), Jasper (marketing)
• Horizontal productivity — Notion AI, Grammarly, Otter.ai
• Agents — browser agents, research assistants, customer support agents, coding agents

The economics

Application margins are 60–70% but with a catch: defensibility is terrible. Most AI applications have thin moats because the underlying models are available to everyone. Features that are novel today become table stakes in six months. The durable advantages are distribution, UX, and data network effects.

The market is $100B+ and growing at 60% CAGR — the fastest-growth layer in the stack. But margins are under pressure as competition intensifies and users become price-sensitive.

What’s changing

The agent shift. 2026 is the year AI agents went mainstream. Not as a demo — as a product category. Browser agents, research assistants, customer support automation, coding agents — these are shipping with real revenue and real users. The shift from “chatbot that answers questions” to “agent that does things” is the most important product trend in the industry.

Distribution beats technology. The companies winning in applications are not the ones with the best AI. They’re the ones with the best distribution. OpenAI’s ChatGPT has 400M+ weekly active users. Microsoft ships Copilot to 365M Office 365 subscribers. Cursor won because it integrated into existing developer workflows, not because it had a better model.

Vertical SaaS is the most interesting bet. Horizontal AI apps (write better, search smarter) are hard to defend. Vertical AI apps (an AI that handles insurance claims, an AI that drafts legal briefs, an AI that manages clinical trials) benefit from domain-specific data, regulatory requirements, and workflow integration — real moats.

What it means for my layer

This is the layer most adjacent to mine. Applications are the bridge between model capability and business value. For a Data Scientist, the question is: do I build applications myself, or do I buy them?

For generic use cases (writing, coding, search), buy. The horizontal tools are good enough and getting better. For domain-specific use cases (your company’s data, your company’s workflows, your company’s customers), build — but build on top of the good-enough horizontal tools rather than starting from scratch.

The agent trend is the most important thing to watch. If agents fulfill even half of their promise, they will fundamentally change how businesses operate — and what a Data Scientist’s job looks like.

Layer 6: The Value — Business Domains & Decision Intelligence

What it is

This is my layer. The layer where AI meets a specific business context with specific data, specific workflows, specific regulations, and specific decisions to make:

• Decision intelligence — using AI to improve strategic and operational decisions
• Domain-specific AI — industry solutions for healthcare, finance, logistics, energy, manufacturing
• Embedded AI — AI woven into existing business processes (ERP, CRM, supply chain)
• AI-augmented analytics — the evolution of business intelligence toward natural-language querying, automated insight generation, and predictive recommendations
• Governance and risk — the frameworks, controls, and processes that make AI safe to use in regulated environments

The economics

This layer has the lowest margins in the stack — 30–40% — because it’s the most labor-intensive. Every deployment requires domain expertise, integration work, change management, and ongoing support. But it also has the highest durability: when an AI system is embedded in a business process, it’s expensive and risky to replace.

The market addressable here is enormous — $200B+ — but fragmented. It’s not won by a single product. It’s won organization by organization, industry by industry, use case by use case.

What’s changing

The expertise divide is widening. Organizations that know how to integrate AI into their decision processes are pulling ahead. Organizations that treat AI as a magic box or a cost center are falling behind. The difference is not access to models — it’s the organizational capability to ask the right questions, connect models to data, and act on the outputs.

The consulting layer is being reshaped. McKinsey, Accenture, and Deloitte have all launched AI practices. Their margin structure — sell human expertise at $300–$800/hour, augmented by AI — is being squeezed from both sides. Clients expect AI to reduce consulting costs. The consultancies that survive will be the ones that figure out how to deliver more value with fewer people, not the ones that try to protect their billable hour model.

AI-native competition from below. The startups in Layer 5 will eventually try to disintermediate Layer 6. If an AI application is good enough to handle a business process end-to-end, the domain layer above it shrinks. The question for someone in my layer is: which parts of what I do can be automated, and which parts require human judgment that cannot be delegated?

What it means for my layer

This is where I live, so the analysis here is personal.

Layer below (Applications + Infrastructure): The layer below me is getting more capable and cheaper. That’s good because it gives me better tools. It’s dangerous because it raises the question: if the tools are so good, why do they need me?

The answer, I think, is that tools are not solutions. A foundation model + a vector database + a gateway does not make a business decision. Someone still needs to:

• Understand the business context well enough to know which questions are worth asking
• Connect the AI to the right data
• Validate the outputs against domain knowledge
• Navigate the organizational politics of acting on AI-generated insights
• Handle the exceptions, edge cases, and novel situations that the AI hasn’t seen

Layer above (Strategy & Transformation): The layer above me is where the organizational impact lives. The companies that succeed with AI will be the ones that change how they make decisions, not just the ones that add AI to existing processes. This is the hardest part — it’s not technical. It’s cultural and structural.

My role in context: A Data Scientist in 2026 is less of a model builder and more of a translator. I translate between model capability and business value. I help the organization understand what AI can do, what it can’t do, and how to build the systems and processes that turn capability into outcomes. That’s more valuable than knowing how to fine-tune a transformer — and harder to automate.

Threading the layers together

The six-layer model is useful but it’s not the whole picture. The layers interact in ways that matter more than any individual layer’s dynamics.

Value flows down, then up. The initial value capture happened at the bottom — NVIDIA, TSMC, the hyperscalers. That’s where the money went first. But value is migrating upward as models commoditize and applications scale. The bet of this analysis is that the durable value will settle at the top — in business domains — because that’s where the switching costs are highest and the distribution moats are strongest.

The middle is getting squeezed. Foundation model providers are moving up into applications. Hyperscalers are moving down into custom silicon. The middleware layer (Layer 4) is under pressure from both directions. The application layer (Layer 5) is crowded and commoditizing. The most comfortable positions are the ends: silicon at the bottom (high defensibility) and business domains at the top (high switching costs).

Distribution is the meta-moat. Across every layer, the companies that win are the ones with distribution, not the ones with the best technology. OpenAI’s distribution (ChatGPT brand, enterprise sales). NVIDIA’s distribution (CUDA ecosystem, hardware vendor lock-in). Microsoft’s distribution (Office, Azure, GitHub). Technology advantages are temporary. Distribution advantages compound.

For a Data Scientist, the strategic question is: Which layer do I operate in, and how do I build defensibility there?

My answer: defensibility in the business domain layer comes from domain expertise, organizational trust, and the ability to connect AI capabilities to decision processes. Not from knowing the latest model architecture. Not from being able to train a LoRA. From understanding the business well enough to know what AI should do, and from having the organizational credibility to make it happen.

What comes next

This is version 1.0.0. It’s deliberately exploratory — a first attempt at a mental model, not a settled thesis. As the industry evolves and my understanding deepens, I expect to:

• Refine the layer definitions based on what the data shows
• Add specific company analyses and case studies
• Update the margin and market figures as better data becomes available
• Deepen the analysis of my own layer (business domains) — this is the section I’m least satisfied with and most interested in improving

The next update will focus on Layer 6 in detail, with concrete examples of how AI is changing decision-making in specific industries.

First published June 4, 2026. This is a living document — see the changelog for updates.