$NDVA $MU $SNDK $LITE $SKHY $000660 EXECUTIVE CONCLUSION
Semi Doped is a semiconductor podcast launched in 2026 and hosted by Austin Lyons and Vikram Sekar. Sekar, who leads this follow-up, is an electrical engineer with a Ph.D. from Texas A&M University, 15 years of semiconductor experience, more than 20 papers, more than 750 citations, and several patents. Guest Valentin “Val” Bercovici is Chief AI Officer at WEKA; prior roles include CTO positions at NetApp/SolidFire, work on Windows Shadowcopy, storage-standard leadership, and a founding role on the Cloud Native Computing Foundation governing board. The discussion is technically credible but commercially conflicted because WEKA sells the category being advocated; its benchmarks require independent validation.
The core thesis is directionally correct: inference is making memory, storage, networking, and state placement first-order determinants of throughput, latency, concurrency, utilization, power, and gross margin. The hierarchy spans accelerator SRAM, HBM, host DRAM, CXL memory, local NVMe, pod-scale context storage, and durable network storage. HBM remains the hot tier for active decode; DRAM remains staging and overflow. Flash retains reusable KV and restores it before decode, avoiding prefill and idle accelerator time. It is a state-reuse asset, not an HBM substitute.
The key insight is the gap between logical cacheability and physical cache hits. Reusable prefixes may be evicted, misrouted, invalidated by prompt or model changes, blocked by tenant isolation, or too slow to retrieve. Economics depend on hits after capacity, retention, routing, locality, security, version, and service-level constraints—more than on maximum context length.
The investment conclusion is constructive but selective. Highest-confidence beneficiaries are HBM, high-performance server DRAM, advanced packaging, low-latency networking, DPUs, enterprise SSD controllers, and inference-orchestration software. Flash-backed context memory is an incremental demand vector for enterprise NAND, especially high-endurance TLC and high-capacity QLC, but depends on actual prefix reuse, physical hit rates, concurrency, software integration, and latency objectives. HBF, large-scale CXL pooling, a broad return to true SLC, 10M-token production contexts, and widespread SaaS-neocloud M&A are options, not base cases.
INFERENCE-MEMORY ECONOMICS
Training requires parameters, optimizer states, gradients, activations, checkpoints, data, and collective communication. Inference removes optimizer states and gradients but adds variable request state, batching, concurrency, prefix reuse, preemption, routing, and per-user KV. Training emphasizes job completion; inference emphasizes cost per useful output, utilization, tail latency, time to first token, inter-token latency, power, and concurrency.
Prefill often has higher arithmetic intensity because prompt tokens are processed in parallel. Decode is sequential and repeatedly reads weights and accumulated attention state, making it frequently memory-bandwidth-bound at low batch sizes. These are workload regimes, not universal rules. Disaggregating prefill and decode can improve utilization but forces KV movement between systems, increasing the value of high-speed networking, direct accelerator I/O, topology-aware scheduling, and cache-locality management.
A simplified KV formula is 2 × layers × KV heads × head dimension × bytes per element × cached tokens. Grouped-query, multi-query, latent, sparse, and sliding-window attention, lower precision, and state-space components can sharply reduce requirements; parameter count alone is not a sufficient proxy.
WEKA’s Llama 3.3 70B example estimates 326 KB of FP16 KV per token: 32.6 GB at 100K tokens, 326 GB at 1M tokens, 3.26 TB for 100 concurrent 100K-token sessions, and 32.6 TB for 100 concurrent 1M-token sessions. These are illustrative, not universal; allocation varies by architecture, precision, page size, fragmentation, batching, padding, and runtime.
Persistent agent memory is broader than KV. Systems can summarize, externalize facts, checkpoint, store tool outputs, retrieve documents, or restart with compact prompts. KV is model-, tokenizer-, quantization-, and engine-specific and can be invalidated by upgrades. Long-running agents therefore need not cause linear, indefinite raw-KV growth.
NETWORKED FLASH: VALUE AND LIMITS
The claim that WEKA can be “faster than DRAM” is misleading unless narrowly defined as aggregate sequential bandwidth through a specific topology. WEKA disclosed approximately 300 GB/s of direct storage-to-GPU bandwidth per host in an 8-node, 72-drive configuration. An AMD MI325X provides 6 TB/s of HBM bandwidth; an 8-GPU platform provides 48 TB/s. WEKA’s figure is approximately 20x below 1 MI325X and 160x below the 8-GPU platform. Networked flash can outperform a single SSD, a constrained CPU-to-GPU path, or a poor local stack; it cannot match HBM or DRAM in latency, fine-grained random access, or bandwidth per compute device.
The architecture works because flash need not serve each attention read during decode. Reusable KV is pre-staged into DRAM or HBM. The relevant comparison is retrieval time versus avoided prefill. At 300 GB/s, 20 GB transfers in approximately 67 milliseconds before overhead. If recomputation takes several seconds of expensive accelerator time, retrieval is attractive despite being orders of magnitude slower than HBM.
Benefits are highest for large prefixes, expensive models, high reuse, shared context, and uncongested networks; they weaken for small prefixes, fast models, low reuse, congested fabrics, or strict tail latency. “DRAM-class” claims should specify whether they mean aggregate bandwidth, application throughput, time to first token, or load/store latency.
CACHEABILITY AND PRICING
Logical cacheability is potential reuse; physical hit rate is valid state actually found and used. Prefix matches are sensitive to system prompts, tools, timestamps, metadata order, whitespace, tokenizers, model versions, and agent branches. Capacity, eviction, routing, isolation, encryption, retention policy, and latency targets further reduce reuse.
Session stickiness preserves locality but creates hot spots; shared context improves balancing but adds movement. Routing must trade locality against queue depth and accelerator availability. Cache policy should reflect reuse probability, recomputation cost, size, age, model version, tenant priority, and service levels.
DeepSeek pricing shows the potential value. V4-Flash cached input is $0.0028 per 1M tokens versus $0.14 for a miss, a 50x discount. V4-Pro is $0.003625 versus $0.435, a 120x discount. Output remains $0.28 and $0.87 per 1M tokens, respectively. Cached-prefix economics can be extraordinary, but output and reasoning may still dominate total cost.
Pricing is not proof of hit rate or infrastructure cost. Providers can subsidize cached tokens, cross-subsidize from output, accept lower margins, or benefit from lower power and financing costs. The guest’s statement that “pricing is the ultimate benchmark” is valid for customer economics, not hardware efficiency or provider profitability. The claim that DeepSeek’s lowest pricing is available only from China or Mongolia and that US, European, or Singapore hosting is 87x more expensive was not corroborated in DeepSeek’s public pricing documentation.
DEEPSEEK V4 AND KV COMPRESSION
DeepSeek V4 validates architectural efficiency. V4-Pro has 1.6T total parameters and 49B activated; V4-Flash has 284B total and 13B activated. Both support 1M-token contexts. At 1M tokens, DeepSeek reports V4-Pro at 27% of V3.2’s single-token inference FLOPs and 10% of its KV cache; V4-Flash is at 10% of the FLOPs and 7% of the KV cache. These are model-relative reductions, not evidence of a universal 90% decline.
The architecture combines compressed sparse attention, heavily compressed attention, sparse selection, shared-key-value multi-query attention, sliding-window state, lower precision, and specialized layouts. On-disk KV avoids repeated prefill for shared prefixes; it does not show that SSDs can serve active decode at HBM-equivalent performance. Sliding-window attention preserves recent fine-grained context while older state is separately compressed and sparsely accessed; it does not simply discard unattended history.
TurboQuant was overstated. It reports quality-neutral KV quantization at 3.5 bits per channel and marginal degradation at 2.5 bits. Relative to FP16, theoretical raw reductions are approximately 4.6x and 6.4x before metadata, scales, alignment, residuals, workspace, and kernel overhead—not inherently 10x. Tail-quality loss, rare-token retrieval, long-context recall, dequantization cost, and implementation efficiency require validation.
The Jevons-paradox framing is plausible but unquantified. Lower bytes per token and cheaper cached input can unlock longer contexts, higher concurrency, more branches, and more agents. A 10x reduction in KV per token can be offset by 10x context or concurrency. The claim that every 100x unit reduction causes 10,000x aggregate growth lacks evidence. Demand remains constrained by adoption, reliability, energy, networking, regulation, context quality, and budgets. Efficiency can increase tokens while reducing memory per task and shifting demand among HBM, DRAM, and NAND.
NVIDIA CMX, STX, AND WEKA AMG
NVIDIA CMX validates a flash-backed context tier. NVIDIA defines G1 as GPU HBM, G2 as system memory, G3 as local SSD, G4 as shared durable storage, and CMX as a G3.5 pod-level flash tier. BlueField-4, Spectrum-X Ethernet, RDMA, DOCA Memos, NIXL, and Dynamo coordinate placement and pre-stage KV into G2 or G1. NVIDIA claims up to 5x higher sustained tokens per second and 5x better power efficiency than “traditional storage” for selected long-context and agentic workloads. The baseline is not HBM or DRAM.
CMX extends NVIDIA’s control into storage I/O, context software, metadata, scheduling, and rack topology, increasing content and lock-in. Independent storage vendors risk commoditization if NVIDIA controls protocols, DPUs, networking, and orchestration.
WEKA Augmented Memory Grid persists reusable KV in a “token warehouse,” decouples state from GPU hosts, reduces repeated prefills and session stickiness, and improves fleet balancing. WEKA reports approximately 300 GB/s per host, a 41x time-to-first-token improvement at 128K context, and a separate test with approximately 68% cache hits, 16K average context, INT4 Llama 3.1 405B, 8-way H100 systems, and a 72-drive WEKApod. These are vendor-generated and configuration-specific. Production evidence should include P50/P99 time to first token, inter-token latency, task completion, tokens per watt, utilization, contention, and total cost per successful task.
NAND TIERING, ENDURANCE, AND HBF
Inference creates heterogeneous NAND workloads. Read-heavy weights, embeddings, prefixes, and cold context favor QLC; frequently written KV, metadata, journals, and hot state favor TLC, pseudo-SLC, or SLC buffering. Placement should reflect temperature, reuse, write intensity, compressibility, and recomputation cost; a broad return to SLC is unlikely.
WEKA argues that global queue visibility and write striping across NVMe allow enterprise TLC to support KV without SLC. Distributed placement, overprovisioning, controller awareness, and parallelism help, but are not unique; competitors can use log-structured placement, flexible data placement, zoned namespaces, telemetry, wear leveling, and workload-aware admission.
Endurance must be modeled from sustained writes. A 10 PB array at 100 GB/s absorbs 8.64 PB per day, or 0.864 drive writes per day before write amplification. At 1 TB/s, it absorbs 86.4 PB per day, or 8.64 drive writes per day—unsuitable for ordinary QLC and challenging for many TLC configurations without large overprovisioning, write reduction, or replacement. The transcript’s 2:1 or 3:1 overprovisioning examples are not industry norms. Ephemeral KV can use weaker durability because lost cache is recomputable, but metadata requires protection and mass cache loss can trigger a recomputation storm.
Sandisk and SK hynix are pursuing HBF standardization. Sandisk targets HBM-comparable bandwidth, 8x-16x HBM capacity at similar cost, samples in 2H26, and initial AI-inference devices in early 2027. These are targets, not commercial proof.
HBF’s strongest early use case is likely read-dominant model weights, including inactive experts, less-used layers, and model variants; read-mostly prefixes, retrieval data, indexes, and multimodal assets are also plausible. Hot KV is less certain because of endurance, random access, page granularity, and latency. The claim that HBF will “probably have to use SLC” is speculative. Packaging is the larger risk: co-packaging improves bandwidth and power but raises thermal, yield, qualification, repairability, and replacement concerns. HBF remains medium-term optionality pending working silicon, sustained bandwidth, latency, endurance, power, thermal results, software support, and design wins.
CXL AND AMD MEXT
CXL and flash-backed context memory serve different domains. CXL provides coherent, load/store-accessible memory at hundreds-of-nanoseconds incremental latency; NVMe and networked flash are generally microsecond-scale or higher. CXL is relevant to host-memory expansion, pooling, databases, virtualization, metadata, and selected AI staging. CMX matters when petabyte capacity and cost per bit outweigh load/store latency.
Google’s 2023 research was skeptical of large-scale CXL pooling because of cost, complexity, latency, and limited utility, contradicting the unsupported claim that Google broadly reversed its position. CXL 4.0 nevertheless doubles transfer rates from 64 GT/s to 128 GT/s and adds bundled-port and reliability features. Near-term revenue is more likely from memory expansion, retimers, switches, and selected deployments than universal pooling.
AMD acquired MEXT in June 2026 for AI-driven predictive memory intended to make flash behave more like DRAM, expand usable capacity, and improve TCO. This is predictive tiering, not DRAM substitution: cold pages move to flash and are prefetched before reuse. Success depends on prediction accuracy and timing, drift, false-positive traffic, false-negative page faults, model overhead, and SSD endurance. Strategic value is plausible across EPYC, Instinct, ROCm, Pensando, Linux, and hypervisors, but financial relevance requires production evidence of lower DRAM at unchanged P99 latency and disclosed endurance.
HARDWARE, EDGE AI, AND SOFTWARE ECONOMICS
Inference hardware will remain heterogeneous because no architecture optimizes every mix of latency, throughput, context, batch size, model size, sparsity, precision, power, and cost. Google separates TPU 8t for training and embedding-heavy workloads from TPU 8i for serving and high-concurrency reasoning; TPU 8i has 3x more on-chip SRAM than its predecessor to retain more KV locally. This supports structural divergence between training and inference.
The transcript incorrectly calls NVIDIA’s December 2025 Groq transaction an acquisition. It was a non-exclusive inference-technology license; Groq’s founder, president, and additional employees joined NVIDIA while Groq remained independent and GroqCloud continued. Compiler and runtime expertise remains critical because scheduling, memory management, expert partitioning, quantization, communication overlap, and routing determine utilization.
The claim that 90%-99% of robotics inference will be local is plausible but unsupported. Latency, privacy, safety, connectivity, and resilience favor local processing; cloud systems remain useful for planning, fleet learning, updates, and exceptions. A hierarchy of local control and perception, nearby edge inference, and central frontier models is more probable.
AI-native software has material marginal inference cost. Token use can pressure gross margins, but routing, caching, smaller models, distillation, quantization, batching, speculative decoding, limits, and workflow redesign help. The correct metric is cost per successful task after retries, tools, verification, and guardrails.
Owning infrastructure helps only when demand is large, stable, predictable, and highly utilized. API expense is replaced by depreciation, financing, power, networking, facilities, operations, maintenance, and obsolescence. A hybrid structure is more likely: self-hosted or dedicated capacity for predictable volume, frontier APIs for difficult or changing tasks, and routing by quality, latency, and price.
SpaceX’s June 2026 agreement to acquire Cursor developer Anysphere for $60B is a real example of infrastructure, model, data, and application integration, but is idiosyncratic given SpaceX’s capital base, compute footprint, ambitions, and all-stock currency. It does not validate the prediction that “most SaaS companies and most neoclouds will have to merge.”
MARKET CONDITIONS AND INVESTMENT IMPLICATIONS
TrendForce forecasts conventional DRAM contract prices rising 13%-18% sequentially in 3Q26 and NAND prices rising 10%-15%. Micron’s fiscal 3Q26 revenue reached $41.46B, GAAP gross margin 84.6%, operating margin 80.4%, and operating cash flow $25.39B; fiscal 4Q26 guidance is approximately $50B of revenue and 86% gross margin. Micron also reports high-volume HBM4, PCIe Gen6 enterprise SSD production, and initial shipments of a 245 TB QLC SSD.
These figures validate scarcity but signal cycle risk. Gross margins above 80% are abnormal and incentivize capacity and customer optimization. HBM packaging, yield, qualification, power, and infrastructure can prolong scarcity, but annualizing peak margins overstates normalized earnings.
Positioning is crowded. SK hynix’s US-listed shares opened 14% above the offering price on July 10, 2026 after a $26.5B share sale. Korean shares remained approximately 630% above the prior year despite a 25% decline from a recent record; Micron had risen approximately 711%. Forward P/E multiples of approximately 5.8x for SK hynix and 7x for Micron are based on exceptional earnings. Low peak-cycle multiples do not imply low valuation risk.
HBM producers retain the strongest structural position because hot weights and active KV remain near compute. Risks are customer concentration, packaging constraints, pricing normalization, and lower bytes per token.
Server DRAM benefits from staging, host processing, overflow, SOCAMM, and CXL. Lower DRAM per rack need not mean lower aggregate bits if system counts and pools expand.
Enterprise NAND is the most underappreciated beneficiary through prefixes, weights, embeddings, multimodal assets, checkpoints, and agent state. Upside is strongest when enterprise mix and controller content rise, not commodity bits alone.
SSD controllers and firmware can capture value through placement, queue management, telemetry, endurance control, compression, encryption, and direct accelerator paths.
Networking, DPUs, NICs, switches, retimers, and optics are high-confidence beneficiaries because disaggregation and shared context increase bytes moved per token.
NVIDIA CMX strengthens the full-stack moat; open and multi-vendor architectures create opportunities for merchant alternatives.
AMD MEXT adds software optionality but remains execution-dependent.
CXL should be underwritten on actual memory-expansion and switching shipments, not universal pooling.
HBF offers optionality for Sandisk, SK hynix, bonding, controllers, and accelerator partners, but should receive little base-case revenue before design wins.
Storage software must prove physical hit rates, tail latency, endurance, security, observability, and framework integration.
SCENARIOS
BASE CASE—55%: Tokens and context concurrency grow rapidly, while efficiency offsets meaningful bytes per token. HBM remains critical; server DRAM grows but cedes selected warm state to flash; enterprise NAND benefits from weights, prefixes, agent data, and CMX. CXL expands gradually. HBF reaches samples and limited design wins but is not material before 2027-2028. Memory earnings stay strong but normalize from 2026 peaks.
BULL CASE—25%: Persistent coding, research, cybersecurity, and enterprise-agent swarms produce high physical reuse; large contexts become productive; CMX-style racks become standard; concurrency grows faster than compression reduces bytes per token; enterprise SSD mix upgrades; HBF wins accelerator support. HBM, DRAM, NAND, networking, DPUs, controllers, and orchestration expand as complements, sustaining above-normal margins.
BEAR CASE—20%: Agents summarize, retrieve, checkpoint, and compress rather than retain long raw contexts. Recurrent and state-space architectures reduce KV. Physical hits stay below logical cacheability because of branching, model changes, isolation, and invalidation. Context storage remains niche while supply expands, HBM yields improve, packaging constraints ease, and capex moderates. Memory pricing normalizes sharply; HBF and CXL are delayed.
CATALYSTS, RISKS, AND OPEN QUESTIONS
Catalysts include CMX/STX shipments, qualified storage partners, enterprise SSD mix, high-endurance drive demand, cache-read token shares, cache-write pricing, retained-context size, HBM and DRAM content per rack, HBF silicon and design wins, CXL switch deployments, and customer-reported cost per successful task.
Risks include lower physical hit rates; model compression; summarization and retrieval substituting for raw KV; cache invalidation from model or tokenizer changes; network congestion; SSD endurance and overprovisioning; security and isolation constraints; platform lock-in; HBF packaging and thermal failure; slow CXL software adoption; memory-capacity expansion; and peak-margin valuation.
Key questions:
Physical hit rate by workload, model, context length, tenant, and retention horizon.
Avoided prefill compute per retrieved byte and P50/P99 benefit.
Value from stateless serving and fleet balancing versus request acceleration.
Sustainable SSD writes, write amplification, service life, and replacement cost.
Fully loaded TCO across accelerators, memory, SSDs, networking, DPUs, software, power, cooling, financing, and operations.
Whether production workloads retain raw KV or shift to compressed state, retrieval, and external memory.
Whether CMX economics accrue mainly to NAND, controllers, storage software, or NVIDIA’s control plane.
FINAL ASSESSMENT
Inference infrastructure is becoming a distributed memory system in which state reuse, placement, networking, and scheduling determine accelerator economics. The investable thesis is hierarchy expansion, not flash replacement of DRAM: HBM remains active memory; DRAM remains host and staging memory; CXL provides coherent expansion; local and shared flash retain reusable state and avoid prefill; durable storage remains necessary; HBF may add a high-capacity near-accelerator tier.
The transcript’s qualitative direction is stronger than its quantitative claims. Assertions of 80%-90% agent-token share, 95% effective cache hits, 10M-token near-term contexts, universal SLC demand, flash “faster than DRAM,” broad CXL obsolescence, and inevitable SaaS-neocloud consolidation are insufficiently supported. Highest-confidence exposures remain HBM, advanced server DRAM, networking, DPUs, enterprise SSD controllers, and cross-layer inference software. Enterprise NAND has meaningful upside if context-memory systems move from demonstrations to production, but the market should be sized from retained and reused bytes, not total generated tokens.
Near-term fundamentals are exceptional, but the principal investment risk is whether current earnings and valuations already discount sustained scarcity, rapid agent adoption, high physical cache reuse, and disciplined supply. The architecture is favorable; risk-reward depends on entry price, product mix, and the durability of supernormal margins.
SOURCES
Primary source: Semi Doped transcript featuring Vikram Sekar and Valentin Bercovici. Supporting attributions: Semi Doped; WEKA; DeepSeek V4 paper and pricing; NVIDIA CMX/STX; AMD on MEXT; CXL Consortium; Sandisk/SK hynix HBF materials; Google Research on CXL; Google Cloud on TPU 8; Groq; TrendForce; Micron; and Reuters on SpaceX/Anysphere and SK hynix. The transcript appears recorded later in 2026; no exact date is embedded. WKA/Weta/Wacker, MEX/Next, Ver Rubin, “Grock with a Q,” Cerebrus, and Rocky were normalized to WEKA, MEXT, Vera Rubin, Groq, Cerebras, and RoCE.
