<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Local AI &#8211; 4GQTV</title>
	<atom:link href="https://4gqtv.com/tag/local-ai/feed/" rel="self" type="application/rss+xml" />
	<link>https://4gqtv.com</link>
	<description>Entertainment, Movies, Games, Cosplay and Interviews</description>
	<lastBuildDate>Sat, 21 Nov 2026 13:55:03 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.5</generator>

<image>
	<url>https://4gqtv.com/wp-content/uploads/2026/02/4GQTVIcon-150x150.png</url>
	<title>Local AI &#8211; 4GQTV</title>
	<link>https://4gqtv.com</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Inkling 2 Mini Puts a Looped 26B Model in a 20GB Download, but Watch for Its Thinking Spirals</title>
		<link>https://4gqtv.com/inkling-2-mini-puts-a-looped-26b-model-in-a-20gb-download-but-watch-for-its-thinking-spirals/</link>
					<comments>https://4gqtv.com/inkling-2-mini-puts-a-looped-26b-model-in-a-20gb-download-but-watch-for-its-thinking-spirals/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Sat, 21 Nov 2026 13:55:03 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[AI Models]]></category>
		<category><![CDATA[Inkling 2 Mini]]></category>
		<category><![CDATA[llama.cpp]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Weights]]></category>
		<category><![CDATA[Thinking Machines Lab]]></category>
		<guid isPermaLink="false">https://4gqtv.com/inkling-2-mini-puts-a-looped-26b-model-in-a-20gb-download-but-watch-for-its-thinking-spirals/</guid>

					<description><![CDATA[Thinking Machines' Inkling 2 Mini is an Apache 2.0, 20GB looped MoE model for Macs and 24GB+ GPUs. Here's how looping works, what the benchmarks say, and the catches.]]></description>
										<content:encoded><![CDATA[<p>Last week I wrote that <a href="https://4gqtv.com/thinking-machines-opens-up-inkling-2-small-under-apache-2-0-but-this-594b-model-wont-fit-your-gaming-pc/">Inkling 2 Small</a> was a great open-weights release that almost nobody could run at home. Thinking Machines Lab clearly heard that complaint from a lot of people. On Friday, Nov. 20, the company released <a href="https://thinkingmachines.ai/news/inkling-2-mini/" target="_blank" rel="noopener">Inkling 2 Mini</a>, a model built from the start to fit on a laptop, and the weights are up on <a href="https://huggingface.co/thinkingmachines/Inkling-2-Mini" target="_blank" rel="noopener">Hugging Face</a> under the Apache 2.0 license.</p>
<p>The headline numbers are friendly to home hardware. It&#8217;s a Mixture-of-Experts model with 26.4 billion total parameters, 3.4 billion of them active per token, and the official checkpoint is about 20GB. It takes text and images as input, and Thinking Machines lists a context window of up to 1 million tokens. The interesting part is how it gets more out of those parameters. Inkling 2 Mini is a looped transformer, which means it runs part of its own network more than once for every token it generates.</p>
<h2>Why a looped model is a big deal for local AI</h2>
<p>Here&#8217;s how the loop works. According to the <a href="https://thinkingmachines.ai/model-card/inkling-2-mini/" target="_blank" rel="noopener">official model card</a>, the model has 24 layers. The first six and last six run once. The middle 12 run three times with the same shared weights. So every token gets 48 layer passes while the file on your SSD only stores 24 layers&#8217; worth of weights. Thinking Machines says each extra pass adds about 1 billion effective active parameters, which is how it arrives at 5.4 billion &#8220;weight activations&#8221; per token.</p>
<p>The company&#8217;s reasoning is that the middle of a transformer does most of the abstract thinking, while the outer layers mostly translate between tokens and internal representations. In its own experiments, looping the middle 12 layers beat looping the last 12 on 27 of 28 evaluations. <a href="https://huggingface.co/blog/inkling2mini" target="_blank" rel="noopener">Hugging Face&#8217;s launch write-up</a> adds a useful detail: each pass through the looped core starts from the same output of the first six layers, and the passes trade information through small learned &#8220;cross-pass messages&#8221; with gates that decide how much to keep.</p>
<figure class="wp-block-image size-large"><img fetchpriority="high" decoding="async" width="1424" height="1230" src="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-looped-architecture-diagram.jpg" alt="Diagram of the Inkling 2 Mini architecture: layers 0-5, three passes through shared core layers 6-17, layers 18-23, plus cross-pass messages, hybrid attention and a 128-expert MoE router" class="wp-image-4022" srcset="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-looped-architecture-diagram.jpg 1424w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-looped-architecture-diagram-300x259.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-looped-architecture-diagram-1024x884.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-looped-architecture-diagram-768x663.jpg 768w" sizes="(max-width: 1424px) 100vw, 1424px" /><figcaption>Inkling 2 Mini stores 24 layers but runs the middle 12 three times, for 48 layer visits per token. Image: <a href="https://huggingface.co/blog/inkling2mini" target="_blank" rel="noopener">Hugging Face</a></figcaption></figure>
<p>Why should a gamer with a decent PC care? Memory is the wall for local AI. A model that does more computation per token without getting bigger on disk or in VRAM is exactly the tradeoff home users want. If you remember the PS2 era, it&#8217;s a bit like developers squeezing more out of a fixed 32MB of RAM by being clever with the hardware instead of waiting for more of it. There&#8217;s a cost, though, and Thinking Machines is upfront about it: because the middle layers run three times, the compute per token is higher than the 3.4 billion active parameters suggest. Speed depends heavily on your runtime.</p>
<p>The weights were trained and ship natively in NVFP4, with the last four layers in MXFP8. That&#8217;s why the checkpoint stays near 20GB without a separate quantization step that loses quality. Officially supported local hardware is NVIDIA&#8217;s DGX Spark and Apple silicon Macs with 32GB or more of memory. For other GPUs, the model card says they&#8217;re &#8220;possible,&#8221; with 32GB or more of VRAM recommended.</p>
<p>Software support was ready on launch day, which isn&#8217;t something you can always count on. The model card lists SGLang, vLLM, NVIDIA Dynamo, TokenSpeed, Unsloth, llama.cpp, Llama App, Ollama and Exo Labs. <a href="https://ollama.com/library/inkling-2-mini" target="_blank" rel="noopener">Ollama&#8217;s library page</a> lists a 20GB build with the full 1M context. The <a href="https://huggingface.co/ggml-org/Inkling-2-Mini-GGUF" target="_blank" rel="noopener">ggml-org GGUF repo</a> has an IQ4_NL file at about 17.6GB and Q8_0 at about 27.8GB. <a href="https://huggingface.co/unsloth/Inkling-2-Mini-GGUF" target="_blank" rel="noopener">Unsloth&#8217;s quants</a> range from about 7.8GB for an extreme 1-bit version up to 22.9GB. By Unsloth&#8217;s own measurements, its 4-bit builds pick the same top token as its 8-bit reference about 98% of the time. That suggests a 24GB graphics card is a realistic target for a 4-bit quant, though long contexts will eat into that headroom fast. Thinking Machines also released a research variant, <a href="https://huggingface.co/thinkingmachines/TML-d24-varloop-research" target="_blank" rel="noopener">TML-d24-varloop-research</a>, that lets you pick how many loop passes to run. Fewer passes means faster output, and Hugging Face says it may also mean slightly lower quality.</p>
<p>On benchmarks, the company says Inkling 2 Mini leads every open-weights model under 100 billion total parameters on DeepSWE v1.1 (48.2%) and the Vals Index v2.2 (46.5%). In its comparison table against Gemma 4 31B, Muse Glimmer, Qwen3.8-27B and Nemotron 3.5 Super VL, it posts 79.2% on SWE-bench Verified (bash only) against 77.8% for Qwen3.8-27B, and 79.0% on BrowseComp. Hugging Face counts 32 wins out of 48 benchmarks in that comparison. These are vendor-reported numbers with varying evaluation setups, so treat them as a starting point, not a verdict.</p>
<figure class="wp-block-image size-large"><img decoding="async" width="1280" height="670" src="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-thinkyville-agents-demo.jpg" alt="Thinkyville, a pixel-art village built by Inkling 2 Mini where 20 villagers are each driven by the model running offline on one MacBook" class="wp-image-4023" srcset="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-thinkyville-agents-demo.jpg 1280w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-thinkyville-agents-demo-300x157.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-thinkyville-agents-demo-1024x536.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-mini-thinkyville-agents-demo-768x402.jpg 768w" sizes="(max-width: 1280px) 100vw, 1280px" /><figcaption>Thinkyville: all 20 villagers are run by Inkling 2 Mini on a single offline MacBook, per Thinking Machines. Image: <a href="https://thinkingmachines.ai/news/inkling-2-mini/" target="_blank" rel="noopener">Thinking Machines Lab</a></figcaption></figure>
<h2>The catches: knowledge gaps, thinking spirals and early tooling</h2>
<p>The thing I respect most about this launch is how plainly Thinking Machines lists the weak spots. Looping helps reasoning, but the company says it doesn&#8217;t let the weights store more facts. Knowledge mostly comes down to total parameter count, so Inkling 2 Mini&#8217;s factual recall is roughly what you&#8217;d expect from a model its size. The plan is to cover that gap with search and tools. That&#8217;s fine on a desk with Wi-Fi, but it matters for those &#8220;runs offline on a plane&#8221; demos.</p>
<p>Multi-turn chat isn&#8217;t its strength either. Thinking Machines reports 50.0% on MultiChallenge and 33.4% on ComplexConstraints v1.1, behind Qwen3.8-27B on both. The model card also warns about &#8220;repetitive reasoning,&#8221; where the model gets stuck re-checking its own work. It recommends caps on output tokens and agent turns, and says the model may take destructive actions like overwriting files without asking. Run it in a sandbox if you hand it a terminal.</p>
<p>Early users are seeing the same things. In the <a href="https://news.ycombinator.com/item?id=50581260" target="_blank" rel="noopener">Hacker News discussion</a>, which passed 1,100 points, one tester said a simple Python CSV task went great but a coin-identification prompt spiraled past 12,000 tokens. <a href="https://news.ycombinator.com/item?id=50583503" target="_blank" rel="noopener">Another commenter</a> running Unsloth&#8217;s Q4 build said a WordPress plugin prompt hit their 19,200-token thinking limit without finishing. They also said that once tools were allowed it wrote good code but still used the wrong hook name. These are anecdotes, not benchmarks, but they match the company&#8217;s own warnings closely. The model card also admits that support for looped models in local runtimes is new and speed &#8220;varies by runtime and hardware.&#8221; Expect a few weeks of llama.cpp and Ollama updates before performance settles.</p>
<p>There&#8217;s also a research angle that goes beyond local inference. A looped model can &#8220;think&#8221; more inside its layers without writing that reasoning out, and that makes it harder to oversee by reading its chain of thought. Thinking Machines tested this using the variable-loop model. It says the amount of work the model could do without a written chain of thought grew from 16 seconds to 37 seconds of equivalent human time as it went from one pass to three, while the chain of thought got shorter but stayed readable. That&#8217;s an honest, measured result, and it&#8217;s good to see the company publish it rather than skip the question. Whether it holds up for much bigger looped models is still an open question.</p>
<h2>Cody&#8217;s take</h2>
<p>This is the Inkling I was hoping for. Apache 2.0, a roughly 20GB footprint, day-one GGUF and Ollama builds, and a new architecture that open-source tools now have to support. That&#8217;s a fun weekend for anyone with a 24GB-plus GPU or a 32GB Mac. The Thinkyville demo, with 20 villagers all run by one model on a single offline MacBook, shows the kind of multi-agent tinkering a small, efficient model makes possible.</p>
<p>I&#8217;d keep expectations in check, though. The benchmark lead over Qwen3.8-27B is narrow on many tests and comes from Thinking Machines&#8217; own evaluations. The thinking spirals early testers describe are a real usability problem, and runtime support is still settling. If you already have a local coding setup you like, run Inkling 2 Mini next to it with a token cap and compare results on your own tasks before switching. If you&#8217;ve been waiting for a frontier-lab model that fits on a gaming PC, this one is worth the download.</p>
<p><em>Featured image: a flight simulator Thinking Machines says Inkling 2 Mini built offline on a MacBook Pro during a flight. Image: <a href="https://thinkingmachines.ai/news/inkling-2-mini/" target="_blank" rel="noopener">Thinking Machines Lab</a>.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/inkling-2-mini-puts-a-looped-26b-model-in-a-20gb-download-but-watch-for-its-thinking-spirals/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>SenseTime&#8217;s SenseNova Loop-9B Runs Its Middle Layers Twice to Punch Above Its Size, but Local Setup Needs a Forked llama.cpp</title>
		<link>https://4gqtv.com/sensetimes-sensenova-loop-9b-runs-its-middle-layers-twice-to-punch-above-its-size-but-local-setup-needs-a-forked-llama-cpp/</link>
					<comments>https://4gqtv.com/sensetimes-sensenova-loop-9b-runs-its-middle-layers-twice-to-punch-above-its-size-but-local-setup-needs-a-forked-llama-cpp/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Wed, 18 Nov 2026 20:50:43 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[AI Models]]></category>
		<category><![CDATA[Hugging Face]]></category>
		<category><![CDATA[llama.cpp]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open-weight models]]></category>
		<category><![CDATA[SenseTime]]></category>
		<guid isPermaLink="false">https://4gqtv.com/sensetimes-sensenova-loop-9b-runs-its-middle-layers-twice-to-punch-above-its-size-but-local-setup-needs-a-forked-llama-cpp/</guid>

					<description><![CDATA[SenseTime's Apache 2.0 SenseNova Loop-9B reuses its middle layers to act like a bigger model in about 10 to 15 GiB of VRAM. Here's what the self-reported benchmarks show and why local setup still takes work.]]></description>
										<content:encoded><![CDATA[<p>Most of the open-weight AI news lately has been about size. Bigger mixtures of experts, bigger context windows, bigger download folders. So it&#8217;s a nice change of pace that one of the more interesting releases this week goes the other way. SenseTime has put out <a href="https://www.sensetime.com/en/news/sensenova-loop-9b-release-20261118-1435">SenseNova Loop-9B</a>, a roughly 9-billion-parameter multimodal model that gets more &#8220;thinking&#8221; out of its small size by running the middle of its network twice.</p>
<p>The final release landed on <a href="https://huggingface.co/sensenova/SenseNova-Loop-9B">Hugging Face</a> on November 17 under the Apache 2.0 license. It replaces the Loop-9B Preview SenseTime shipped in late October. On November 18 the company added official <a href="https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF">GGUF</a> and 4-bit <a href="https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ">AWQ</a> builds, and those are the ones local AI folks will actually care about. The pitch is a model that fits on a mid-range gaming GPU and holds its own against models three times its size. Some of that pitch holds up. The rest still needs proving, and you&#8217;ll need some patience with the software side.</p>
<h2>How the loop works</h2>
<p>A normal transformer runs your prompt through each layer once and then it&#8217;s done. Loop-9B has 32 distinct decoder layers, but per the <a href="https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF">model card</a>, layers 4 through 27 form a shared block that runs twice in a row, with the output of the first pass feeding into the second. That works out to 56 layer computations per forward pass while storing only 32 layers&#8217; worth of weights. SenseTime describes the result as about 9B total parameters and about 14.6B &#8220;inference-equivalent&#8221; parameters.</p>
<p>Two other design choices sit on top of that. The decoder uses hybrid attention, with every group of four layers made up of three linear-attention layers and one full-attention layer. That&#8217;s a big part of why a 512K-token context window is practical on modest hardware. Inside the looped block, SenseTime uses what it calls two-stream manifold-constrained hyper-connections (mHC): two parallel residual streams, with learned mappings that decide how each layer reads from them, mixes them and writes back. The model takes text and images as input and outputs text.</p>
<figure class="wp-block-image size-large"><img decoding="async" width="1856" height="878" src="https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart.jpg" alt="Bar charts comparing SenseNova Loop-9B with its Preview, Qwen3.8-27B and other models on AutomationBench, Terminal-Bench 2.1, NL2Repo, AIDABench, BIRD, tau-cubed Banking and GAIA" class="wp-image-4002" srcset="https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart.jpg 1856w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart-300x142.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart-1024x484.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart-768x363.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-benchmark-chart-1536x727.jpg 1536w" sizes="(max-width: 1856px) 100vw, 1856px" /><figcaption>SenseTime&#8217;s own benchmark comparison for Loop-9B. These are company-reported numbers. Image: SenseTime / SenseNova</figcaption></figure>
<p>The model card includes a loop-scaling experiment that&#8217;s honestly the most interesting part of the release. SenseTime took the same 9B model and varied how many times the block loops, holding training data constant. Performance went up across all four tasks it tested, with the best results at three or four loops. AIDABench-QA and an office task suite peaked at three loops, while LiveCodeBench V6 and AIDABench data visualization peaked at four. The shipped model still defaults to two loops to keep inference speed reasonable. SenseTime says it wants to explore &#8220;Dynamic Loop&#8221; designs that pick the loop count per input, which would let easy prompts stay cheap and hard ones get more compute.</p>
<p>On the benchmark side, the jump from the Preview is big on paper. SenseTime reports AutomationBench going from 14.33 to 36.67, Terminal-Bench 2.1 from 40.45 to 59.55 and GAIA from 37.86 to 69.90. Against Qwen3.8-27B, the release ties on AutomationBench at 36.67, edges ahead on AIDABench (65.31 vs. 65.01) and trails on NL2Repo (41.36 vs. 44.60). The company&#8217;s own chart also shows the 27B Qwen ahead on Terminal-Bench 2.1, 63.40 to 59.55. So &#8220;competitive with a 27B model&#8221; is fair for some tasks, but it doesn&#8217;t win across the board.</p>
<p>The big caveat: every one of these numbers is self-reported. As <a href="https://ai-oriens.com/en/news/sensenova-loop-9b-release/">ORIENS</a> points out in its coverage, none of them have been checked by a third party yet. Agent benchmarks are especially sensitive to the harness, prompts and tool setup, so I&#8217;d wait for community runs before treating any of this as settled.</p>
<h2>Running it at home is not plug-and-play yet</h2>
<p>The memory story is the real draw. SenseTime says the 4-bit weights take about 4.9 GB, and it publishes estimated minimum GPU memory figures. With an 8-bit KV cache, it lists about 10 GiB at 64K context, about 12 GiB at 128K and about 15 GiB at 256K. With a 4-bit KV cache, those drop to roughly 9, 10 and 12 GiB. In plain terms, a 12GB or 16GB graphics card is the sweet spot if you want long context. The GGUF repo lists file sizes of 5.82 GB for Q4_K_M, 6.67 GB for Q5_K_M, 9.8 GB for Q8_0 and 18.42 GB for full BF16, plus a separate 0.92 GB vision projector file for image input.</p>
<p>Keep in mind that smaller memory doesn&#8217;t mean faster. Because the middle block runs twice, each token costs more compute than a plain 9B model would. That&#8217;s what the 14.6B inference-equivalent figure is telling you. You&#8217;re trading speed for capability per gigabyte, and on a VRAM-limited card that&#8217;s often a good trade. Just don&#8217;t expect small-model speeds.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="2000" height="1021" src="https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram.jpg" alt="Architecture diagram of SenseNova Loop-9B showing pre-loop layers, a looped block of shared middle layers, post-loop layers and the LM head, with a detail view of the two-stream mHC residual mixing" class="wp-image-4001" srcset="https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram.jpg 2000w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram-300x153.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram-1024x523.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram-768x392.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/sensenova-loop-9b-architecture-diagram-1536x784.jpg 1536w" sizes="auto, (max-width: 2000px) 100vw, 2000px" /><figcaption>The Loop-9B design: a shared block of middle layers runs more than once, with mHC mixing two residual streams. Image: SenseTime / SenseNova</figcaption></figure>
<p>Then there&#8217;s the software. Because the looped architecture is new, mainstream tools don&#8217;t support it out of the box yet. The GGUF instructions tell you to build <a href="https://github.com/OpenSenseNova/llama.cpp">SenseTime&#8217;s own fork of llama.cpp</a> rather than the upstream project. For server deployments, the card points to an open <a href="https://github.com/vllm-project/vllm/pull/64832">vLLM pull request</a> and an open <a href="https://github.com/sgl-project/sglang/pull/47281">SGLang pull request</a>. As of today, neither has been merged. Loading the full-precision weights in Transformers also requires <code>trust_remote_code=True</code>, which means running SenseTime&#8217;s custom model code. That&#8217;s normal for new architectures, but it&#8217;s worth knowing before you run it.</p>
<p>Access isn&#8217;t completely open either. The license is Apache 2.0, but the main BF16 repository is gated on Hugging Face, so you have to log in and accept the access conditions before downloading. When I checked the Hugging Face API, the GGUF and AWQ repos weren&#8217;t gated. SenseTime is also upfront about weak spots in its &#8220;Known Limitations&#8221; section: the model can miss requirements when it&#8217;s given lots of tool definitions or stacked constraints, its coding and agent performance &#8220;can be inconsistent,&#8221; and complex 3D or animation-heavy web pages may take two or three rounds of feedback. I appreciate a model card that says that out loud.</p>
<p>Loop-9B also isn&#8217;t the only small model trying this week. On November 17, OVHcloud <a href="https://corporate.ovhcloud.com/en/newsroom/news/ovh-olala-mini/">announced Olala mini 7A1B</a>, its first in-house LLM. It&#8217;s a 7B mixture-of-experts model with 1B active parameters across 256 experts per layer and hybrid Differential-TPA and Mamba-3 attention. OVH says it runs on GPUs with as little as 8 GB of memory, reaches 160 tokens per second on an RTX A4500 and does 50 tokens per second on a regular CPU. Its context window is a much shorter 32K tokens for now, with 128K promised for version 1.1 by the end of the year. OVH says the model will be on Hugging Face under Apache 2.0, and its formal launch is at the OVHcloud AI Summit on November 19. When I searched Hugging Face this afternoon, I couldn&#8217;t find the weights yet, so treat those speed claims as promises for now.</p>
<h2>Cody&#8217;s take</h2>
<p>I like this release more for the idea than the scoreboard. Looping layers to get more depth without more weights is a smart fit for the hardware most of us actually own, where VRAM runs out long before patience does. The loop-scaling data hints at a future where a small model can &#8220;think longer&#8221; on hard prompts instead of you downloading a bigger one. That&#8217;s a genuinely exciting direction for local AI.</p>
<p>But it&#8217;s early. The benchmarks are SenseTime&#8217;s own, the strongest claim (matching a 27B model) only holds on some tests, and right now you need a forked llama.cpp or unmerged vLLM and SGLang branches to run it. If you like tinkering and have a 12GB or 16GB card, the Q4_K_M GGUF is a fun weekend experiment. If you just want something that works in your usual app today, I&#8217;d wait for upstream support and some independent testing. Either way, I&#8217;m glad somebody is chasing efficiency instead of size.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/sensetimes-sensenova-loop-9b-runs-its-middle-layers-twice-to-punch-above-its-size-but-local-setup-needs-a-forked-llama-cpp/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Alibaba&#8217;s Qwen4-Max Is a 2.4T Flagship With Lower Prices, but Local AI Fans Can&#8217;t Download It</title>
		<link>https://4gqtv.com/alibabas-qwen4-max-is-a-2-4t-flagship-with-lower-prices-but-local-ai-fans-cant-download-it/</link>
					<comments>https://4gqtv.com/alibabas-qwen4-max-is-a-2-4t-flagship-with-lower-prices-but-local-ai-fans-cant-download-it/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Mon, 16 Nov 2026 20:52:14 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[Alibaba]]></category>
		<category><![CDATA[Large Language Models]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Weights]]></category>
		<category><![CDATA[Qwen]]></category>
		<guid isPermaLink="false">https://4gqtv.com/alibabas-qwen4-max-is-a-2-4t-flagship-with-lower-prices-but-local-ai-fans-cant-download-it/</guid>

					<description><![CDATA[Alibaba's Qwen4-Max launches with 2.4T parameters, a 1M-token context window and $1.50 per million input tokens, but it's API-only. Here's what's new, what the benchmarks say, and why the closed weights matter.]]></description>
										<content:encoded><![CDATA[<p>Alibaba&#8217;s Qwen team just shipped its biggest model yet, and the headline numbers are huge. <a href="https://www.alibabacloud.com/en/press-room/alibaba-unveils-qwen4-max" target="_blank" rel="noopener">Qwen4-Max</a> went generally available on Monday, November 16, with 2.4 trillion total parameters, native image and video understanding, a one-million-token context window and lower API prices than the model it replaces. The detail that matters most if you run models on your own hardware is a different one, though: you can&#8217;t download this one.</p>
<p>That&#8217;s a real change in direction for a model family that built much of its reputation on open weights. Qwen4-Max is hosted only, available through Alibaba&#8217;s QwenCloud API and Token Plan and inside Alibaba&#8217;s own apps. Alibaba says smaller Qwen4 models will get open weights &#8220;soon,&#8221; but it hasn&#8217;t given names, sizes, a license or a date.</p>
<p>The model has been around in some form for a few days. According to <a href="https://docs.qwencloud.com/changelog/models?model=qwen4-max-preview" target="_blank" rel="noopener">QwenCloud&#8217;s model changelog</a>, a Qwen4-Max-Preview opened on November 11 for Token Plan subscribers only, and the full <code>qwen4-max</code> model, with tool calling, structured outputs, explicit caching and chain-of-thought summaries, showed up on the 16th.</p>
<h2>What&#8217;s actually new under the hood</h2>
<p>Qwen4-Max is a sparse mixture-of-experts (MoE) model, meaning only a slice of the network fires for each token. Alibaba&#8217;s press release puts it at 2.4 trillion total parameters with 71 billion active per token. That&#8217;s the same total size as Qwen3.8-Max, but with 25% fewer active parameters, which is where a lot of the claimed efficiency comes from.</p>
<p>The <a href="https://qwen.ai/blog?id=qwen4-max" target="_blank" rel="noopener">Qwen team&#8217;s technical write-up</a>, as summarized by <a href="https://www.marktechpost.com/2026/11/16/alibaba-qwen4-max-release/" target="_blank" rel="noopener">MarkTechPost</a> and <a href="https://thenewstack.io/qwen4-max-alibaba-open-weights/" target="_blank" rel="noopener">The New Stack</a>, says the Qwen4 architecture changes three things. Attention now mixes Gated DeltaNet layers, which squeeze history into fixed-size states, with a new Qwen Sparse Attention layer that picks out the relevant chunks of long context. There&#8217;s a gated residual design with four parallel branches meant to keep training stable at this scale. And there&#8217;s an extra 328 billion parameters of &#8220;N-gram Embedding,&#8221; a lookup table for common local token patterns that adds capacity without adding much compute. Max output is listed at around 393K tokens, which is enormous for a single response.</p>
<p>If you&#8217;ve been following linear-attention research, the DeltaNet piece is the interesting part. It&#8217;s the same family of ideas that makes very long contexts cheaper to serve, and seeing it in a flagship from one of the biggest labs is a sign it has moved well beyond research papers.</p>
<figure class="wp-block-image size-large"><img decoding="async" src="https://4gqtv.com/wp-content/uploads/2026/11/qwen4-max-work360-benchmark-chart.jpg" alt="Alibaba Work360 benchmark radar chart comparing Qwen4-Max overall score of 52.69 with Qwen3.8-Max at 47.74 across industry domains"/><figcaption>Alibaba&#8217;s own Work360 benchmark shows Qwen4-Max at 52.69 versus 47.74 for Qwen3.8-Max. Image: Alibaba Group</figcaption></figure>
<h2>The benchmarks are mixed, and the best ones are Alibaba&#8217;s own</h2>
<p>Alibaba&#8217;s pitch is efficiency as much as raw smarts. On Work360, its in-house benchmark covering 721 specialized professional fields, Qwen4-Max scores 52.69 against 47.74 for Qwen3.8-Max. The company says it gets there with 24% to 35% fewer tokens depending on the reasoning-effort setting, and that fewer tokens plus lower prices cut the average cost per Work360 task from $4.43 to $1.67, roughly 62%. The press release also says the model ranked 6th on the Code Arena coding leaderboard.</p>
<p>The obvious caveat: Work360 is Alibaba&#8217;s test, run by Alibaba. In the broader comparison chart covered by MarkTechPost, Qwen4-Max takes first place on 4 of 16 benchmarks, including legal-agent work and Agents&#8217; Last Exam, and edges out rivals on the DeepSWE v1.1 coding test with 75.7. It falls well behind on others. On ProgramBench it scores 31.5, and on Terminal-Bench 4.0 it scores 48.5 while Moonshot&#8217;s Kimi K3.1 posts 67.2, per <a href="https://thenextweb.com/news/alibaba-launches-qwen4-max-flagship-ai-model-without-open-weights" target="_blank" rel="noopener">The Next Web</a>. None of these vendor-run scores had been independently reproduced at launch.</p>
<p>The closest thing to an outside view so far comes from Artificial Analysis, which, according to The New Stack, gave Qwen4-Max a 43 on its Intelligence Index. That&#8217;s 38th out of 81 models, a little above the median for its price bracket but well behind the top closed models from Anthropic, Google and OpenAI. The same testing found Qwen4-Max fairly chatty, generating about 127 million output tokens across the evaluation suite versus a median of 78.6 million, at an average cost of $3.27 per task. So the &#8220;uses fewer tokens&#8221; story holds against its own predecessor, not necessarily against the rest of the field.</p>
<p>Pricing, at least, is straightforward. Per The Next Web, The New Stack and MarkTechPost, the API starts at $1.50 per million input tokens and $4.50 per million output tokens, down from $2 and $6 for Qwen3.8-Max. Alibaba&#8217;s press release lists cached input at $0.075 per million tokens. Besides QwenCloud, the model is in Qwen Studio, QwenWork and the Qoder coding app, and The New Stack reports Vercel has added it to its AI Gateway.</p>
<p>Alibaba is also leaning on long-running agent demos. It says Qwen4-Max ran a 48-hour, four-round closed loop of molecular design and lab testing that turned up an experimentally validated chip coolant molecule, and that in an Alzheimer&#8217;s research test it made 454 tool calls to design 24 candidate molecules. Those are interesting showcases, but they&#8217;re the company&#8217;s own accounts, not peer-reviewed results.</p>
<figure class="wp-block-image size-large"><img decoding="async" src="https://4gqtv.com/wp-content/uploads/2026/11/alibaba-group-headquarters-hangzhou.jpg" alt="Alibaba Group headquarters buildings with a white lattice facade at the Xixi campus in Hangzhou, China"/><figcaption>Alibaba Group&#8217;s headquarters in Hangzhou, where the Qwen team is based. Photo: Thecraft / Wikimedia Commons, CC BY-SA 3.0</figcaption></figure>
<h2>Why the closed weights sting for local AI</h2>
<p>For the local AI crowd, Qwen has been a go-to family for years: lots of sizes, permissive licenses and fast community quantizations. And the open approach had just reached the top of the lineup. When Alibaba introduced Qwen3.8-Max in August, it promised to release the weights of a flagship Max model for the first time, and The New Stack notes it followed through on August 12 with Qwen3.8-2.4T-A95B, though under a custom license rather than Apache 2.0.</p>
<p>Qwen4-Max goes back to the older playbook, where Max models like Qwen3.6-Max-Preview and Qwen3.7-Max stayed proprietary. Practically, that means no GGUF builds, no running it on your own rig, and no fine-tuning. To be fair, a 2.4T model was never going to fit on a gaming PC or a single workstation card even with aggressive quantization, so most home users lose very little today. The bigger question is the smaller Qwen4 models Alibaba promised. If they bring the new attention and embedding design to sizes that do fit on consumer GPUs, under a real open license, that will matter far more to local AI than the Max model ever could.</p>
<p>There&#8217;s a safety angle in the announcement too. Alibaba says it filtered training data by intent, keeping defensive security material like vulnerability detection while dropping offensive attack content, and added post-training aimed at chemical, biological, radiological and nuclear risks. That&#8217;s welcome, but it&#8217;s also the kind of claim that&#8217;s hard to check from outside when the weights stay locked up.</p>
<h2>Cody&#8217;s take</h2>
<p>Qwen4-Max looks like a serious, cheaper flagship. If you already build on Qwen&#8217;s API, a price cut plus a million-token window is good news, and the architecture work is worth watching. But the strongest numbers come from Alibaba&#8217;s own benchmark, independent scoring puts it in the middle of the pack, and for anyone who chose Qwen because they could own the weights, this release has nothing for you yet. I&#8217;m much more interested in the smaller open Qwen4 models than in the Max launch. Until Alibaba puts a date, sizes and a license on those, I&#8217;d treat that promise as a maybe.</p>
<p><em>Featured image: Qwen4-Max Work360 benchmark chart from <a href="https://www.alibabacloud.com/en/press-room/alibaba-unveils-qwen4-max" target="_blank" rel="noopener">Alibaba&#8217;s official press release</a>, Image: Alibaba Group. Headquarters photo by Thecraft via <a href="https://commons.wikimedia.org/wiki/File:Alibaba_group_Headquarters.jpg" class='wp-img-bg-off' rel='mygallery' target="_blank" rel="noopener">Wikimedia Commons</a>, <a href="https://creativecommons.org/licenses/by-sa/3.0/" target="_blank" rel="noopener">CC BY-SA 3.0</a>.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/alibabas-qwen4-max-is-a-2-4t-flagship-with-lower-prices-but-local-ai-fans-cant-download-it/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Edge0-145B Brings a 145B AI Model to One GPU, but the &#8220;Runs on a Laptop&#8221; Claim Needs Fine Print</title>
		<link>https://4gqtv.com/edge0-145b-brings-a-145b-ai-model-to-one-gpu-but-the-runs-on-a-laptop-claim-needs-fine-print/</link>
					<comments>https://4gqtv.com/edge0-145b-brings-a-145b-ai-model-to-one-gpu-but-the-runs-on-a-laptop-claim-needs-fine-print/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Sun, 15 Nov 2026 13:54:50 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[Edge0]]></category>
		<category><![CDATA[Hugging Face]]></category>
		<category><![CDATA[LLM]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Mixture of experts]]></category>
		<category><![CDATA[Open-weight models]]></category>
		<guid isPermaLink="false">https://4gqtv.com/edge0-145b-brings-a-145b-ai-model-to-one-gpu-but-the-runs-on-a-laptop-claim-needs-fine-print/</guid>

					<description><![CDATA[Edge0's open-weight Edge0-145B uses a latent mixture-of-experts design and SSD streaming to run a 145B model on one consumer GPU. Here's what's real, what's internal testing, and why the laptop claim needs context.]]></description>
										<content:encoded><![CDATA[<p>Every few weeks somebody promises a &#8220;frontier&#8221; AI model that runs on your own computer, and every few weeks the fine print turns out to be doing a lot of heavy lifting. Edge0&#8217;s new <a href="https://edge0.ai/blog/introducing-edge0-145b-frontier-ai-capabilities-on-your-own-computer">Edge0-145B</a> is the most interesting version of that pitch I&#8217;ve seen in a while, because this time the weights are actually public, the inference engine is on GitHub, and you can check the claims yourself. Some of them hold up. Some of them need an asterisk the size of a GPU.</p>
<p>Edge0, whose launch post lists an AutoArk contact address for integrations, published the 145B model and a smaller Edge0-120B on Hugging Face on Nov. 11 and wrote them up in a long launch post on Nov. 12. The headline claim is bold: &#8220;the world&#8217;s first open-source on-device AI model with 145 billion parameters,&#8221; running on &#8220;a compatible laptop&#8221; at a reported 29GB of peak memory and more than 80 tokens per second. The launch also landed near the top of Hacker News, where the <a href="https://news.ycombinator.com/item?id=50475713">discussion thread</a> passed 350 points and 100 comments, much of it arguing about exactly what that 29GB number means.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1586" height="992" src="https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers.jpg" alt="Edge0 diagram explaining four design layers of Edge0-145B: shared-matrix sparse expert architecture, native on-device training, VRAM RAM and SSD loading tiers, and CPU plus GPU scheduling with speculative decoding" class="wp-image-3975" srcset="https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers.jpg 1586w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers-300x188.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers-1024x640.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers-768x480.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-architecture-layers-1536x961.jpg 1536w" sizes="auto, (max-width: 1586px) 100vw, 1586px" /><figcaption>Edge0&#8217;s own breakdown of how it fits a 145B model on personal hardware. Image: Edge0, via the <a href="https://edge0.ai/blog/introducing-edge0-145b-frontier-ai-capabilities-on-your-own-computer">Edge0 launch post</a></figcaption></figure>
<h2>How a 145B model squeezes onto one GPU</h2>
<p>The trick is a heavily sparse mixture-of-experts design. According to the <a href="https://huggingface.co/Edge0/Edge0-145B-q3k">official model card</a>, Edge0-145B has 48 layers with 512 experts each, but only 10 routed experts plus one shared expert fire for any given token. That works out to about 10.12 billion active parameters per token out of 145 billion total, with a 256K-token context window.</p>
<p>What makes it different from a straight quantization is what Edge0 calls a &#8220;latent MoE.&#8221; Instead of every expert running at the model&#8217;s full 2,560-wide hidden size, each token is projected down to a narrow latent width (1,792 on the 145B, 1,280 on the 120B), the experts do their work there, and the result is projected back up. The company says it built this on top of Alibaba&#8217;s Qwen3.8-Flash-Next, replaced the routed experts in all 48 layers, then post-trained the rebuilt model natively at 4 bits rather than compressing it after the fact. The payoff is that each expert becomes a small fixed-size chunk, about 1.63MB on the 145B, which the engine can shuffle between VRAM, system RAM and SSD.</p>
<p>That shuffling is the real engineering story here. The launch post describes a three-tier setup: frequently used &#8220;hot&#8221; experts live in GPU memory, an expert library of roughly 40GB sits in system RAM, and a huge n-gram embedding table (51.2 billion parameters in FP8, per the model card) stays on the SSD and gets read on demand. The CPU and GPU split the work, and speculative decoding with a small draft head spreads the cost of every weight read across more output tokens. The engine itself, edge0-strata, is a fork of the open-source <a href="https://github.com/Niko1221/Strata">Strata</a> project, and it ships alongside an <a href="https://github.com/Edge0-AI/Edge0">Apache 2.0 inference framework</a> with OpenAI- and Anthropic-compatible local endpoints plus an Edge0 Chat desktop app for Windows and macOS.</p>
<p>Honestly, this is the kind of systems work I love seeing in local AI. Most people running models at home hit the memory wall long before they run out of compute, and designing the architecture around that limit instead of shrinking a cloud model until it fits is a smart direction.</p>
<h2>The fine print on &#8220;runs on a laptop&#8221;</h2>
<p>Here&#8217;s where it gets complicated. The launch post says the model runs on &#8220;a compatible personal laptop, such as a MacBook Pro,&#8221; with 29GB peak memory, 80+ tokens per second decode and around 120W typical power. But it never names the exact laptop, chip or memory configuration behind those numbers, and the company&#8217;s own documentation tells a different story.</p>
<p>The <a href="https://github.com/Edge0-AI/Edge0/blob/main/edge0-infer/docs/edge0-145b.md">Edge0 145B reference doc</a> lists requirements of an NVIDIA GPU with CUDA 13 and at least 16GiB of VRAM plus 20GiB of system memory on Linux or Windows, or Apple Silicon with Metal on macOS. The macOS measurements, it says, were taken on an Apple M2 Ultra, which is a desktop chip found in the Mac Studio and Mac Pro, not a MacBook. The full speed table on the model card is mostly desktop hardware too: an RTX 5090 D 24GB hits 121.2 tokens per second in agent decoding with the Q3_K build, an RTX 4090 manages 91.7, and the M2 Ultra comes in at 57.0.</p>
<p>The &#8220;16GiB VRAM + 20GiB RAM&#8221; row that looks like a modest-PC result is the same RTX 5090 D with its VRAM and RAM artificially capped, and it drops to 44.9 tokens per second in agent decode and 335 tokens per second of 16K prefill. That&#8217;s still very usable, but it&#8217;s a long way from the 80+ figure in the headline. The card even notes that with expert residency turned on, the Q3_K build occupies about 23GiB of VRAM plus 31GiB of RAM on that card.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1280" height="720" src="https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-quality-vs-memory.jpg" alt="Edge0 chart plotting mean benchmark score against peak memory, showing Edge0-145B at 29GB and 72.8 versus Qwen3.8-Flash-Next at an estimated 70GB" class="wp-image-3976" srcset="https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-quality-vs-memory.jpg 1280w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-quality-vs-memory-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-quality-vs-memory-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/edge0-145b-quality-vs-memory-768x432.jpg 768w" sizes="auto, (max-width: 1280px) 100vw, 1280px" /><figcaption>Edge0&#8217;s quality-versus-memory chart. The memory figures for rival models are the company&#8217;s estimates, and its vertical axis does not start at zero. Image: Edge0, via the <a href="https://edge0.ai/blog/introducing-edge0-145b-frontier-ai-capabilities-on-your-own-computer">Edge0 launch post</a></figcaption></figure>
<p>Then there&#8217;s the download. &#8220;29GB&#8221; describes peak memory during inference, not the size of the model. When I added up the files in the Hugging Face repositories today, the <a href="https://huggingface.co/Edge0/Edge0-145B-q3k">Q3_K build</a> came to about 106GB and the <a href="https://huggingface.co/Edge0/Edge0-145B-NVFP4">NVFP4 build</a> to about 114GB, because that giant embedding table and the full expert pack have to live somewhere, and that somewhere is your SSD. The files also aren&#8217;t ordinary GGUF or safetensors checkpoints; they&#8217;re a custom pack that only Edge0&#8217;s own engine runs. Hugging Face&#8217;s own parameter counter reads just the skeleton file, which is why the site shows a number far below 145B. The independent site Traictory, which <a href="https://traictory.com/news/2026-11-14-edge0-145b-on-device">dug into the repositories</a> on Nov. 14, reached the same conclusion: the weights are real, but the laptop in the spec table is &#8220;a sentence, not a machine.&#8221;</p>
<p>The benchmarks deserve the same caution. Edge0 reports 64.6 on DeepSWE 1.1, 86.52 on Terminal-Bench 2.1, 90.4 on GPQA-Diamond and 80.6 on IFBench, and compares those against Qwen3.8-Flash-Next, DeepSeek-V4-Flash and Claude Opus 4.6. Those are strong numbers, but the post states plainly that the Edge0, Qwen, DeepSeek and Claude results all come from Edge0&#8217;s internal evaluations, and the full setup is promised in a technical report that isn&#8217;t out yet. By the company&#8217;s own table, the base Qwen3.8-Flash-Next still edges out Edge0-145B on HLE and GPQA-Diamond. In other words, the rebuilt model looks roughly on par with its parent while needing far less memory, which is impressive on its own without the &#8220;rivals frontier cloud models&#8221; framing.</p>
<p>Finally, the word &#8220;open-source.&#8221; The inference framework is Apache 2.0, but the model weights use the custom <a href="https://huggingface.co/Edge0/Edge0-145B-q3k">Edge0 Community Model License 1.0</a>, layered on top of Qwen&#8217;s own community license. It allows personal use, research, modification, non-commercial sharing and commercial use of the outputs, but selling or bundling the model, running a paid model service, or shipping a commercial coding or office assistant built on it requires a separate written license. That&#8217;s open weights, not open source in the OSI sense, and if you&#8217;re thinking about building a product on it, that distinction matters.</p>
<h2>Cody&#8217;s take</h2>
<p>I&#8217;m genuinely excited about what Edge0 is doing, even with all those caveats. A 145B-class model that Edge0 clocks at over 90 tokens per second of agent decoding on an RTX 4090, and stays usable when squeezed into 16GB of VRAM, is a real step forward for people who want capable local AI without renting cloud GPUs. The latent-MoE approach and the SSD-streaming engine are worth watching even if this exact model isn&#8217;t the one you end up using.</p>
<p>What I&#8217;d like to see next is simple: name the laptop, publish the technical report, and let independent testers reproduce those benchmark numbers. Until then, treat &#8220;runs on a laptop&#8221; as &#8220;runs on a very well-equipped machine with a fast SSD and about 106GB of free space,&#8221; and you&#8217;ll set your expectations right. If you&#8217;ve got a beefy NVIDIA card and the disk space, the <a href="https://github.com/Edge0-AI/Edge0/releases">Edge0 Chat app</a> is the easiest way to try it on Windows. Just know that, for now, it runs on Edge0&#8217;s own engine rather than dropping into the llama.cpp, Ollama or LM Studio setups most of us already use.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/edge0-145b-brings-a-145b-ai-model-to-one-gpu-but-the-runs-on-a-laptop-claim-needs-fine-print/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Thinking Machines Opens Up Inkling 2 Small Under Apache 2.0, but This 594B Model Won&#8217;t Fit Your Gaming PC</title>
		<link>https://4gqtv.com/thinking-machines-opens-up-inkling-2-small-under-apache-2-0-but-this-594b-model-wont-fit-your-gaming-pc/</link>
					<comments>https://4gqtv.com/thinking-machines-opens-up-inkling-2-small-under-apache-2-0-but-this-594b-model-wont-fit-your-gaming-pc/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Fri, 13 Nov 2026 13:55:51 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[AI Models]]></category>
		<category><![CDATA[Inkling 2 Small]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Source AI]]></category>
		<category><![CDATA[Open Weights]]></category>
		<category><![CDATA[Thinking Machines Lab]]></category>
		<guid isPermaLink="false">https://4gqtv.com/thinking-machines-opens-up-inkling-2-small-under-apache-2-0-but-this-594b-model-wont-fit-your-gaming-pc/</guid>

					<description><![CDATA[Thinking Machines released Inkling 2 Small's weights under Apache 2.0. Here's what the 594B MoE does well, where benchmarks show limits, and what hardware it really needs.]]></description>
										<content:encoded><![CDATA[<p>Thinking Machines Lab kept its promise. A week after it unveiled <a href="https://thinkingmachines.ai/news/inkling-2-small/" target="_blank" rel="noopener">Inkling 2 Small</a> as a closed model on its Tinker platform, the company put the full weights on <a href="https://huggingface.co/thinkingmachines/Inkling-2-Small" target="_blank" rel="noopener">Hugging Face</a> on November 12 under an Apache 2.0 license. That&#8217;s about as permissive as licenses get: you can download it, fine-tune it, ship it inside a commercial product and never ask anyone for permission.</p>
<p>The &#8220;Small&#8221; in the name is doing a lot of heavy lifting, though. This is a 594-billion-parameter Mixture-of-Experts model. Only 17 billion of those parameters fire for any given token, which keeps it fast for its size, but all 594 billion still have to live in memory somewhere. So before anyone starts clearing space on a gaming rig, let&#8217;s break down what actually shipped, what the benchmarks say, and who can realistically run it.</p>
<h2>What Thinking Machines actually released</h2>
<p>According to the <a href="https://thinkingmachines.ai/model-card/inkling-2-small/" target="_blank" rel="noopener">official model card</a>, Inkling 2 Small is a 48-layer decoder-only transformer. Each token gets routed to 8 of 384 experts, and attention alternates between local and global layers. It takes text and images as input (images go through a 292-million-parameter vision encoder) and outputs text, with a context window of up to 1 million tokens. One step backward from the first generation: audio input is gone. The original Inkling handled text, images and audio, and this one doesn&#8217;t.</p>
<p>The more unusual detail is the number format. Thinking Machines says the model was pretrained, midtrained and post-trained natively in NVFP4, Nvidia&#8217;s 4-bit floating-point format that arrived with Blackwell GPUs. Most labs train at higher precision and quantize down later. Here, 4-bit is the native format, and the company recommends the NVFP4 checkpoint for basically everything. An MXFP8 version is also posted, but the model card says its 1 to 2 point edge on evals isn&#8217;t meaningful in practice.</p>
<p>The launch also came with plenty of ecosystem support on day one. The weights work with SGLang, vLLM, TokenSpeed and Unsloth for local inference, and the model is available through APIs at Nvidia, Fireworks, Together, Baseten, Nebius, Modal and Cloudflare. If you want to kick the tires without owning a server rack, that&#8217;s the realistic path.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1280" height="720" src="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-small-linux-pc-emulator-demo.jpg" alt="Browser-based retro PC emulator built by Inkling 2 Small, showing a beige computer with a green-text Linux terminal on screen" class="wp-image-3962" srcset="https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-small-linux-pc-emulator-demo.jpg 1280w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-small-linux-pc-emulator-demo-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-small-linux-pc-emulator-demo-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/inkling-2-small-linux-pc-emulator-demo-768x432.jpg 768w" sizes="auto, (max-width: 1280px) 100vw, 1280px" /><figcaption>One of Thinking Machines&#8217; showcase demos: a PC emulator written in Python by Inkling 2 Small that boots a Linux kernel in the browser. Image: <a href="https://thinkingmachines.ai/news/inkling-2-small/" target="_blank" rel="noopener">Thinking Machines Lab</a></figcaption></figure>
<p>Thinking Machines&#8217; showcase demos lean hard on agentic coding with vision in the loop. The model built a city-builder in Three.js with procedurally generated buildings, textures and traffic, and wrote a PC emulator in Python that boots a real Linux kernel in the browser. Those are cherry-picked demos, of course, but they show what the company tuned for: long, multi-step work where the model checks its own visual output and keeps iterating.</p>
<h2>The benchmarks are strong, but read the whole table</h2>
<p>The headline number is a score of 31 on the Artificial Analysis Intelligence Index, a big jump over the first Inkling generation. ARC Prize reports 96.5% on ARC-AGI-1 and 81.7% on ARC-AGI-2, and Thinking Machines&#8217; own comparison table has it leading several vision benchmarks among the open-weight models it picked, including MMMU Pro and ScreenSpot Pro.</p>
<p>Here&#8217;s the part worth slowing down for. In the company&#8217;s own full benchmark table, Inkling 2 Small isn&#8217;t the top open model across the board. DeepSeek-V4.1-Flash posts a slightly higher Artificial Analysis score (32.6 versus 30.6 in the table) and beats it on several coding tests, like DeepSWE (77.0% versus 72.1%). Claude Haiku 5.5, a closed model, scores higher than both on that index. Inkling 2 Small is clearly in the top tier of open-weight &#8220;flash&#8221;-class models, but calling it the outright king would overstate the data, and to Thinking Machines&#8217; credit, the company publishes the rows where it loses.</p>
<p>The limitations section is refreshingly blunt, too. The company says the model isn&#8217;t the most succinct communicator, doesn&#8217;t always pick the right tools in codebases with lots of skills, overuses slashes in chat, and overreacts to mental-health and other safety-related questions. That last point shows up in the numbers: on Scale AI&#8217;s FORTRESS benchmark, it answers benign prompts 79.7% of the time, while most rivals in the table land in the mid-to-high 90s. In plain terms, it refuses harmless requests more often than its peers.</p>
<p>There&#8217;s also a candid alignment story in the release post. Thinking Machines writes that early checkpoints, when tasks got hard, sometimes fabricated results, ignored instructions and hid what they had done. Midway through the main reinforcement learning run, a monitor flagged the model working around a network block by reaching a third-party service through its search tools. The team says it paused the run, hardened its infrastructure, removed environments that needed web access and added real-time tool-call monitoring before continuing. It&#8217;s unsettling to read, but publishing it is the right call, and it&#8217;s exactly the kind of detail developers should know before handing an agent long, unsupervised jobs.</p>
<p>That candor fits the company&#8217;s broader approach. Back in July, Thinking Machines laid out <a href="https://thinkingmachines.ai/blog/a-safe-path-to-open-weights/" target="_blank" rel="noopener">&#8220;A Safe Path to Open Weights,&#8221;</a> a staged policy where models launch with closed weights first and go public after external safety testing. Inkling 2 Small is the first second-generation model to go through that pipeline, with a one-week gap between the Tinker launch on November 5 and the public weights on November 12.</p>
<h2>Can you run it locally?</h2>
<p>Not on a normal PC, and that&#8217;s the honest answer local AI fans need. The Hugging Face model card says the NVFP4 checkpoint needs a GPU cluster with at least 500GB of combined VRAM, such as two Nvidia B300s or four H200s. The MXFP8 version needs at least 800GB.</p>
<p>Unsloth, which had day-zero access, has posted <a href="https://huggingface.co/unsloth/Inkling-2-Small-GGUF" target="_blank" rel="noopener">GGUF quantizations</a> for llama.cpp-style setups that can spill into system RAM. Even its most aggressive 1-bit dynamic quant (UD-IQ1_S) weighs in around 172GB across its files, and the 4-bit UD-Q4_K_XL is roughly 354GB. Unsloth&#8217;s <a href="https://unsloth.ai/docs/models/inkling" target="_blank" rel="noopener">guide</a> says the 1-bit version keeps much of the full model&#8217;s accuracy, but by its own measurements you&#8217;re still trading away some quality. Realistically, that puts Inkling 2 Small in the territory of high-memory Mac Studios, workstations loaded with several hundred gigabytes of RAM, or rented cloud GPUs. For everyone with a single 16GB or 24GB graphics card, the API providers are the practical route.</p>
<p>The training scale explains the size. Thinking Machines says the first pretraining phase covered 8.4 trillion tokens in 68 hours on Nvidia GB300 NVL72 racks, and the main RL run used 2,048 GPUs for nine days across more than 17 million rollouts. This is a frontier-lab model that happens to have open weights, not a hobbyist model scaled up.</p>
<h2>Cody&#8217;s take</h2>
<p>I&#8217;m genuinely glad to see another serious lab ship a model this capable under Apache 2.0, with no usage caps, no regional exclusions and no &#8220;research only&#8221; asterisk. That matters for startups, researchers and anyone who wants to fine-tune a strong model on their own data without being locked into one provider. The fact that it&#8217;s designed to be a good base for customization, and that Thinking Machines published its weak spots and alignment scares instead of burying them, earns real points from me.</p>
<p>But &#8220;open&#8221; and &#8220;local&#8221; aren&#8217;t the same thing. For most people reading this, Inkling 2 Small will be something you hit through Together, Fireworks or Cloudflare, not something humming away under your desk. If you want an open model you can actually run on a gaming GPU tonight, smaller releases are still the better fit. If you&#8217;re building agents and want a permissively licensed model with strong vision and long-horizon coding chops, this one just jumped onto the shortlist. I&#8217;ll be watching for independent benchmarks over the next few weeks, because the real test is how it holds up outside its makers&#8217; own charts.</p>
<p><em>Featured image: a city simulator built by Inkling 2 Small in Three.js. Image: <a href="https://thinkingmachines.ai/news/inkling-2-small/" target="_blank" rel="noopener">Thinking Machines Lab</a>.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/thinking-machines-opens-up-inkling-2-small-under-apache-2-0-but-this-594b-model-wont-fit-your-gaming-pc/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Zyphra&#8217;s ZONOS3-Mini Puts Open-Source Voice Cloning in Your Browser, but the Big Numbers Are Its Own</title>
		<link>https://4gqtv.com/zyphras-zonos3-mini-puts-open-source-voice-cloning-in-your-browser-but-the-big-numbers-are-its-own/</link>
					<comments>https://4gqtv.com/zyphras-zonos3-mini-puts-open-source-voice-cloning-in-your-browser-but-the-big-numbers-are-its-own/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Wed, 11 Nov 2026 13:51:37 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Source AI]]></category>
		<category><![CDATA[Text-to-Speech]]></category>
		<category><![CDATA[Voice Cloning]]></category>
		<category><![CDATA[ZONOS3-Mini]]></category>
		<category><![CDATA[Zyphra]]></category>
		<guid isPermaLink="false">https://4gqtv.com/zyphras-zonos3-mini-puts-open-source-voice-cloning-in-your-browser-but-the-big-numbers-are-its-own/</guid>

					<description><![CDATA[Zyphra's Apache 2.0 ZONOS3-Mini TTS model clones voices with about 100M active parameters and runs locally via WebGPU, Vulkan and MLX. Here's what it does, and its caveats.]]></description>
										<content:encoded><![CDATA[<p>Most of the voice AI you hear online still lives on somebody else&#8217;s server. You type a line, it ships off to a cloud API, and a few hundred milliseconds later a voice comes back along with a usage bill. Zyphra is betting that doesn&#8217;t have to be the default anymore. On November 9 the San Francisco lab released <a href="https://www.zyphra.com/our-work/zonos3-mini" target="_blank" rel="noopener">ZONOS3-Mini</a>, an open text-to-speech model small enough to run in a browser tab, and it put the weights on <a href="https://huggingface.co/Zyphra/ZONOS3-Mini" target="_blank" rel="noopener">Hugging Face</a> under the Apache 2.0 license.</p>
<p>That license matters. Apache 2.0 is about as permissive as open weights get: you can build it into a commercial app, a game mod or a streaming tool without asking permission or paying royalties. Plenty of recent &#8220;open&#8221; releases come with regional bans, revenue caps or research-only clauses attached. This one doesn&#8217;t.</p>
<p>The headline number is the size. ZONOS3-Mini is a sparse mixture-of-experts model with 800 million parameters in total, but only about 100 million of them do work on any given step. According to the <a href="https://huggingface.co/Zyphra/ZONOS3-Mini" target="_blank" rel="noopener">model card</a>, it routes each token through 2 of 64 experts across an 11-layer backbone. For comparison, its predecessor ZONOS2 weighed in at 8 billion parameters, and Zyphra admits in its <a href="https://www.zyphra.com/our-work/zonos3-mini" target="_blank" rel="noopener">release post</a> that the bulk, plus a high-framerate audio codec, limited where people could actually deploy it.</p>
<h2>What ZONOS3-Mini actually does</h2>
<p>At its core it is a voice-cloning TTS engine. Feed it a reference clip of up to 60 seconds and it will speak new text in that voice, and the <a href="https://github.com/sgl-project/sglang-omni" target="_blank" rel="noopener">SGLang-Omni</a> serving path doesn&#8217;t even need a transcript of the reference audio. You can also skip cloning entirely and pick from 27 bundled default voices, or &#8220;design&#8221; a voice by nudging sliders for age, gender presentation, accent and vocal texture.</p>
<p>The control layer is where it gets interesting for anyone making games, videos or narration. Zyphra&#8217;s demo page walks through the same sentence delivered with warm relief, restrained anger and nervous apprehension, the same voice aged from roughly 25 to 75, and a single line where only a few words get whispered. That last trick is word-level steering: the model watches its own attention alignment while it streams audio so it knows exactly which words it&#8217;s saying, then applies the delivery change only there. There is also a separate edit model for fixing a phrase after the fact without regenerating the whole take.</p>
<p>Zyphra is careful in its own wording here, and that&#8217;s worth repeating. The release post calls the emotion and age settings &#8220;targets&#8221; and notes that the controls &#8220;do not guarantee a specific emotion.&#8221; Translation: it&#8217;s a steering wheel, not an autopilot, and you will still be doing retakes.</p>
<p>Language coverage is broad on paper. The model card lists 43 supported languages, from English, Spanish and Japanese to Hindi, Tagalog and Georgian. The fine print is that automatic text normalization (turning &#8220;4:30&#8221; or &#8220;$20&#8221; into speakable words) only covers nine of them: English, Mandarin, Japanese, Korean, German, Spanish, French, Italian and Portuguese. Everything else passes through as written, so numbers and abbreviations in other languages may need cleanup before you hit generate.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1600" height="900" src="https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart.jpg" alt="Zyphra chart plotting ZONOS3-Mini quality against time to first audio and peak throughput versus other open text-to-speech models" class="wp-image-3948" srcset="https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart.jpg 1600w, https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart-768x432.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/zonos3-mini-pareto-frontier-chart-1536x864.jpg 1536w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption>Zyphra&#8217;s own chart puts ZONOS3-Mini on the speed-versus-quality frontier, measured on one H100 GPU. The results are self-reported. Image: <a href="https://www.zyphra.com/our-work/zonos3-mini" target="_blank" rel="noopener">Zyphra</a></figcaption></figure>
<h2>Runs on phones, laptops and your own GPU</h2>
<p>The deployment story is the strongest part of the release. Zyphra shipped several runtimes on day one, all collected in its <a href="https://github.com/Zyphra/ZONOS3-Mini" target="_blank" rel="noopener">GitHub repository</a> (the runtime code itself is MIT-licensed):</p>
<ul>
<li>A WebGPU build that runs entirely in a supported browser, with no install and no server. The first visit downloads the model and the browser caches it.</li>
<li>A C++/ggml server and command-line tool that targets CPU, CUDA, Metal or Vulkan, with no Python required at runtime.</li>
<li>An MLX server for Apple Silicon Macs and a CUDA server for NVIDIA cards.</li>
<li>Day-zero support in SGLang-Omni, which exposes an OpenAI-compatible <code>/v1/audio/speech</code> endpoint with streaming.</li>
</ul>
<p>That Vulkan path is a quiet win for Linux users and anyone not running NVIDIA hardware, since Vulkan is the cross-vendor option that works on most modern GPUs. And the OpenAI-compatible endpoint means tools already wired up for cloud TTS can, in theory, be pointed at a local server instead with minimal changes.</p>
<p>Zyphra&#8217;s latency claims are specific: under 570 milliseconds to first audio in the browser on a Pixel 9 phone, and under 150 ms on an M3 MacBook Air. On a single NVIDIA H100 data-center GPU, the model card reports a median 80 ms to first audio with eight requests running at once, and peak throughput of 457 seconds of audio generated per second. Those are big numbers, but they are Zyphra&#8217;s own measurements. <a href="https://www.marktechpost.com/2026/11/10/zyphra-zonos3-mini-open-source-tts/" target="_blank" rel="noopener">MarkTechPost</a> flags the benchmark ranking as self-reported, <a href="https://neuralbrew.ai/en/articles/2026/11/zyphra-zonos3-mini-open-source-tts-browser/" target="_blank" rel="noopener">Neural Brew</a> attributes the speed figures to the company, and I haven&#8217;t seen independent testing yet.</p>
<p>The same goes for quality. Zyphra built a new benchmark for this launch, ZTTS2-Eval, covering read speech in 35 languages and real-world recordings in 20, and its capability index ranks ZONOS3-Mini first among open models, ahead of Fish Audio&#8217;s S2 Pro and MOSS TTS v1.5. A company releasing its own scoreboard alongside its own model is not unusual in AI, but it does mean you should treat the ranking as a claim to verify. Zyphra&#8217;s own chart is honest in one respect: on the quality axis, OmniVoice actually scores a bit higher. ZONOS3-Mini&#8217;s pitch is that it&#8217;s on the efficiency frontier, offering near-top quality at a fraction of the latency, not that it sounds best outright.</p>
<p>Under the hood, the speedup comes from a few places. Zyphra trained its own audio codec, ZAC-1, which runs at 15 frames per second instead of the 86 Hz codec ZONOS2 used, so the model has far fewer audio tokens to predict. The backbone borrows ideas from recent language models, mixing Gated DeltaNet layers with global attention in a 3-to-1 ratio, and the whole thing was trained on 112 NVIDIA H100s in two and a half days. A single generation can run up to 10 minutes of audio at 44.1 kHz.</p>
<p>Early interest is real but still modest: the Hugging Face page showed about 4,170 downloads and 112 likes when I checked on Wednesday morning, less than two days after launch.</p>
<h2>Cody&#8217;s take</h2>
<p>I&#8217;ve been waiting for local TTS to get here. When I was a kid wearing out my PS2, voiced dialogue felt like a luxury, and now a 100-million-active-parameter model promises expressive, cloneable voices on a phone with no cloud bill. For indie devs who need placeholder dialogue, YouTubers who want narration without a subscription, or streamers building local overlays and alert voices, an Apache-licensed engine that runs on your own hardware is a genuinely useful tool.</p>
<p>But I&#8217;d be doing you a disservice if I skipped the uncomfortable part. Voice cloning from a short clip, with no gatekeeper, is exactly the capability scammers and impersonators want. The model card I read doesn&#8217;t describe built-in watermarking or consent checks, which means responsibility lands on whoever builds with it. If you ship ZONOS3-Mini in an app, build in consent for cloned voices and make it clear when a voice is synthetic. And please don&#8217;t clone anyone who hasn&#8217;t said yes.</p>
<p>The other caveat is simply time. Every number in this story comes from Zyphra, and the real test is what happens when people run it on mid-range laptops, older phones and non-NVIDIA cards over the next few weeks. If the browser demo holds up on everyday hardware, this could be one of the more practical open AI releases of the fall. If you want to judge for yourself, the <a href="https://zyphra.com/zonos3-mini/localplayground/lite" target="_blank" rel="noopener">Lite browser demo</a> is the fastest way to hear it, as long as your browser supports WebGPU.</p>
<p><em>Featured image: Image: <a href="https://www.zyphra.com/our-work/zonos3-mini" target="_blank" rel="noopener">Zyphra</a>.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/zyphras-zonos3-mini-puts-open-source-voice-cloning-in-your-browser-but-the-big-numbers-are-its-own/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>A Two-Person Lab Trained an Open Image Model for $21K. Here&#8217;s What P-JiT Can and Can&#8217;t Do</title>
		<link>https://4gqtv.com/a-two-person-lab-trained-an-open-image-model-for-21k-heres-what-p-jit-can-and-cant-do/</link>
					<comments>https://4gqtv.com/a-two-person-lab-trained-an-open-image-model-for-21k-heres-what-p-jit-can-and-cant-do/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Mon, 09 Nov 2026 21:05:47 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[AI Image Generation]]></category>
		<category><![CDATA[Diffusion Models]]></category>
		<category><![CDATA[Linum]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Source AI]]></category>
		<category><![CDATA[Text-to-Image]]></category>
		<guid isPermaLink="false">https://4gqtv.com/a-two-person-lab-trained-an-open-image-model-for-21k-heres-what-p-jit-can-and-cant-do/</guid>

					<description><![CDATA[Linum's P-JiT 21K is a 2B open-weight, Apache 2.0 text-to-image model trained for about $21,000. Here's how it works, what it needs to run, and its real limits.]]></description>
										<content:encoded><![CDATA[<p>Two people, 64 Nvidia H100 GPUs and roughly $21,000 in compute. That&#8217;s what it took Linum, a two-person research lab run by brothers Sahil and Manu Chopra, to train <strong>P-JiT 21K</strong>, a 2-billion-parameter text-to-image model they&#8217;ve now put out with open weights under the Apache 2.0 license. In a field where the headline numbers usually come with a lot more zeros, that price tag is the real story here.</p>
<p>The weights went up on <a href="https://huggingface.co/Linum-AI/pjit-21k" target="_blank" rel="noopener">Hugging Face</a> on November 3, the lab&#8217;s <a href="https://www.linum.ai/field-notes/pjit-21k" target="_blank" rel="noopener">Field Notes write-up</a> followed on November 4, and the release is now picking up wider coverage, including a <a href="https://gigazine.net/gsc_news/en/20261109-pjit-21k/" target="_blank" rel="noopener">GIGAZINE piece</a> dated November 9. The inference code is on <a href="https://github.com/Linum-AI/pjit-21k" target="_blank" rel="noopener">GitHub</a>. Before anyone gets too excited, Linum is upfront that this is a research checkpoint, not a polished Stable Diffusion or FLUX competitor. But it&#8217;s an interesting one, and the reasons why matter for anyone who cares about AI that small teams and hobbyists can actually build and run.</p>
<h2>What makes P-JiT different</h2>
<p>Most modern image generators don&#8217;t work on pixels directly. They lean on a separate compression model, usually a variational autoencoder (VAE), that squeezes an image into a smaller &#8220;latent&#8221; representation. The diffusion model does its work in that latent space, and the VAE decodes the result back into a picture at the end. It&#8217;s efficient, but it means the image model is stuck with whatever compression the VAE was built for.</p>
<p>Linum&#8217;s bet is to throw the VAE out entirely. P-JiT (short for Pyramid-JiT) is a pixel-space diffusion transformer: it chops an image into 32×32-pixel patches and learns its own compression as part of generating the image. According to the lab, that lets it represent a 512×512 image with the same number of tokens its previous model, Linum v2, needed for a 256×256 one. Since attention cost grows with the square of the token count, fewer tokens is a big deal. Linum says the architecture uses a 4× smaller attention context and converged in about 11× fewer training samples than Linum v2.</p>
<p>The &#8220;pyramid&#8221; part comes from extra readout heads placed partway through the network that predict quarter- and half-resolution versions of the final image during training. The idea, per Linum, is to give the early layers a stronger learning signal so the model gets to good results faster. Those helper heads are only used in training and aren&#8217;t part of the released model, which works out to about 2.18 billion parameters. Text understanding comes from a frozen Qwen3.5-4B language model acting as the text encoder.</p>
<figure class="wp-block-image size-large"><img decoding="async" src="https://4gqtv.com/wp-content/uploads/2026/11/linum-pjit-21k-samples.jpg" alt="Five sample images generated by Linum's P-JiT 21K model: an elderly woman in a cream headscarf, a man blowing out birthday candles, a storybook bear sharing honey with a rabbit, a giant panda in bamboo, and an alpine wildflower meadow"/><figcaption>Five official samples from P-JiT 21K, shown at the model&#8217;s native resolutions. Image: Linum (from the pjit-21k GitHub repository, Apache 2.0)</figcaption></figure>
<p>Where the $21K actually went is laid out in unusual detail. Pretraining took 8,139 H100-hours on 64 GPUs: 610 million samples at 512px, then 50 million at 1024px. Post-training added another 276 H100-hours. At the $2.50-per-hour rate Linum used for its math, that&#8217;s $20,348 plus $690, or $21,038 total, restarts included. That&#8217;s a sensible way to report it, though it&#8217;s worth remembering it&#8217;s a compute bill. It doesn&#8217;t count the salaries, the data pipeline, or the earlier experiments that got the architecture to this point.</p>
<p>The sample gallery is genuinely good for a model this size. Portraits, food, animals, landscapes and storybook-style illustrations all look clean and natural in the official examples. As always with a lab&#8217;s own picks, those are the best results, not the average one.</p>
<h2>The fine print: no text, short prompts and NVIDIA-only</h2>
<p>This is where the grounded part comes in, because the limitations are real and Linum doesn&#8217;t hide them.</p>
<p>First, <strong>P-JiT can&#8217;t render text</strong>. At all. The team deliberately filtered infographics, slides, posters and book pages out of the training data because they don&#8217;t help its long-term goal of animation tools. If you ask for a sign, logo or title card, the included prompt helper returns an error instead of trying. If you were hoping for a free poster or thumbnail generator, this isn&#8217;t it.</p>
<p>Second, it struggles with the stuff small models always struggle with. Linum lists full-body shots, scenes with many objects (six cars, crowds), multiple people interacting, and world knowledge like the right number of guitar strings or the dots on a die. The write-up even shows a sprinter with mangled legs on the starting blocks as an example failure. The team thinks a bigger model of around 7B parameters and more post-training would help.</p>
<p>Third, prompting takes some work. The model was trained on dense, 50- to 80-word captions, and short prompts give noticeably worse images. By default, the included script sends your prompt to Anthropic&#8217;s Claude Opus 5.5 through OpenRouter to rewrite it into a long caption, which needs an API key and a paid cloud call. You can skip that with a flag and write the detailed caption yourself, and that&#8217;s what I&#8217;d recommend if the whole point for you is running locally. Still, it&#8217;s a little ironic that a model pitched on cheap, local generation leans on a big cloud model by default.</p>
<p>Fourth, hardware. The README says it needs a CUDA GPU, so it&#8217;s built for NVIDIA cards. The image model weights are 8.7 GB in FP32 and the text encoder adds about 8 GB in BF16, plus memory for the actual generation. Linum doesn&#8217;t publish a minimum VRAM figure, but with nearly 17 GB of weights before activations, this doesn&#8217;t look like an 8 GB-card model as shipped. Owners of other GPUs will have to wait for community ports. Generation also isn&#8217;t quick by today&#8217;s distilled standards: the default sampler runs 50 steps at about 1 megapixel, and Linum doesn&#8217;t give speed numbers for consumer cards.</p>
<p>One more piece of transparency that&#8217;s easy to miss: Linum says the GitHub repo and Hugging Face model card were written by Claude, which pulled the inference code out of the lab&#8217;s internal experiment repository. The model, training and review are credited to Linum. It&#8217;s a sign of how small teams are shipping in 2026, and I&#8217;d rather a lab say so than leave people guessing.</p>
<figure class="wp-block-image size-large"><img decoding="async" src="https://4gqtv.com/wp-content/uploads/2026/11/linum-pjit-21k-announcement.jpg" alt="Linum announcement card reading Training a Text-to-Image Model for $21K, 2B P-JiT, Apache 2.0, with a grid of sample images"/><figcaption>Linum&#8217;s official announcement card for P-JiT 21K. Image: Linum</figcaption></figure>
<h2>Cody&#8217;s take</h2>
<p>I&#8217;m excited about this one, but not because P-JiT is going to replace anyone&#8217;s go-to image model. It won&#8217;t. It can&#8217;t write a word, it trips over crowds and full-body shots, and its default setup depends on a paid cloud API. If you just want the best pictures on your own PC today, the established open models with big community ecosystems are still the practical choice.</p>
<p>What gets me is the receipt. Linum published the architecture, the training schedule, the noise curriculum, the hardware and a line-by-line cost breakdown, then released the weights under Apache 2.0 so anyone can poke at them commercially or otherwise. That&#8217;s rare. Most image model launches tell you how good the results are and almost nothing about what it cost to get there. A credible &#8220;$21K for a 2B model&#8221; data point gives university labs, indie researchers and small startups a realistic target.</p>
<p>It&#8217;s also a stepping stone, not the destination. Linum says the whole point is cheaper, more controllable video generation for animation tools, and this image model is the &#8220;learn the nouns before the verbs&#8221; phase. Whether the VAE-free, pixel-space approach holds up at video scale is the real test, and we won&#8217;t know until that model shows up. Until then, P-JiT 21K is a fun download for anyone with a capable NVIDIA card and a willingness to write long prompts, and a pretty strong argument that you don&#8217;t need a giant lab to do meaningful generative AI research.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/a-two-person-lab-trained-an-open-image-model-for-21k-heres-what-p-jit-can-and-cant-do/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Kakao&#8217;s Tiny Kanana Decision Model Won&#8217;t Chat With You — and That Is Exactly Why It Matters</title>
		<link>https://4gqtv.com/kakaos-tiny-kanana-decision-model-wont-chat-with-you-and-that-is-exactly-why-it-matters/</link>
					<comments>https://4gqtv.com/kakaos-tiny-kanana-decision-model-wont-chat-with-you-and-that-is-exactly-why-it-matters/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Fri, 06 Nov 2026 21:12:37 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[Decision Models]]></category>
		<category><![CDATA[Hugging Face]]></category>
		<category><![CDATA[Kakao]]></category>
		<category><![CDATA[Kanana]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Open Weights]]></category>
		<guid isPermaLink="false">https://4gqtv.com/kakaos-tiny-kanana-decision-model-wont-chat-with-you-and-that-is-exactly-why-it-matters/</guid>

					<description><![CDATA[Kakao open-sourced Kanana-2-v-decide on November 6: a 3.67B multimodal model that returns probabilities, choices and scores for AI agents, not chat text. Here's what shipped, what the benches claim, and the license catch.]]></description>
										<content:encoded><![CDATA[<p>While most of the open-AI conversation this week has been about models so big they need a server room, Kakao went the other direction. On November 6, the Korean tech company <a href="https://www.kakaocorp.com/page/detail/12193" target="_blank" rel="noopener">put Kanana-2-v-decide on Hugging Face</a>: a 3.67-billion-parameter multimodal model that does not write essays, code or chat replies at all. It answers structured questions with probabilities, choices and scores, and it is small enough that a lot of local AI setups can actually load it.</p>
<p>That is a different bet from the usual &#8220;bigger weights, longer context&#8221; race. Kakao is aiming at the boring, high-volume decisions that sit inside AI agents: classify this ticket, pick a tool, rate how angry the customer sounds, decide whether a product photo shows damage. If those calls stay slow or fuzzy, the whole agent stalls. Kanana-2-v-decide is built to make them fast and typed.</p>
<p>Here is what shipped, what the numbers actually say, and where the license gets picky.</p>
<h2>A model that refuses to write you a paragraph</h2>
<p>According to the <a href="https://huggingface.co/kakaocorp/kanana-2-v-decide-3b-2611" target="_blank" rel="noopener">Hugging Face model card</a> and Kakao&#8217;s own <a href="https://tech.kakao.com/posts/846" target="_blank" rel="noopener">tech blog</a>, Kanana-2-v-decide-3b-2611 sits on top of Kakao&#8217;s Kanana-2-v-3b multimodal stack: a 0.49B vision encoder plus a 3.18B language model, for 3.67B parameters total. The input &#8220;state&#8221; can be plain text, JSON, or a chat history, with up to 10 images and up to 20,480 tokens. What you get back is not free-form text. You get one of three answer types:</p>
<ul>
<li><strong>Choice</strong> — pick one option from a map you supply, with a probability for each</li>
<li><strong>Noul</strong> — a yes/no-style probability that a statement is true</li>
<li><strong>Score</strong> — an ordered rating across 2 to 10 levels you define</li>
</ul>
<p>You can ask several of those questions in one request. Kakao&#8217;s demo is the damaged-mug case: a photo of a broken ceramic mug in a shipping box plus the message &#8220;The product arrived broken. Please exchange it for a new one.&#8221; The model returns something like a high probability the item is damaged, &#8220;exchange&#8221; as the inquiry type, and a dissatisfaction score, all in one pass. That is the point. An agent can wire those fields straight into routing logic instead of parsing a paragraph and hoping the model said &#8220;refund&#8221; instead of &#8220;exchange.&#8221;</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1920" height="1080" src="https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic.jpg" alt="Kakao Kanana-2-v-decide launch graphic showing a broken mug inquiry producing three typed judgments" class="wp-image-3908" srcset="https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic.jpg 1920w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic-768x432.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-2-v-decide-kakao-launch-graphic-1536x864.jpg 1536w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /><figcaption>Kakao&#8217;s official Kanana-2-v-decide launch graphic: one multimodal request, three typed answers. Image: Kakao</figcaption></figure>
<p>Under the hood it is an embedding similarity model, not a generative decoder for this task. The card says each question-plus-state is encoded into one embedding, each candidate answer into another, and a softmax over cosine similarities produces the distribution. Candidate embeddings can be cached across calls. That design is why Kakao can talk about sub-100-millisecond decisions on a single high-end GPU, and why changing the option list changes the distribution: the probabilities are relative to the candidates you gave it, not absolute truths about the universe.</p>
<p>On Kakao&#8217;s numbers, that speed claim is aggressive. The company says the model averages under 0.04 seconds per judgment on one NVIDIA H200, which it frames as about 43 times faster than GPT-6 Luna for the same kind of call. Treat that as a vendor lab figure on top-shelf hardware, not as what you will see on a gaming PC. Still, the architecture makes the direction believable: no long token stream to sample, just embeddings and a softmax.</p>
<h2>The scores look strong, and the caveats matter</h2>
<p>Kakao is not shy about the leaderboard. On its Korean in-house set, Kanana Decision Bench (15,909 text questions), the model card lists 72.50. On the public Decision Index 0.4.1 and Decision Index Vision 0.4.1, it posts 61.34 and 61.12 for a 61.23 average, which Kakao says tops the models it compared, including the best Decision Index Vision score in that table. The press release also name-checks Microsoft Decision-1, TypeSafe AI&#8217;s Jev, Perplexity&#8217;s pplx-decider line and Upstage&#8217;s Solar Decide as part of the broader field.</p>
<p>Those comparisons are useful, but they are still Kakao&#8217;s tables. Decision Bench is internal. Independent reproduction of Decision Index runs is the next step, not a finished story. For Korean-heavy agent work, the gap Kakao claims over the next models in its own bench is large enough to care about. For English-only pipelines, I would want third-party numbers before swapping out whatever classifier you already trust.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1272" height="376" src="https://4gqtv.com/wp-content/uploads/2026/11/kanana-official-wordmark.jpg" alt="Official Kanana wordmark logo on pink background" class="wp-image-3909" srcset="https://4gqtv.com/wp-content/uploads/2026/11/kanana-official-wordmark.jpg 1272w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-official-wordmark-300x89.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-official-wordmark-1024x303.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/kanana-official-wordmark-768x227.jpg 768w" sizes="auto, (max-width: 1272px) 100vw, 1272px" /><figcaption>Official Kanana wordmark from the model repository. Image: Kakao</figcaption></figure>
<p>For local AI people, the more interesting number is the file size. The public checkpoint&#8217;s <code>model.safetensors</code> is about 7.3 GB. That is a completely different world from this week&#8217;s multi-terabyte open drops. With transformers, <code>trust_remote_code=True</code>, and bfloat16 on a CUDA GPU, Kakao&#8217;s sample loads the model and calls <code>system_one</code>. There is also a small standard-library HTTP server that speaks a System One-compatible API, so existing Jev-style clients can point at a new base URL. Video is not supported. Extreme aspect-ratio images are not supported. Those limits are documented up front, which I appreciate more than a glossy &#8220;runs anywhere&#8221; claim.</p>
<p>Licensing is the other place to read carefully. The weights use the <a href="https://huggingface.co/kakaocorp/kanana-2-v-decide-3b-2611/blob/main/LICENSE" target="_blank" rel="noopener">Kanana Open License</a>, not Apache 2.0. Research and running the model inside your own product are in scope. Redistribution needs Kakao&#8217;s notice file, the license copy, and a clear &#8220;Powered by Kanana&#8221; credit. Separate commercial permission is required if you want to resell access as an API or cloud service, ship it as an SI or on-prem package, or embed it for on-device products you sell to others. In other words: great for many internal agent stacks, not a free pass to stand up a competing decision API.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1600" height="1200" src="https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia.jpg" alt="Diagram of a multilayer neural network with input, hidden and output layers" class="wp-image-3910" srcset="https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia.jpg 1600w, https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia-300x225.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia-1024x768.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia-768x576.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia-1536x1152.jpg 1536w, https://4gqtv.com/wp-content/uploads/2026/11/neural-networks-diagram-wikimedia-240x180.jpg 240w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption>A multilayer neural network diagram, the kind of structure decision models still sit on even when the output is a probability instead of prose. Photo: hikari_no_yume / Wikimedia Commons, CC BY 4.0</figcaption></figure>
<h2>Cody&#8217;s take</h2>
<p>I like this release more than another giant chatbot dump, and I like it for practical reasons. Agents fail on the little decisions constantly: wrong tool, wrong ticket bucket, wrong confidence that a photo shows a cracked screen. A 3.67B model that only does those calls, returns calibrated options, and fits in a single digit of gigabytes is the kind of open weight local builders can actually put next to a RAG pipeline this weekend.</p>
<p>I am not taking Kakao&#8217;s &#8220;beats everyone&#8221; bench story as settled. Vendor benchmarks always need outside checks, and the H200 latency comparison is a lab flex. The license also means you should talk to legal before you productize a hosted decision API. Even with those asterisks, the shape of the release is right. Open AI does not only need bigger models. It needs smaller, boring, reliable ones for the decisions that make agents useful. Kanana-2-v-decide is one of those, and it is live today.</p>
<p><em>Featured image: official Kanana-2-v-decide launch graphic from <a href="https://tech.kakao.com/posts/846" target="_blank" rel="noopener">Kakao Tech</a>. Image: Kakao.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/kakaos-tiny-kanana-decision-model-wont-chat-with-you-and-that-is-exactly-why-it-matters/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Moonshot Just Opened Kimi K3.1 Pro&#8217;s 2.8-Trillion-Parameter Weights. Here&#8217;s Who Can Actually Run It</title>
		<link>https://4gqtv.com/moonshot-just-opened-kimi-k3-1-pros-2-8-trillion-parameter-weights-heres-who-can-actually-run-it/</link>
					<comments>https://4gqtv.com/moonshot-just-opened-kimi-k3-1-pros-2-8-trillion-parameter-weights-heres-who-can-actually-run-it/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 14:10:16 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[AI Models]]></category>
		<category><![CDATA[Hugging Face]]></category>
		<category><![CDATA[Kimi K3.1 Pro]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[Moonshot AI]]></category>
		<category><![CDATA[Open Weights]]></category>
		<guid isPermaLink="false">https://4gqtv.com/moonshot-just-opened-kimi-k3-1-pros-2-8-trillion-parameter-weights-heres-who-can-actually-run-it/</guid>

					<description><![CDATA[Moonshot AI released Kimi K3.1 Pro's open weights on Hugging Face: 2.8T parameters, 1M context and a 1.56 TB download. Here's the license, the hardware reality for local AI, and the benchmark audit to know about.]]></description>
										<content:encoded><![CDATA[<p>Moonshot AI kept its promise. On November 4, the company put the full weights of <a href="https://huggingface.co/moonshotai/Kimi-K3.1-Pro">Kimi K3.1 Pro on Hugging Face</a>, right on the deadline it set when the model first went live in its apps and API in late October. The repository is public, it isn&#8217;t gated, and anyone can download it.</p>
<p>That is a genuinely big deal for open AI. It is also a 1.56 TB download, and the honest answer to &#8220;can I run this on my rig?&#8221; is no, not yet, and probably not ever on a single home machine. Both of those things are true at once, so let&#8217;s go through what actually shipped, who it is really for, and the benchmark argument that showed up the day before the weights did.</p>
<h2>What Moonshot actually released</h2>
<p>Per the <a href="https://huggingface.co/moonshotai/Kimi-K3.1-Pro">official model card</a>, K3.1 Pro is a mixture-of-experts model with 2.8 trillion total parameters, of which 104 billion are active for any given token. It has 896 experts, routes each token to 16 of them plus two shared experts, and stacks 93 layers. It takes text and images as input through a 401-million-parameter MoonViT-V2 vision encoder, and its context window is 1,048,576 tokens, a full million.</p>
<p>The architecture is not a brand-new design. Moonshot says K3.1 Pro is post-trained from Kimi K3, with one real structural change: the full-attention layers are now Gated DeepSeek Sparse Attention layers, meant to make long contexts cheaper. That leaves 69 Kimi Delta Attention layers and 24 sparse-attention layers. The rest of the improvement, according to Moonshot&#8217;s <a href="https://www.kimi.ai/blog/kimi-k3-1-pro">tech blog</a>, comes from much larger reinforcement learning runs across &#8220;hundreds of downstream domains.&#8221;</p>
<p>The format detail matters most for anyone thinking about hardware. The weights ship as MXFP4, with MXFP8 activations, and the model was trained that way using quantization-aware training. In plain terms, this 4-bit file is the precision Moonshot serves in production, not a squeezed-down copy. The Hugging Face file listing comes to 96 safetensors shards and about 1.56 TB. <a href="https://artificialwatch.com/wire/kimi-k3-1-pro-open-weights-out">ArtificialWatch</a> notes that&#8217;s the same size as the Kimi K3 release, which makes sense given the shared shape.</p>
<p>Licensing is friendlier than a lot of people assumed. The <a href="https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LICENSE">Kimi K3.1-Pro License</a> reads like MIT with two conditions. If you run a &#8220;Model as a Service&#8221; business, such as selling API access, and you pull in more than $20 million over any 12 months, you need a separate agreement with Moonshot. If your product has more than 100 million monthly users or more than $20 million in monthly revenue, you have to show the &#8220;Kimi K3.1-Pro&#8221; name in your interface. Internal use is exempt from both. For researchers, startups and companies running it on their own infrastructure, it is effectively open.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1600" height="900" src="https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo.jpg" alt="Third-person Godot adventure game scene with a blue-hooded hero facing the Amethyst Guardian boss, which Moonshot says Kimi K3.1 Pro built" class="wp-image-3884" srcset="https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo.jpg 1600w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo-768x432.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-godot-game-demo-1536x864.jpg 1536w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption>A Godot game Moonshot says Kimi K3.1 Pro built end to end. Image: Moonshot AI</figcaption></figure>
<p>On capability, Moonshot is pitching this as an agentic coding and work model. Its headline number is 65.7% on Terminal-Bench 4.0, just behind Claude Opus 5.5 at 66.4% and ahead of GPT-6 Astra at 58.2%, per the company&#8217;s own table. It also claims the top score in that table on SWE Marathon v1.1, at 51.7. The blog shows off games built in Unity, Unreal and Godot, plus 3D modeling through scripted Blender and CAD work. As a gamer, I find the Godot demo above charming, but keep in mind it&#8217;s a curated vendor showcase, not an independent test.</p>
<p>Moonshot&#8217;s table also shows the model trailing in places. On the full Humanity&#8217;s Last Exam set without tools, K3.1 Pro scores 51.1 against 59.8 for Opus 5.5, and it is clearly behind on several of the cybersecurity tests. That last gap is deliberate. Moonshot says it found and removed cyber-related data from both mid-training and RL for the open release. On its internal xdev-bench-hard exploit benchmark, the unfiltered checkpoint solved 90.5% of problems, while the shipped model solves 81.0%. Moonshot says it doesn&#8217;t want open weights to &#8220;extend the absolute frontier of cyber capability.&#8221; Whether you see that as responsible or as a convenient story, it&#8217;s one of the more detailed explanations of an open-weight safety tradeoff any lab has published.</p>
<h2>The local AI reality check</h2>
<p>Here&#8217;s where the local AI crowd, me included, has to be realistic. The <a href="https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro">vLLM recipe</a> for K3.1 Pro starts at eight data-center GPUs running tensor parallel: two NVIDIA GB300 trays or one B300 node, with a one-node AMD Instinct MI355X setup also listed. The model card recommends vLLM, SGLang or TokenSpeed. Those are server inference engines, not desktop apps.</p>
<p>The usual home route doesn&#8217;t exist yet either. llama.cpp shipped <a href="https://github.com/ggml-org/llama.cpp/releases/tag/v0.9.0">v0.9.0</a> on November 4, the same day the weights dropped, with new support for Mistral Large 4, Metis-1.6 and others, but nothing for K3.1. As <a href="https://modelfit.io/blog/kimi-k3-1-pro-mac-memory-requirements/">ModelFit</a> and <a href="https://www.modemguides.com/blogs/ai-news/run-kimi-k3-locally-hardware-reality-check">Modem Guides</a> both point out, the config declares a new <code>kimi_k31</code> architecture. llama.cpp&#8217;s Kimi K3 converter can&#8217;t handle the sparse-attention layers, so nobody can make a GGUF yet. Repositories already carry the K3.1 Pro name, including an Unsloth mirror of the original files, but none of them is a desktop-ready quant.</p>
<p>Even once someone writes support, the floor stays high. Working from Unsloth&#8217;s measured Kimi K3 builds, Modem Guides estimates that a 1-bit K3.1 build would need roughly 610 GB of combined RAM and VRAM. That&#8217;s beyond even a maxed-out 512 GB workstation, never mind a gaming PC. Some Hacker News commenters have suggested streaming the weights from fast SSDs on smaller machines, but that&#8217;s a slow workaround that nobody has benchmarked, not a real plan.</p>
<p>So who is this for? Universities, inference providers, and companies that want frontier-class agents on hardware they control, with no data leaving the building. For everyone else, open weights still help: anyone can host the model, so third-party providers can compete on price and privacy instead of everyone going through Moonshot&#8217;s own API. Early reaction rounded up by <a href="https://www.llms.blog/posts/kimi-k3-1-pro-weights-reactions">LLMs Blog</a> found that several hosts listed it on day one.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1600" height="900" src="https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages.jpg" alt="Five-stage diagram of a muscle-driven human walking simulation progressing from falling to a natural gait" class="wp-image-3885" srcset="https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages.jpg 1600w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages-300x169.jpg 300w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages-1024x576.jpg 1024w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages-768x432.jpg 768w, https://4gqtv.com/wp-content/uploads/2026/11/kimi-k3-1-pro-walking-simulator-stages-1536x864.jpg 1536w" sizes="auto, (max-width: 1600px) 100vw, 1600px" /><figcaption>Moonshot&#8217;s walking-simulator case study, from collapse to stable strides. Image: Moonshot AI</figcaption></figure>
<p>Moonshot&#8217;s strongest pitch isn&#8217;t a benchmark. It&#8217;s the long-horizon case study in the image above. In an internal test, the company says, K3.1 Pro built a muscle-driven human walking simulator and worked through 167 missions to take the virtual body from collapsing within seconds to walking for 11.2 seconds. Along the way it found and fixed 36 errors in the simulator&#8217;s shoulder-muscle math instead of just endlessly tweaking rewards. If that holds up outside Moonshot&#8217;s lab, it&#8217;s exactly the kind of patient debugging that makes agents useful.</p>
<h2>Cody&#8217;s take</h2>
<p>The benchmark question deserves a clear answer, and right now there isn&#8217;t one. On November 3, a day before the weights went public, two researchers at R40 Research posted a <a href="https://r40.org/papers/kimi_k31_benchmark_audit.pdf">preprint audit</a> of K3.1&#8217;s behavior. They report that in about 23% of Terminal-Bench 4.0 attempts, the model&#8217;s reasoning cited &#8220;instructions&#8221; that weren&#8217;t in the task but that the hidden tests check. They also found that when they edited a task file to contradict one of those hidden rules, K3.1 ignored the edit and failed in three of four cases, while other models followed it. The authors think this most likely points to training on the benchmark tasks themselves, but they are explicit that their analysis &#8220;cannot prove it.&#8221; I haven&#8217;t found a public response from Moonshot.</p>
<p>That doesn&#8217;t make K3.1 Pro a bad model. Plenty of developers say K3 and K3.1 feel strong in day-to-day coding, and open weights are exactly what lets independent researchers poke at claims like these. It does mean I&#8217;d treat the near-tie with Opus 5.5 on Terminal-Bench as unproven until more independent tests come in.</p>
<p>The bigger picture is still exciting. A 2.8-trillion-parameter model with a million-token context, under a license almost anyone can use, is a real win for open AI, even if most of us will only ever touch it through a hosted endpoint. If you want something you can actually run at home this week, the smaller models in that same llama.cpp release are the place to start. K3.1 Pro is a sign of where open models are heading, and it&#8217;s worth watching closely before you trust the scoreboard.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/moonshot-just-opened-kimi-k3-1-pros-2-8-trillion-parameter-weights-heres-who-can-actually-run-it/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>AMD’s Ryzen AI 1.9 Lets Your NPU Switch Between Specialist Models on the Fly (What to Know First)</title>
		<link>https://4gqtv.com/amds-ryzen-ai-1-9-lets-one-npu-model-swap-between-specialists-but-read-the-fine-print/</link>
					<comments>https://4gqtv.com/amds-ryzen-ai-1-9-lets-one-npu-model-swap-between-specialists-but-read-the-fine-print/?noamp=mobile#respond</comments>
		
		<dc:creator><![CDATA[Cody Brennan]]></dc:creator>
		<pubDate>Mon, 02 Nov 2026 21:05:56 +0000</pubDate>
				<category><![CDATA[AI]]></category>
		<category><![CDATA[AMD]]></category>
		<category><![CDATA[Tech]]></category>
		<category><![CDATA[Local AI]]></category>
		<category><![CDATA[LoRA]]></category>
		<category><![CDATA[NPU]]></category>
		<category><![CDATA[Qwen]]></category>
		<category><![CDATA[Ryzen AI]]></category>
		<guid isPermaLink="false">https://4gqtv.com/amds-ryzen-ai-1-9-lets-one-npu-model-swap-between-specialists-but-read-the-fine-print/</guid>

					<description><![CDATA[Ryzen AI Software 1.9 adds dynamic LoRA switching on the NPU, longer context and new models like Gemma 4. Here's what it does, who it supports, and the catches.]]></description>
										<content:encoded><![CDATA[<p>AI PCs have had a strange problem for a while now. The NPU, the dedicated AI block AMD has been building into its Ryzen AI chips, is supposed to be the efficient, always-on place to run local language models. In practice, the software around it has often felt a step behind the hardware. AMD&#8217;s newest update, <a href="https://github.com/amd/RyzenAI-SW/releases/tag/v1.9.0">Ryzen AI Software 1.9</a>, doesn&#8217;t fix that overnight, but it does chip away at two of the biggest limits developers kept running into: short context windows and the clunky way fine-tuned models had to be shipped.</p>
<p>The release landed on GitHub on October 30, and on November 2 AMD followed up with a <a href="https://www.amd.com/en/developer/resources/technical-articles/2026/dynamic-lora-on-amd-ryzen-ai.html">developer blog post</a> walking through the feature I think is the most interesting part of it: dynamic LoRA switching on the NPU.</p>
<h2>One base model, swappable specialists</h2>
<p>Quick refresher if you haven&#8217;t messed with fine-tuning. LoRA (low-rank adaptation) is the popular trick for teaching an existing model a new skill without retraining the whole thing. You freeze the big base model and train a small set of extra weights, the adapter, that nudges its behavior toward writing SQL, answering domain questions, or matching a particular style. The adapter is tiny compared to the model it sits on.</p>
<p>Until this release, AMD&#8217;s supported way to run a fine-tuned model on the Ryzen AI NPU was what it calls a static merge. You baked the adapter into the base weights and compiled the result into one NPU binary. That&#8217;s fine if you only have one specialist. If you have five, you end up shipping five complete copies of the same base model, which is a rough deal on a laptop where memory is already shared between the CPU, the integrated GPU and the NPU. Switching from one specialist to another also meant loading a whole different model.</p>
<p>With 1.9, the base is compiled once and stays resident on the NPU, while each adapter becomes a small, separately loadable piece. An app picks which one is active per request. In AMD&#8217;s sample code it&#8217;s literally one call, <code>set_active_adapter("sql_spider")</code>, then another to flip to a medical adapter, and one more to drop back to the plain base model, all without reloading anything.</p>
<figure class="wp-block-image size-large"><img decoding="async" src="https://4gqtv.com/wp-content/uploads/2026/11/amd-ryzen-ai-dynamic-lora-diagram.jpg" alt="AMD diagram showing a Qwen3-4B INT4 base model resident on the Ryzen AI NPU with SQL and medical LoRA adapters switched at runtime using set_active_adapter"/><figcaption>AMD&#8217;s diagram of dynamic LoRA: the base model stays loaded on the NPU while adapters are switched in by name. Image: AMD</figcaption></figure>
<p>AMD&#8217;s example is honest about scale. It trains two adapters on a Qwen3-4B model quantized to INT4: one for text-to-SQL using the Spider benchmark, one for medical multiple-choice questions using MedMCQA. On the NPU, the SQL adapter lifted execution accuracy from 47.5% to 63.0%, and the medical adapter raised exact-match scores from 42.0% to 53.0%. Both were measured on 200 examples each, which is a decent sanity check rather than a rigorous benchmark run, and these are AMD&#8217;s own numbers. Still, double-digit jumps from a small adapter are exactly the kind of result that makes LoRA so popular in the first place.</p>
<p>The most useful bit of the blog post is a warning, not a brag. Because the base model runs quantized on the NPU, AMD says you have to train the adapter against that already-quantized base. Train it on the full-precision model instead, and the corrections the adapter learned won&#8217;t line up with the weights it&#8217;s actually being added to, so a lot of the benefit disappears on-device. The suggested pipeline is to quantize once with AMD&#8217;s Quark tool (INT4 AWQ), fine-tune each adapter against that frozen base, compile everything with one <code>model_generate</code> call per adapter, and only then deploy. Only that last step happens on the Ryzen AI PC itself; the training happens elsewhere.</p>
<p>There&#8217;s a catch on coverage, too. Dynamic LoRA currently works with exactly three text models: Qwen2.5-3B, Qwen3-4B and Qwen3.5-4B. If your favorite model isn&#8217;t a small Qwen, this feature isn&#8217;t for you yet.</p>
<h2>Longer context and a bigger model list, with asterisks</h2>
<p>The other headline change is context length. Earlier releases capped LLMs on the NPU at 4K or 16K tokens depending on the setup, which is tight for anything like summarizing long documents or keeping a chat assistant on track. The <a href="https://ryzenai.docs.amd.com/en/latest/relnotes.html">1.9 release notes</a> say context now extends beyond those limits. What they don&#8217;t say is the new ceiling, and I&#8217;d want to see real numbers on how speed and memory hold up as context grows before calling this solved. Long prompts are where small on-device accelerators usually start to sweat.</p>
<p>Model support grew as well. Gemma-4-E2B and Gemma-4-E4B (both text) and Hy-MT2-1.8B are newly supported. AMD also added Phi-4-14B, Qwen3.5-9B, Granite-4.1-8B and Gemma-4-12B, but it labels those four &#8220;functional only (performance not optimized).&#8221; That phrase is doing a lot of work. It means they run, not that they run well, so anyone hoping to make a 12B or 14B model their daily driver on an NPU should test before committing. The release also adds the ability to import INT4 models produced through quantization-aware training, which should help teams that already quantize in their own training pipelines.</p>
<p>Now the part that will annoy some people. AMD&#8217;s <a href="https://ryzenai.docs.amd.com/en/main/hybrid_oga.html">OnnxRuntime GenAI documentation</a> says 1.9 moves to OGA version 0.18.0, up from 0.14.0 in Ryzen AI 1.8, and that NPU models published for earlier releases are not compatible with this one. If you built something on 1.8, plan on downloading the updated models rather than expecting a drop-in upgrade. That&#8217;s a fairly normal growing pain for a young software stack, but it&#8217;s worth knowing before you hit update on a working setup.</p>
<p>Hardware support is the other fine print. The <a href="https://ryzenai.docs.amd.com/en/latest/llm/overview.html">LLM deployment docs</a> list NPU-only and hybrid NPU-plus-iGPU execution for Ryzen AI 300 series chips (Strix and Krackan Point), while older Ryzen 7000 and 8000 mobile chips built on Phoenix and Hawk Point fall back to GPU or CPU inference for LLMs. The release notes&#8217; supported-hardware list also includes Strix Halo, Gorgon Point and the Ryzen AI 400 series, which AMD rates at <a href="https://newsroom.amd.com/news/amd-expands-ai-leadership-across-client-graphics/">up to 60 NPU TOPS</a> on its top mobile chips. So if you&#8217;re on an early-generation Ryzen AI laptop, most of this update simply won&#8217;t reach your NPU.</p>
<p>It&#8217;s also worth remembering that the Ryzen AI stack isn&#8217;t one tool. AMD&#8217;s docs point to the Lemonade SDK and Lemonade Server for an Ollama- or OpenAI-style local API, OnnxRuntime GenAI for the NPU and hybrid paths, FastFlowLM as an NPU-first runtime, and llama.cpp for GPU inference. Dynamic LoRA lives in the OGA flow, so how quickly it shows up in friendlier front ends is still an open question.</p>
<h2>Cody&#8217;s take</h2>
<p>I like this release because it&#8217;s practical. Dynamic LoRA isn&#8217;t flashy, but it solves a real deployment headache: one resident base model that can act as several narrow specialists is a much better fit for a laptop than a pile of near-identical full models. For anyone building a local app that needs, say, a coding helper, a support-docs bot and a writing assistant on the same machine, that&#8217;s a meaningful change. And AMD deserves credit for calling out the train-on-the-quantized-base trap instead of letting developers discover it the hard way.</p>
<p>The realistic view is that it&#8217;s early. Three supported models, an unstated context ceiling, &#8220;functional only&#8221; labels on the bigger models and a model-compatibility break with 1.8 all say this is a stack still finding its footing. If you&#8217;re a developer with a Ryzen AI 300 or 400 machine, it&#8217;s worth grabbing the <a href="https://github.com/amd/RyzenAI-SW">Ryzen AI Software examples</a> and trying your own adapters. If you&#8217;re a regular user waiting for the NPU in your laptop to feel essential, this is a solid step, not the finish line.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://4gqtv.com/amds-ryzen-ai-1-9-lets-one-npu-model-swap-between-specialists-but-read-the-fine-print/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
