Artificial Analysis × Bloom: How AI Models Get Scored

The Artificial Analysis team held their first event in Korea with Bloom.

You've probably seen the bar chart that gets quoted every time a new AI model comes out. Even if you don't know the company's name, you know that chart. The team that makes it came over from San Francisco. They're an independent organization that doesn't belong to any lab, and every result they produce is public on their website.

We also brought in two teams building models in Korea. SK Telecom and Trillion Labs each walked through how they're building their models, and George Cameron, co-founder of Artificial Analysis, joined by video at 3:30 a.m. San Francisco time to give the keynote. Thank you for staying up that late.

Here's a write-up of what came out of the room.

88 percent said they already look at benchmarks

271 people signed up, and the fifth floor of D.Camp Mapo (a startup hub in western Seoul) was packed. The demographics we pulled from the sign-up form explained the topic on their own: 88 percent said they already look at benchmarks when choosing a model, and 36 percent train models themselves or build them into products.

We also put up a list of who was in the room. From the model and AI side: SK Telecom and Trillion Labs, plus NAVER and NAVER Cloud, Upstage, KT, Krafton AI, NC AI, Lablup, FriendliAI, Moreh, MAUM AI, TwelveLabs, RLWRLD, Coxwave, VESSL AI, VUNO, Cochl, and Mind AI. From the enterprise side: Samsung Electronics, Hyundai Motor, Hyundai Mobis, Hyundai AutoEver, LG CNS, LG Electronics, HD Hyundai Robotics, Coupang, Toss and its parent Viva Republica, Lotte Innovate, CJ ENM, SK Innovation, SK On, TMAP Mobility, Samsung Medical Center, and Pulmuone. From global companies: Google Korea, Microsoft, AWS, Akamai, Arize AI, Sendbird, Photoroom, Mysten Labs, eBay Japan, Concentrix, Megazone Cloud, Cloocus, and Bespin Global. From academia, media, and investment: KAIST, Seoul National University, Yonsei, Korea University, Hanyang, Sungkyunkwan, Seoul National University of Science and Technology, Oxford, plus the Chosun Ilbo, JoongAng Ilbo, Electronic Times, AI Times, Venture Square, Samsung Venture Investment, and Mirae Asset Capital.

We let people write questions on the sign-up form, and 227 of them did. The most common theme was how the scores get made (55), followed by the gap between benchmark scores and real performance (45), evaluating Korean models and sovereign AI (29), picking models with cost and speed in mind (22), evaluating agents (14), and benchmark contamination and overfitting (9). We put three standout questions on screen as written. When the index gets updated, a model's score can go up even though the model hasn't changed, so how do you tell real improvement apart? How should sovereign AI performance be evaluated, and what does benchmarking for Korean models need? And how much do benchmark scores actually get used when picking a real product? Our goal for the night was to answer these as well as we could.

The whole stack, from models to inference to hardware

Henry Jin from Artificial Analysis, who leads their Asia Pacific partnerships, introduced the team first. Artificial Analysis is an independent AI benchmarking and analysis company that evaluates the full stack, from the model to inference to the hardware layer. They look at intelligence, speed, price, and other relevant traits, and the goal is to help developers and companies make the most informed decisions when they use and build with AI.

He said they've worked closely with the Korean AI community over the past year. Meeting AI labs, the government, various organizations, and the developer and research communities, they saw firsthand how fast Korean AI has moved, and that was a big reason they wanted to hold this event.

The magic still comes from the model

George started with a short intro to the company. It's an independent benchmarking company that millions of users and major AI companies refer to. What he's especially proud of is that companies cite them to show the tradeoffs developers have to make, rather than quoting a single intelligence bar chart. That's why their charts are all 2x2s. OpenAI used intelligence versus cost, Jensen Huang used intelligence versus speed, and Anthropic used the coding agent benchmark alongside the speed benchmark.

They benchmark agents; language, image, video, speech, and music models; and hardware and inference. The first point he raised stuck with me. People talk about agents a lot these days, and most models get used in agent form, but it's worth being clear about where the magic comes from. The harness provides tools that interact with the real world, like code execution and web search, but the model is still what drives what's possible.

As evidence, he showed their coding agent index, which tests a model running inside a harness, so it measures both. Smarter models still do better. Putting a great harness on a weak model doesn't beat putting an average harness on a good model. A few years ago that was an open question, and so far it has held true.

Using Fable for every task could bankrupt you

The Intelligence Index combines ten evaluations. It covers general knowledge and hallucination, scientific reasoning, coding, and agentic workloads, and the leaders are Anthropic's Claude Fable 5.1 and GPT-6. He said that wouldn't surprise anyone in this room.

What he wanted to talk about, though, was that there are still reasons to use other models.

For a lot of companies, using Claude Fable 5.1 for every task could bankrupt the company.

So picking a model today is a complicated tradeoff between different criteria.

Korea is right behind the US and China

The part about Korea got the most nods of the night. Over the past year, Korea has moved into the top three countries, behind the US and China. Countries like Canada and Spain have one or two individual models, but Korea has several AI companies with their own models, and it happened quite fast.

He gave credit to the companies that built the models and also to the Korean government's sovereign AI initiative. Artificial Analysis has taken part in it too, and they've signed an MOU with NIPA (the National IT Industry Promotion Agency) and plan to keep supporting it as an independent evaluation partner.

He also looked at open weights models from another angle. Back in early 2024, when people were talking about Llama 2 70B and Mixtral 8x7B, it was an open question whether open weights would stay competitive and whether labs would keep paying the cost of releasing them. What's been proven since is that labs still have reasons to release open weights, because it meets users' need for flexibility when they run models on their own hardware or fine-tune them. He doesn't think open weights will overtake proprietary models anytime soon, but he expects them to keep delivering intelligence just below the frontier. The current leader is Z.ai's GLM 5.3.

Price per token is outdated

When he got to how 2026 is about more than intelligence, he made a joke. The Artificial Analysis website has covered intelligence, speed, and cost since day one, but people only started caring about the last two charts this year.

Cost matters now because there's a 100x gap between the leading models and the cheap ones. On the September 14 screenshot of their site that I put up at the opening, the cost of solving the same single task ranged from $0.06 to $8.75. Completing the Intelligence Index tasks costs more than $7 with Claude Fable 5.1, while other models that still do the tasks very well cost less than 10 cents. One input that drives this cost is the number of output tokens a model uses to finish a task, which depends on how verbose the model is and how many agent steps it takes. Some models use 20 steps on a task and some use 100. When an agent moves through a complex environment checking this file and confirming it follows that policy, it chews through a huge number of tokens.

He added that AI labs may not be eager to get you thinking this way, since they get paid by the token.

You can't just use price per token anymore. That's outdated. In the world of agents, you have to think about output token volume and cost per task.

The cost per task he described includes answer tokens, reasoning tokens, cache writes and cache hits, and input tokens.

The ecosystem has forces pushing costs down and forces pushing them up at the same time, and that split creates the 100x gap you see today. On the down side, smaller models and sparsity mean fewer active parameters and less compute per token, software efficiency gains cut compute further, and on hardware, at scale, Blackwell chips are cheaper than Hopper and handle more tokens. On the up side are bigger models. Demand for more intelligence is never satisfied, which is why everyone got so excited when new versions of Fable and GPT-6 came out a few weeks ago. Add to that more reasoning tokens and agents making more requests per task.

Intelligence gets dramatically cheaper once it's reached

One pattern from these efficiency forces was especially practical. In AI, once a certain level of intelligence is reached, that intelligence gets dramatically cheaper very quickly. They tracked the price at the moment each new Intelligence Index threshold was hit and how it changed afterward. A score of 10 was reached in mid-to-late 2024, and the cost has since dropped more than 100x. The same pattern repeats at other levels. New intelligence arrives, competition follows, and efficiency drives the cost down.

If Fable feels too expensive for your use case right now, you may not have to wait long before it becomes affordable.

He said this matters when you think about what will be possible at next year's costs, and that it's not a bad lens for people building startups either.

Draw your own Pareto curve from your own workload

Map the intelligence and cost tradeoff at a high level and there's a big Pareto frontier between the leading models at over $7 per task and much cheaper ones. You should pick from that frontier, but there are reasons to pick other models too: when you weigh intelligence differently for a specific workload, when you only want open weights, or when you want a model from a particular country.

This is where a new product came up. The intelligence benchmark is a combined, general-purpose measure, good for high-level understanding, but everyone has their own workload. So they recently launched Optima, which lets anyone build a custom benchmark and get a Pareto curve for their own workload. You scroll down on the homepage, log in, and upload the context for your use case, and it generates a benchmark and gives you a chart of each model's tradeoffs on that benchmark. He said it can save more than 10x in cost and time and carries the expertise they built making frontier evaluations.

Models are shifting from reactive to proactive

He also went over what's coming next. On agents, the way coding agents completely changed developers' workflows is now spreading to general work across the economy, and agents including Claude CoWork have started bringing large-scale automation of knowledge work to the public. Continual learning has already started in forms like CLAUDE.md files. In the future it could look like models updating their own weights, but we're not there yet.

On models and cost, he repeated that for many use cases cost has become as much of a priority as intelligence. And as models become natively multimodal, the usage paradigm is changing too. The big trend is models becoming much more proactive instead of reactive. We expect people to come back, ask questions, and gather context, so it's fair to expect that from models too.

He also named sovereign AI as a trend, one that applies to Korea in particular. He said what Korea has done is really impressive, and that the government's foundation model program took real nerve, as much as what the companies achieved on their own.

Nobody can properly measure good Korean yet

The first question was the sharpest of the night. Someone pointed out that the world has a lot of languages besides English, including Chinese and Korean, and asked how performance in those languages gets evaluated, and whether human judgment inevitably comes into it, unlike objective performance testing. As a personal take, they said they really dislike Claude's Korean. Ask it to phrase something in Korean and English idioms and English proverbs pop out. That doesn't fit Korean sensibilities and isn't something Koreans grasp intuitively, yet without some deliberate control the model just does it naturally.

George's answer had two layers. He thinks these models abstract basic language understanding very well. Because they abstract the concept of language itself in latent space, they handle different languages natively, so in actual task completion there isn't a big gap between languages. That's also why Artificial Analysis treats that analysis lightly.

Then he got candid. The nuance of how these models communicate isn't properly captured by the benchmarks out there today, whether they made them or someone else did.

I don't think this topic is being covered well. Honestly, that includes us.

He said it's an important area, he hopes more research comes out, and they want to publish more on it too. He'd like the community to help build benchmarks in this space. They may release a personality benchmark, probably starting in English, and he said it would be great if the international community could look at it and add perspectives from other languages.

Expect more metrics where higher isn't better

Someone who uses AI benchmarks heavily at work asked the next question. You said agents will get much more capable over the next few years. From your experience running evaluations, where are human-made data and human evaluation still most needed: computer use, tool use, or multitasking?

George said the past year to year and a half has focused on reinforcement learning and getting tasks done. On data, he splits it two ways: getting models to do new tasks, and getting them to do current tasks better. Getting them to do new things will likely keep taking the form of more complex RL environments, and there's a long way to go. Right now we expect models to successfully finish very clearly defined tasks, but they can't go out on their own, proactively gather context, ask the right questions, and discover new things. We're working with these models inside a very limited paradigm.

Making them better at what they already do needs more human data, he said. It's needed to improve how models communicate and how they work with us, along with RL environments that push models to do better. He expects we'll talk much more than we do today about metrics where higher isn't better: efficiency, cost efficiency, time efficiency, and communication style. Going back to the earlier question, some people will prefer one communication style and some another. The Artificial Analysis Intelligence Index will stay a higher-is-better metric, but he expects more fine-grained metrics to grow up around it to help people understand models.

Edge and physical AI are homework for later

The last question was about the roadmap for edge. The person asking said they'd seen Artificial Analysis start working with Liquid AI, that the site already covers mobile, and that they expect laptops, workstations, and data center systems too. They said their own team is also a Liquid AI partner and a day-zero partner for edge devices, automatically benchmarking AI models on more than 100 edge devices. They asked about robotics, wearables, and other edge devices, and whether there are plans to go beyond two mobile devices.

George thanked them for noticing the edge benchmarks and said he'd like to expand, but the AI world is big and keeps growing, so it comes down to prioritizing what they can realistically build. Edge benchmarking will stay as it is for now, but they're interested in expanding later, and as the range of devices running AI widens, there will be more physical AI and robotics benchmarks.

He said he'd spent the first two days of that week in Korea. He couldn't stay for the event as planned because something urgent called him back to San Francisco early, and he'd be glad to come in person next time. He said he'd heard a lot of interest in physical AI from Korean AI users and would look into it, and asked anyone with ideas on what it should look like to send them by Twitter DM, email, or the contact form on the website.

The second round was about building a model people can actually use

The second session was Cheon Seong-jun from SK Telecom. The topic was the thinking and planning behind A.X K2, released in July for the second round of Korea's national sovereign AI foundation model program, and the vision going forward. SK Telecom was selected as one of the elite teams in the Ministry of Science and ICT's sovereign AI foundation model program and is one of the three teams that passed the second-stage evaluation. The talk was titled "Toward a Production-Ready 688B-A33B MoE," and the emphasis was on "Production-Ready."

This wasn't his first time in this room. Back in April, at the same venue, there was an NVIDIA Nemotron developer day, and he said then that in the first round SK Telecom and the other elite teams had succeeded in building models worth using, and that the second round was about building models people can actually use. He came back this time carrying that promise.

He tied what it takes to get to production-ready to the earlier question. There's definitely nuance that benchmarks can't measure. Only people looking at it directly can judge it, and AI models still can't evaluate it well. You get a production-ready model by developing with that nuance and usability in mind. Picking up George's point that efficiency keeps getting more important, he said the plan is to build increasingly efficient models, from the architecture to the training process to deployment.

They couldn't shrink the model, so the architecture had to do the work

A.X K2 has 688B parameters with 33B active and was released under Apache 2.0. When they released the first-round model, the most common feedback was that it was too big to use. That really meant cost worries, so it had to get faster and run inference on fewer GPUs. Meanwhile the scale of the foundation model race keeps growing. Trillion-parameter models are being released elsewhere, and you even hear that frontier labs are past 10T. That's a rumor, he noted. They couldn't make the model smaller, so they had to find the answer in architecture design.

A diagram he drew himself held all the distinctive parts. The Indexer and Selector make attention sparse. MoE is a sparse model by default, but adding sparsity to attention itself keeps the cost of reading context closer to the number of selected tokens instead of growing quadratically. MLA uses a compressed latent space to shrink the KV cache, and GatedNorm and the Attention Output Gate reduce outliers in parameter space and keep attention sinks from being misused. That's three threads: select fewer tokens, store less state, and control outliers and sinks. They aren't proposing every component themselves, but he said combining them this way and implementing them in this form is a novel contribution.

Building a new structure made the road to fast training and deployment long and hard. To train fast they had to work at the kernel level, and when the kernel paths for training and inference didn't go the way they wanted, they used kernel fusion. They pulled in complicated techniques like that to squeeze out efficiency.

Observed behavior became the spec for the next round of training

For usability, what they paid special attention to this time was fitting as much of the improvement process as possible into a single training loop. They didn't have the compute to run it many times, so it had to work in one go.

The loop had five stages. Look at model responses from internal usability tests and Arena; turn capability gaps and response patterns into a failure analysis; translate data, rewards, and safety targets into a training design; update with RLHF and RLVR; then re-evaluate with benchmarks, users, and red teams and go back to failure analysis. One line from the slides summed up the loop exactly.

Observed behavior became the specification for the next training iteration.

They placed people carefully inside that loop so humans actually evaluated usability, and they pulled improvements from that. They also built internal and custom benchmarks, and did red teaming both in-house and through outside sourcing to get feedback. He said they've kept building the ability to improve the model continuously and to do it on their own. Seeing usability improve meaningfully this time was a rewarding result.

97.1 on AIME, and 40 points up on long context

When it came to results, he said people always want the scores, and pulled out the numbers. At release two months earlier, it was tied for first on AIME 2026 on MathArena, and when he checked again before coming that day it was still tied for first at 97.1. It wasn't on the slides, but internal testing also showed it performing at gold medal level on the 2026 International Mathematical Olympiad.

They had found long context was weak in the first-round model and worked hard on it. It scored 66.0 on AA-LCR, up 40 points from the first round. In Korean, he said, they confirmed once more that it's the best, with 80.5 on KMMLU-Pro and 91.6 on CLIcK.

688B running on four B200s

The obvious question is how a model that grew to 688B, from 571B in the first round, got more efficient and faster. The answer is that they trained it in FP8. Through the first round they trained in BF16, but this time they trained natively in 8-bit and also released an FP4 version, quantized further. That choice was possible because GatedNorm and sparse gated attention suppress the outliers that amplify quantization error.

Minimum model memory shows the result of this design directly. The first round in BF16 needed 1038GB. A.X K2 in FP8 needs 646GB, and NVFP4 needs 370GB. NVFP4 delivers quality comparable to FP8 and serves on four NVIDIA B200 GPUs. They also support GGUF, widening runtime compatibility and options for local deployment.

To speed up inference further, they made two kinds of speculative decoding available. EAGLE3 raises throughput up to 1.64x and cut time per output token by 28 percent when serving production traffic. DSpark gets about 1.8x and cut it by 44 percent in single-stream evaluation. Exact verification preserves the target model's output distribution. Both are available on Hugging Face.

He said getting good scores each round matters, but they're also looking further ahead and trying to do as much as they can on their own. They're building the whole process themselves, from creating data to improving the model, and he expects the skills and know-how from the second round to carry into the third. He hopes the model gets used widely in "AI for Everyone," a government-led program that started recently, and said they'll keep working so that they can build bigger models well too.

It's software, so why is it this hard?

Someone who works at a company building AI that understands physics asked a good question. A model is software in the end. If it has a structure, it can be written in code, and some people know how. In China, hardworking researchers publish their recipes, and 10,000 or 20,000 papers come out a year. So why is this so hard, and what difficulties did you actually run into?

Cheon laughed and said he'd been wanting to brag about this somewhere but never had the chance. Limited resources are a very hard constraint in themselves, and because of that you have to read a lot of papers to filter out bad options as much as possible. Model development throws up countless choices, and filtering them well takes a lot of studying.

One of the hardest parts, he said, is how much ground you have to cover. Call it a foundation model and you're under pressure to be good at almost everything. Just collecting physics data for the specialized physics model the questioner is building is hard. Now imagine doing that across every domain. You need more people, more resources, and more work, and dividing it up and organizing it systematically is hard. Model experts, linguists, service developers, full-stack engineers, kernel engineers to make training fast, and specialists for each of the software stacks used to train a foundation model all have to work together, and that's where it gets difficult.

You only look at the true edge cases

There was also a question about the emphasis on human in the loop and evaluation. People are a finite resource and even one person's consistency can vary. You can track progress on benchmark metrics, but how did you get confidence in the accuracy of human evaluation and that you're really on track, and what did you ask of the evaluation team?

He said there are various techniques, like shuffling the order or having several people vote by majority. More important was how you build the data. It matters to build the data first around problems that sit right on a very ambiguous line, whether that's about nuance or about whether an answer is actually bad. You raised scalability, he said, and putting people in the loop does hurt scalability, so to do it well you only look at the cases that are truly on the edge.

So they worked hard to cut the numbers down and make the loop run fast. He said they're putting a lot of effort into making the stages flow smoothly from one task to the next, with information passing between them organically.

Complex physical infrastructure that runs itself

The last session was Ahn Hong-jun from Trillion Labs. He opened by saying it was his first talk in front of such a large community and thanked us for the honor of the invitation.

Trillion Labs is a foundation model company in Korea working hard to keep up with frontier models. It's getting enough attention to be named as a Korean regional AI at NVIDIA GTC. They've open-sourced 7B, 21B, and 70B dense models, and releasing a 70-billion-parameter model along with its intermediate training checkpoints was a first in Korea. Trida was built on the side to develop tech stacks like diffusion LLMs, and Tri-21B-Think, released earlier this year, ranked in the top 30 on the Artificial Analysis Intelligence Index. What they're building now is Gravity, an Industry Foundation Model series aimed at AI factories, and he said it's much better than those.

The talk centered on the future they're aiming for in the end: AI agents that can truly self-drive complex physical infrastructure. Picture an agent with a bird's-eye view of a facility investigating incidents, reasoning about causes, and coordinating responses. He walked through, from hands-on experience, what they're wrestling with to build that, how they define and solve the problem, and the technical pieces that went into it.

If you can't read the drawings, you can't do anything on site

They split the tasks this model has to understand and solve into four. One is understanding the outline of the whole infrastructure, which means reading P&IDs (piping and instrumentation diagrams), very large drawings of how the modules connect. Two is understanding data from the many sensors installed across the infrastructure to keep failures from blowing up. Three is understanding the strict manuals and complex module documents that exist, because you can't run things by guesswork when something goes wrong. Four is understanding time series data built up from the sensors over long periods. They're developing the model to understand all of these tasks together.

He said it's actually being deployed at a 1.2 gigawatt thermal power plant. It's a copilot that has to understand a complex system, detect problems when sensor values suddenly stop coming in or spike, and go further to recommend actions to fix them.

To define the problem, they built an internal benchmark called PowerBench. He explained the P&ID understanding task within it. The P&ID is the plant's ground truth. Questions on site get answered on the drawing, and the manuals and text don't have the answer.

If you can't read the drawing, you can't act in the plant.

The tasks fall into three types. Connectivity and topology asks things like which valves you need to close to isolate P-101B from the discharge header. Symbol and tag reading asks what the tag is on the reflux drum's relief valve. Safety and process logic asks which interlock trips on high-high level in V-201. An LLM judge grades answers against annotated ground truth, and that score becomes the RL reward directly.

Reward can't reach a policy that can't read

Why the training pipeline has to go in that order was the clearest explanation of the night. The base VLM has never seen P&ID conventions. RL only sharpens what the policy can already sample, so rollouts from a policy that can't read the drawings never reach the reward. So they raise the prior first and run RL after.

Mid-training gets full-resolution drawings and crops of them, standards and engineering documents that explain how to read the drawings, and QA sets grounded in the drawings. SFT has two layers: QA-format SFT sets answer style and output format, and agent trajectory SFT adds zoom traces to create a cold start for RL. The division of labor is mid-training for symbols, SFT for format and the tool loop, and RL for where to look.

Feed in the whole sheet and you stall at prefill

He said he was skeptical at first about why they needed to design a harness. Couldn't they just hand over the drawing and let the model solve it? But when they gathered real drawing data, the drawings were close to A0 size at 16 megapixels. Feed in something that size all at once and prefill alone takes forever. Downscale to 4 megapixels and it fits the budget, but local information like tags and symbols blurs and disappears entirely.

So they kept the full overview and let the model zoom when needed. The action space has zoom, which re-renders a region at native resolution and returns a readable crop, and answer, which ends the episode. The model can zoom again inside a crop. When a query comes in about a problem and what to do, the model zooms step by step, hunting for the components likely behind the problem, and then gives its answer. Episodes are variable length, with the agent deciding where to look and how often, and RL runs over the whole trajectory so the policy learns where to look. The objective is in the GRPO family and the reward is an LLM judge.

GPU utilization from 30 percent to 80 percent

For large-scale RL they went asynchronous. The reason synchronous doesn't scale was clear. Agent rollouts have a long tail, so every step waits for the slowest one, and then everything stops again for the weight sync. The trainer sits idle while rollouts run, and the rollout servers sit idle while training runs. GPUs ran at around 30 percent, and it got worse as the number of turns grew.

In async, workers stream trajectories nonstop and the learner trains without waiting. The only thing that stops the GPUs is the weight sync. The key number here is max_head_offpolicyness, how many versions behind the learner a trajectory is allowed to be. People usually set it to 1 or 2, but they found a recipe that pushes it to 8 and training still stayed stable. Throughput rose more than 6x over synchronous, and GPU utilization went from 30 percent to 80 percent. The bubbles where everything waited on rollouts were gone.

They removed clipping and training still worked

Someone asked how they compensated, since it seemed like training for a long time with offpolicyness at 8 could break the model. He said they've never run 2,000 or 3,000 steps at a time, but it has never broken. That wasn't from looking at failure cases and fixing them. It came from trimming the algorithm down a lot in their own way.

First, they removed the clipping usually used with PPO-based algorithms. Training worked fine without it, so they took it out. More important was capping the inference sampling ratio to handle off-policy data. If something looks like it's about to blow up, they mask it out so it doesn't contribute to training. They also have to balance the trainer and inference so neither side sits idle. Even with a max of 8, the actual average is around 3 point something or 4 point something, so keeping that ratio steady mattered. Internally they confirmed it runs fine even at 12, but 8 gave the best wall-clock time, so that's what they use.

Honestly, at this point, I don't know

The last question reached the farthest edge of the talk. Someone from a sound AI startup noted that the public materials mention inputs like design drawings, time series data, and maintenance history. But in data centers and large power facilities, there's a window where warning signs show up first as sound or vibration, before they register in telemetry. How would the world model handle that window?

Ahn paused and called it a really good question. He said he'd talked a lot with the people on the ground during this project, and to realize the big vision, they'd eventually need a world model that can predict that kind of data too. Then he gave a candid answer.

Honestly, at this point, I don't know. That may sound like a lazy answer, but right now I really don't have a feel for it.

He said they could probably get as far as understanding it, by somehow synthesizing problems tied to that data. But finding earlier leading indicators, predicting from them, and then acting is a very hard problem. It stuck with me more because it came from someone actually solving the problem.

What's the biggest barrier to bringing AI into an organization?

We wrapped up with small group discussions, the highlight of any Bloom event. Groups of four to six shared their name, where they work, and why they came, then kept talking. The biggest laugh came from how we picked group leaders: whoever spends the most of their own money on AI leads the group, and company cards don't count.

The Artificial Analysis team prepared the questions. What's the biggest barrier when bringing AI into an organization and scaling it? What matters most for AI progress over the next few years? What new AI use cases are you seeing lately? And what's the most interesting recent AI news? We told people they didn't need to cover all four and could start with whichever pulled them in.

We asked each group to post their takeaways in the #event-recap channel on Discord, and people stayed after the event to keep talking. It was a night with the team that builds the benchmark, the teams that build the models on it, and the people who choose and use those models all in one room. I'm grateful we got to host Artificial Analysis's first event in Korea, and I hope George can make it in person next time.

Originally written in Korean on Brunch, September 17, 2026.