我昨天通过早期体验试用了 Gemini 3,有几点想法—— 首先,我通常会对公开基准测试保持谨慎,因为在我看来,这些测试相当有可能被“针对性优化”。最终结果取决于团队的自律和克制力(而与此同时,他们在激励上往往倾向于相反方向),不要通过在文档嵌入空间对测试集邻近数据做各种复杂操作,从而对测试集进行过度拟合。现实情况是,因为别人都在这么做,所以这种压力非常大。 去和模型对话吧。也去和其他模型对话(“体验 LLM 循环” —— 每天用一个不同的大模型)。 我昨天的初体验是正面的:在个性、写作、整体氛围、代码、幽默感等方面都表现不错,作为日常主力模型有很大潜力,很明显是一线(tier 1)级别的大模型,恭喜团队! 接下来几天/几周,我最感兴趣、也最关注的是各种私有评测组成的“集成结果”(ensemble over private evals),现在很多个人和组织都开始自己构建这些评测,并偶尔在这里分享。 I played with Gemini 3 yesterday via early access. Few thoughts - First I usually urge caution with public benchmarks because imo they can be quite possible to game. It comes down to discipline and self-restraint of the team (who is meanwhile strongly incentivized otherwise) to not overfit test sets via elaborate gymnastics over test-set adjacent data in the document embedding space. Realistically, because everyone else is doing it, the pressure to do so is high. Go talk to the model. Talk to the other models (Ride the LLM Cycle - use a different LLM every day). I had a positive early impression yesterday across personality, writing, vibe coding, humor, etc., very solid daily driver potential, clearly a tier 1 LLM, congrats to the team! Over the next few days/weeks, I am most curious and on a lookout for an ensemble over private evals, which a lot of people/orgs now seem to build for themselves and occasionally report on here.

Andrej KarpathyFounder, Eureka Labs AI

观点原文

我昨天通过早期体验试用了 Gemini 3,有几点想法——

首先,我通常会对公开基准测试保持谨慎,因为在我看来,这些测试相当有可能被“针对性优化”。最终结果取决于团队的自律和克制力(而与此同时,他们在激励上往往倾向于相反方向),不要通过在文档嵌入空间对测试集邻近数据做各种复杂操作,从而对测试集进行过度拟合。现实情况是,因为别人都在这么做,所以这种压力非常大。

去和模型对话吧。也去和其他模型对话(“体验 LLM 循环” —— 每天用一个不同的大模型)。

我昨天的初体验是正面的:在个性、写作、整体氛围、代码、幽默感等方面都表现不错,作为日常主力模型有很大潜力,很明显是一线(tier 1)级别的大模型,恭喜团队!

接下来几天/几周,我最感兴趣、也最关注的是各种私有评测组成的“集成结果”(ensemble over private evals),现在很多个人和组织都开始自己构建这些评测,并偶尔在这里分享。

I played with Gemini 3 yesterday via early access. Few thoughts -

First I usually urge caution with public benchmarks because imo they can be quite possible to game. It comes down to discipline and self-restraint of the team (who is meanwhile strongly incentivized otherwise) to not overfit test sets via elaborate gymnastics over test-set adjacent data in the document embedding space. Realistically, because everyone else is doing it, the pressure to do so is high.

Go talk to the model. Talk to the other models (Ride the LLM Cycle - use a different LLM every day). I had a positive early impression yesterday across personality, writing, vibe coding, humor, etc., very solid daily driver potential, clearly a tier 1 LLM, congrats to the team!

Over the next few days/weeks, I am most curious and on a lookout for an ensemble over private evals, which a lot of people/orgs now seem to build for themselves and occasionally report on here.

AI

相关观点