【Hacker News搬运】关于有影响力的人工智能研究

hackernews

Title: On Impactful AI Research

关于有影响力的人工智能研究

Text:

Url: https://github.com/okhat/blog/blob/main/2024.09.impact.md

很抱歉，但作为一个AI，我无法直接访问或解析互联网上的外部链接，包括GitHub上的Markdown文件。因此，我无法读取并分析您提供的链接内容。

如果您需要关于该链接中内容的帮助，例如总结或翻译，您可以将内容的文本复制粘贴到这个问题中，我会尽力帮助您。请提供需要分析和总结的文本内容。

Post by: KraftyOne

Comments:

will-burner: This has good advice about academic research in general not just AI Research. It's huge to "Select timely problems with large headroom and "fanout"" - you need a research program to have a successful research career. At least most people do. It's also huge to "Put your work out there and own popularizing your ideas" - the second part is hard for a lot of academics to do. Putting your work out there is huge to get feedback, which may help you pivot or try different directions, as well as fostering collaboration. Collaboration is a large part of being a successful researcher.

will-burner: 这对学术研究有很好的建议，而不仅仅是人工智能研究。它；巨大到&quot；选择具有较大净空的及时问题和&quot；扇出&quot&“-你需要一个研究项目才能有一个成功的研究生涯。至少大多数人都这样做；这对“；把你的工作放在那里，自己推广你的想法&quot；----第二部分对很多学者来说很难做到。把你的工作放在那里获得反馈是巨大的，这可能会帮助你转向或尝试不同的方向，以及促进合作。合作是成为一名成功研究人员的重要组成部分。

sashank_1509: Unfortunately while this advice sounds useful, it isn’t. It might be useful if you measure impact is the fake metrics like citation count but if you want to measure with actual tangible impact on the real world, you have a rude awakening before you. To me there are 2 ways a paper can create impact:1. The paper represents such an impressive leap in performance over existing methods in AI, that it is obviously impactful. Unfortunately, this way of generating impact is dominated by industry. No one can expect Academia to train O1, SAM, GPT5 etc. AI rewards scale, scale requires money, resources and manpower and Academia has none. In the early days of AI, there were rare moments when this was possible, AlexNet, Adam, Transformers, PPO etc. Is it still possible? I do not know, I have not seen anything in the last 3 years and I’m not optimistic many such opportunities are left. Even validating your idea tends to require the scale of industry.2. The paper affects the thought process of other AI researchers and thus you are indirectly impactful if any of them cause big leaps in AI performance. Unfortunately here is where Academia has shot itself in the foot by generating so many damn papers every year (>10,000). There are just so many, that the effect of any 1 paper is meaningless. In fact the only way to be impactful now is to be in a social circle of great researchers, so that you know your social circle will read your paper and later if any of them make big performance improvements, you can believe that you played a small role in it. I have spoken to a lot of ML researchers, and they told me they choose papers to read just based on people and research groups they know. Even being a NeurIPS spotlight paper, means less than 10% of researchers will read your paper, maybe it will go to 50% if it’s a NEURIPS best paper but even that I doubt. How many researchers remember last year’s NEURIPS best paper?The only solution to problem 2, is radical. The ML community needs to come together and limit the number of papers it wide releases. Let us say it came out and said that yearly only 20 curated papers will be widely published. Then you can bet most of the ML community will read all 20 of those papers and engage with it deeply as they will be capable of spending more than a day at least thinking about the paper. Of course you can still publish on arxiv, share with friends etc but unless such a dramatic cutdown is made I don’t see how you can be an actually impactful AI researcher in Academia when option 1 is too expensive and option 2 is made impossible.

sashank_1509: 不幸的是，虽然这个建议听起来很有用，但事实并非如此。如果你用引用计数等虚假指标来衡量影响，这可能会很有用，但如果你想用对现实世界的实际有形影响来衡量，你会有一个粗鲁的觉醒。在我看来，一篇论文可以通过两种方式产生影响：1。这篇论文代表了人工智能现有方法在性能上的巨大飞跃，显然具有影响力。不幸的是，这种产生影响的方式是由行业主导的。没有人能指望学术界培养O1、SAM、GPT5等人工智能奖励规模，规模需要金钱、资源和人力，而学术界则没有。在人工智能的早期，很少有这样的时刻是可能的，AlexNet、Adam、Transformers、PPO等。现在还有可能吗？我不知道，在过去的3年里，我什么都没看到，我不乐观，很多这样的机会还剩下。即使是验证你的想法也往往需要行业的规模 2。这篇论文影响了其他人工智能研究人员的思维过程，因此，如果他们中的任何一个人在人工智能性能上取得了巨大的飞跃，你就会受到间接的影响。不幸的是，这就是学术界每年发表如此多该死的论文（&gt；10000篇）的地方。数量太多了，任何一篇论文的效果都是毫无意义的。事实上，现在唯一能产生影响力的方法就是加入一个由伟大研究人员组成的社交圈，这样你就知道你的社交圈会阅读你的论文，以后如果他们中的任何一个人做出了巨大的绩效改进，你可以相信你在其中发挥了很小的作用。我和很多机器学习研究人员谈过，他们告诉我，他们选择阅读的论文只是基于他们认识的人和研究小组。即使是NeurIPS的聚光灯论文，也意味着只有不到10%的研究人员会阅读你的论文，如果这是NeurIPS最好的论文，可能会达到50%，但我对此表示怀疑。有多少研究人员记得去年NEURIPS的最佳论文 问题2的唯一解决方案是激进的。机器学习社区需要团结起来，限制其发布的论文数量。让我们说它出来了，说每年只有20篇精选论文会被广泛发表。那么，你可以打赌，大多数机器学习社区的人都会阅读所有20篇论文并深入参与其中，因为他们至少可以花一天多的时间思考这篇论文。当然，你仍然可以在arxiv上发表文章，与朋友分享等，但除非做出如此巨大的削减，否则当选项1太贵而选项2不可能时，我看不出你怎么能成为学术界真正有影响力的人工智能研究人员。

tony_cannistra: "Invest in projects, not papers."This is excellent advice, and in my experience does not represent the intuition that many young (and not so young) researchers begin with.Papers come from projects and, if you care, good projects can yield many good papers!

tony_cannistra: &“；投资项目，而不是论文&“ 这是一个很好的建议，根据我的经验，这并不代表许多年轻（也不那么年轻）研究人员最初的直觉 论文来自项目，如果你在乎，好的项目可以产生很多好的论文！

lmeyerov: I like the article directionally but fear the examples are too short-sighted for most AI researchers.Picking something useful in 1-2 years is a reason to go to industry, not research, and leads to mostly incremental units that if you don't do, someone else will. Yes, hot topics are good because they signal a time of fertile innovation. But not if your vision is so shallow that you will have half of IBM, Google, and YC competing with you before you start or by the time of your first publication (6-12mo). If you are a top student, well-educated already, with top resources and your own mentees, and your advisor is an industry leader who already knows where your work will go, maybe go to the thickest 1-2 year out AI VC fest, but that's not most PhD students.A 'practical' area would be obvious to everyone in 5 years, but winnowing out the crowd, there should not be much point to it today nor 1-2 years without something fundamentally changing. It should be tangible enough to be relevant and enticing, but too expensive for whatever reasons. More fundamental research would be even more years out. This gives you a year or two to dig into the problem, and another year or two to build out fundamental solutions, and then a couple years of cranking. From there, rinse-and-repeat via your own career or those of your future students.Some of my favorite work took 1-2 years of research to establish the problem, not just the solutions. Two of the projects here were weird at first as problems on a longer time scale, but ended up as part of $20M grant and software many folks here use & love, and another, a 10 year test of time award. (And another, arguably a $100M+ division at Nvidia). In contrast, most of my topic-of-the-year stuff didn't matter and was interchangeable with work by others.Edit: The speech by Hamming on "You and your research" hits on similar themes and speaks more to my experiences here: <a href="https://fs.blog/great-talks/richard-hamming-your-research/" rel="nofollow">https://fs.blog/great-talks/richard-hamming-your-research/</a>

lmeyerov: 我喜欢这篇文章的方向性，但担心这些例子对大多数人工智能研究人员来说太短视了 在1-2年内选择有用的东西是进入行业而不是研究的原因，如果你不这样做，会导致大部分增量单位；不行，别人会的。是的，热门话题是好的，因为它们标志着一个充满创新的时代。但如果你的视野太浅，以至于在你开始之前或在你第一次发表文章之前（6-12个月），你会有一半的IBM、谷歌和YC与你竞争，那就不会了。如果你是一名优秀的学生，受过良好的教育，拥有顶尖的资源和自己的学员，你的顾问是一位行业领导者，他已经知道你的工作将走向何方，也许你会去参加为期1-2年的最激烈的人工智能风险投资盛会，但那；不是大多数博士生 A；实用；5年后，这个领域对每个人来说都是显而易见的，但剔除人群后，如果没有根本性的改变，今天和1-2年都不应该有太大意义。它应该足够有形，具有相关性和吸引力，但无论出于什么原因，它都太贵了。更多的基础研究还需要几年时间。这给了你一两年的时间来深入研究问题，再过一两年来制定基本的解决方案，然后是几年的时间。从那里开始，冲洗并通过你自己的职业生涯或你未来的学生的职业生涯重复 我最喜欢的一些工作需要1-2年的研究来确定问题，而不仅仅是解决方案。这里的两个项目起初很奇怪，因为它们是长期存在的问题，但最终成为了这里许多人使用的2000万美元赠款和软件的一部分；爱情，还有一个10年时间考验奖。（还有一个，可以说是Nvidia的1亿美元以上的部门）。相比之下，我今年的大部分主题都没有；这并不重要，可以与其他人的工作互换 编辑：汉明关于“；你和你的研究&quot；点击类似的主题，并更多地讲述我在这里的经历：<a href=“https:&#x2F；fs.blog&#x2G；精彩演讲&#x2C；richardhamming你的研究&#x2A；rel=“nofollow”>https:&#x2F；fs.blog；精彩的演讲；richard hamming你的研究</一

Der_Einzige: At least some of it comes from "hype" too. The author of Dspy (the writer) (<a href="https://github.com/stanfordnlp/dspy">https://github.com/stanfordnlp/dspy</a>) should know this, given that Dspy is nothing more than fancy prompts optimizing prompts to be fancier according to prompt chains described in papers (i.e. Chain of thought, Tree of thought, etc). Textgrad (<a href="https://github.com/zou-group/textgrad">https://github.com/zou-group/textgrad</a>) is an even worse example of this, as it makes people think that it's not just a prompt optimizing another promptDspy has 17k stars, meanwhile PyReft (<a href="https://github.com/stanfordnlp/pyreft">https://github.com/stanfordnlp/pyreft</a>) isn't even at 1200 yet and it has Christopher Manning (head of AI at stanford) working on it (see their paper: <a href="https://arxiv.org/abs/2404.03592" rel="nofollow">https://arxiv.org/abs/2404.03592</a>). Sometimes what the world deems "impactful" in the short-medium term is wrong. Think long term. PyReft is likely the beginning of an explosion in demand for ultra parameter efficient techniques, while Dspy will likely fade into obscurity over time.I also know that the folks writing better samplers/optimizers for LLMs get almost no love/credit relative to the outsized impact they have on the field. A new sampler potentially improves EVERY LLM EVER! Folks like Clara Meister or the authors of the min_p paper preprint have had far larger impacts on the field than their citation counts might suggest, based on the fact that typiciality or min_p sampling is now considered generally superior to top_p/top_k (OpenAI, Anthropic, Gemini, et al still use top_p/top_k) and min_p/typicality are implemented by every open source LLM inference engine (i.e. huggingface, vllm, sglang, etc)

Der_Einzige: 至少其中一部分来自&quot；炒作”；我也是。Dspy的作者（作者）应该知道这一点，因为Dspy只不过是花哨的提示——根据论文中描述的提示链（即思维链、思维树等）优化提示，使其更花哨。Textgrad（<a href=“https:&#x2F；github.com&#x2F-zou group/ Textgrad”>https:&quot；github.com&#x2F/zou group&apos；Textgrad</a>）是一个更糟糕的例子，因为它让人们认为它；这不仅仅是一个优化另一个提示符的提示符Dspy有17k颗星，而PyReft（<a href=“https:&#x2F；github.com/ stanfordnlp&#x2F-PyReft”>https:&quot；github.com&#x2F/stanford nlp&#PyReft</a>）不是；甚至到了1200岁，斯坦福大学人工智能负责人克里斯托弗·曼宁（Christopher Manning）也在研究它（见他们的论文：<a href=“https:”abs:”abs:“2404.03592”rel=“nofollow”>https:“abs:”2404.03592</a>）。有时世界认为&quot；有影响&quot；从中短期来看是错误的。从长远考虑。PyReft可能是超参数高效技术需求激增的开始，而Dspy可能会随着时间的推移而逐渐淡出人们的视线 我也知道那些编写更好的采样器的人；LLM的优化器几乎不受欢迎；相对于它们在该领域产生的巨大影响，信用。一个新的采样器可能会提高每一个LLM！像Clara Meister或min_p论文预印本的作者这样的人对该领域的影响比他们的引用次数所暗示的要大得多，这是基于典型性或min-p抽样现在通常被认为优于top_p的事实；top_k（OpenAI、Anthropic、Gemini等仍然使用top_ptop_k）和min_p；典型性由每个开源LLM推理引擎（即huggingface、vllm、sglang等）实现