Skip to main content

OpenAI says it found widespread task issues in SWE-Bench Pro, estimates ~30% of tasks are broken, and retracts its earlier recommendation to adopt the benchmark (OpenAI)

OpenAI:
OpenAI says it found widespread task issues in SWE-Bench Pro, estimates ~30% of tasks are broken, and retracts its earlier recommendation to adopt the benchmark  —  Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.



from Techmeme https://ift.tt/ktOv2pC
via IFTTT
Labels:

Comments

Post a Comment

Popular posts from this blog

How Telegram, using Google and Amazon's cloud services, humiliated and outmaneuvered Russia's state telecom regulator, Roskomnadzor, and survived the ban (Isabelle Khurshudyan/Washington Post)

Isabelle Khurshudyan / Washington Post : How Telegram, using Google and Amazon's cloud services, humiliated and outmaneuvered Russia's state telecom regulator, Roskomnadzor, and survived the ban   —  MOSCOW — The man who took on the Kremlin and won is rarely heard from: a tech nomad in self-imposed exile with a reclusive aura … source http://www.techmeme.com/200628/p7#a200628p7
Post a Comment
Read more

Microsoft brings SwiftKey back to the iOS App Store citing "customer feedback", after delisting the keyboard app on October 5, 2022 (Jay Peters/The Verge)

Jay Peters / The Verge : Microsoft brings SwiftKey back to the iOS App Store citing “customer feedback”, after delisting the keyboard app on October 5, 2022   —  Microsoft's SwiftKey keyboard has unexpectedly returned to the App Store.  The company officially discontinued support for the keyboard and removed … from Techmeme https://ift.tt/tV0BEvC via IFTTT
Post a Comment
Read more

Spotify cites Edison Research and its own internal data to claim it "recently became" the most-used podcast platform in the US and says it has 3.2M podcasts (Sarah Perez/TechCrunch)

Sarah Perez / TechCrunch : Spotify cites Edison Research and its own internal data to claim it “recently became” the most-used podcast platform in the US and says it has 3.2M podcasts   —  In September, analysts at eMarketer predicted Spotify was poised to overtake Apple Podcasts in U.S. listenership sometime this year. from Techmeme https://ift.tt/2ZAdoDU via IFTTT
Post a Comment
Read more