1.1.2 Research
As the research community takes the “bigger, better” approach, new models often require a massive amount of data and tens of millions of dollars in computing. The estimated market cost to train DeepMind’s AlphaStar and OpenAI’s GPT-3 is in the tens of millions each45. Most companies and academic institutions can’t afford to pursue pure research.
Outside academic institutions, there are only a handful of machine learning research labs in the world. Most of these labs are funded by corporations with deep pockets such as Alphabet (Google Brain, DeepMind), Microsoft, Facebook, Tencent6. You can find these labs by browsing the affiliations of published papers at major academic conferences including NeurIPS, ICLR, ICML, CVPR, ACL. In 2019 and 2020, Alphabet accounts for over 10% of all papers at NeurIPS[1, 2].
🌳 Tip 🌳
Not all these industry labs publish papers -- companies like Apple and Tesla are notoriously secretive. Even if an industry lab publishes, it might only publish a portion of its research. Before joining an industry lab, you might want to consider its publishing policy. Joining a secretive lab might mean that you won’t be able to explain to other people what you’ve been working on or what you’re capable of doing.
4: State of AI Report 2019 by Nathan Benaich and Ian Hogarth.
5: OpenAI’s massive GPT-3 model is impressive, but size isn’t everything by VentureBeat.
6: In an earlier draft of this book, I included Uber AI and Element AI. However, Uber AI research lab was laid off in April 2020, and Element AI was sold for cheap in November 2020.