Instead of working like a novelist or an academic to write Who Makes the NBA?, Stephens-Davidowitz worked like one of us: Against a tight deadline, worried more about directional correctness than scientific rigor, and responsible for not just producing his analysis but also presenting it. (View Highlight)
The biggest conclusion—for better or for worse, I suppose—is that AI is a long way from replacing us. When instructing ChatGPT to do its works, Stephens-Davidowitz found that “clarity and precision are paramount” and “any ambiguity can lead to unintended consequences.” This suggests that, at least when doing open-ended analysis on new datasets,9 it’s important to be analytically fluent. ChatGPT can port the Tidyverse from R to English—but it still needs to be talked to using the vocabulary of the Tidyverse. (View Highlight)
None of this is terribly surprising; these sorts of shortcomings are well-documented at this point. As I wrote earlier this year—and as Andrej Karpahty, one of OpenAI’s leading researchers, said in a recent tweet—the randomness inherent in LLMs make them undisciplined order-takers. Instead, I though, we should treat them like creative but unruly prima donnas: (View Highlight)
One of the most striking things about LLMs is that they’re much better at the creative parts of analysis than they are at the mechanical parts. Ask ChatGPT to write a SQL query against an artificially simple schema; it’s a junior analyst, at best. But ask it to come up with possible hypotheses to explain why there’s some anomaly in a metric, and it does better than I would. … (View Highlight)
I’m not sure that I want to say that all the query-writing chatbots should pivot into analytical reasoning bots. That’s the job I want, to be at the top of the creative food chain with an army of machines doing the mundane tedium below me. But the army we’re building—creative, unpredictable, and equally prone to both novel ideas and lies—is becoming more man than machine. If we’re looking for chatbots that make us all better at working with data, without having to rebuild an entire BI tool in the process, our best option may be to let them do the job we wanted to do all along. (View Highlight)
AI, for me, just eliminates all the parts of my job or work process that I hate. I hate cleaning datasets, merging datasets. … It does all of that for you and frees you to be creative and ask the questions. (View Highlight)
So what do all of us need to learn? The answer, it seems, is nothing in particular. We just need to start using tools like ChatGPT, and, as we all did with Google, develop an intuition for how they work. Though Stephens-Davidowitz lists a number of specific tips and tricks in the book’s appendix, he ultimately concluded that the best way to make the most of ChatGPT is just to “practice with it.” (View Highlight)
Every analyst who’s ever had to work quickly should sympathize with this. If not for other people’s reactions, how do you know if your result is crazy? The numbers add up; the reasoning makes sense. Analysis isn’t an irrefutable science, but a kind of quantitative logical argument. Some ideas make more sense than others, but until we test them—assuming we even can—we won’t know if the rock actually floats or not. (View Highlight)
