Article
The rules for using AI to do research
bernoff.com
Quoted on this wiki
Every place a page here uses this source, in the order the words come in it.
AI is not a good tool for primary research. Primary research is research intended to generate new facts, such as surveys, scientific experiments, data analysis, and interviews. Large language models like ChatGPT operate on existing content, rather than generating new facts. “This is what we used to call Web research (or if you’re old enough, library research): finding, aggregating, and citing primary content that other people have already created.” Google and its direct search competitors were an incredibly blunt instrument for secondary research. Google searches (at least until the AI summaries started to appear) were limited to the exact keywords you used. Worse, SEO practitioners gamed the algorithm so that the top search results were not the most useful, but those that were most skilled at playing the game.
Why AI is popular for research “Large language models like ChatGPT operate on existing content, rather than generating new facts.” But secondary research is still essential. This is what we used to call Web research (or if you’re old enough, library research): finding, aggregating, and citing primary content that other people have already created. If you’ve ever searched for a statistic online, you’ve done secondary research. (It’s also, sadly, what people mean when they tell you not to believe the conventional wisdom and to “do your own research,” which boils down to finding stuff that reinforces your existing point of view.)
Why AI is popular for research “Primary research is research intended to generate new facts, such as surveys, scientific experiments, data analysis, and interviews.” But secondary research is still essential. This is what we used to call Web research (or if you’re old enough, library research): finding, aggregating, and citing primary content that other people have already created. If you’ve ever searched for a statistic online, you’ve done secondary research. (It’s also, sadly, what people mean when they tell you not to believe the conventional wisdom and to “do your own research,” which boils down to finding stuff that reinforces your existing point of view.)
Google and its direct search competitors were an incredibly blunt instrument for secondary research. Google searches (at least until the AI summaries started to appear) were limited to the exact keywords you used. Worse, SEO practitioners gamed the algorithm so that the top search results were not the most useful, but those that were most skilled at playing the game. “Like Google, Perplexity will generate as series of links with its answers.” ChatGPT and its competitors can do the same thing, but are more focused on generating answers than links unless you tell them to behave more as a search tool. Google search now includes Gemini AI, that does pretty much the same thing.
Google and its direct search competitors were an incredibly blunt instrument for secondary research. Google searches (at least until the AI summaries started to appear) were limited to the exact keywords you used. Worse, SEO practitioners gamed the algorithm so that the top search results were not the most useful, but those that were most skilled at playing the game. “And you can engage it in a dialogue, telling it to generate answers that are successively more useful to you.” ChatGPT and its competitors can do the same thing, but are more focused on generating answers than links unless you tell them to behave more as a search tool. Google search now includes Gemini AI, that does pretty much the same thing.
Google and its direct search competitors were an incredibly blunt instrument for secondary research. Google searches (at least until the AI summaries started to appear) were limited to the exact keywords you used. Worse, SEO practitioners gamed the algorithm so that the top search results were not the most useful, but those that were most skilled at playing the game. “It’s takes off from what you write, cognitively mapping it to other similar meanings, not just the literal words you use.” ChatGPT and its competitors can do the same thing, but are more focused on generating answers than links unless you tell them to behave more as a search tool. Google search now includes Gemini AI, that does pretty much the same thing.
Why AI is popular for research “AI is not a good tool for primary research.” But secondary research is still essential. This is what we used to call Web research (or if you’re old enough, library research): finding, aggregating, and citing primary content that other people have already created. If you’ve ever searched for a statistic online, you’ve done secondary research. (It’s also, sadly, what people mean when they tell you not to believe the conventional wisdom and to “do your own research,” which boils down to finding stuff that reinforces your existing point of view.)
Find sources. AI tools like Perplexity make this much more efficient. But it’s important to note that effective research means using AI to find sources, not believing what AI tells you as a summary of what “the web” as a whole says. “But in AI summaries, it’s very hard to tell the difference between a number from a legitimate, dependable source or one pulled from a random site or completely made up.” Understand what sources say. AI is excellent at summarizing content. But here again, it can make mistakes and show you what you want to see. So it’s effective to use content summaries to help get the gist of long papers, for example, but before citing what the AI has shown you, you have to check that it’s an accurate description if what the paper says. (Papers have abstracts, which function as far more effective, presumably human-generated, summaries.)
Find sources. AI tools like Perplexity make this much more efficient. But it’s important to note that effective research means using AI to find sources, not believing what AI tells you as a summary of what “the web” as a whole says. “This makes it even more imperative that researchers verify the links and content of what they’re finding.” Understand what sources say. AI is excellent at summarizing content. But here again, it can make mistakes and show you what you want to see. So it’s effective to use content summaries to help get the gist of long papers, for example, but before citing what the AI has shown you, you have to check that it’s an accurate description if what the paper says. (Papers have abstracts, which function as far more effective, presumably human-generated, summaries.)
Here’s how AI changes those steps: “But it’s important to note that effective research means using AI to find sources, not believing what AI tells you as a summary of what “the web” as a whole says.” Vet sources. AI hallucinates sources (I’ve seen it generate links to New York Times articles that weren’t real). Where there are no definitive sources, it’s quite likely to pull information from random Reddit posts or biased sites full of false information. This was always a problem: Web searches had many of the same drawbacks. But in AI summaries, it’s very hard to tell the difference between a number from a legitimate, dependable source or one pulled from a random site or completely made up. This makes it even more imperative that researchers verify the links and content of what they’re finding. Don’t just quote AI, it could very well be wrong.
Find sources. AI tools like Perplexity make this much more efficient. But it’s important to note that effective research means using AI to find sources, not believing what AI tells you as a summary of what “the web” as a whole says. “Don’t just quote AI, it could very well be wrong.” Understand what sources say. AI is excellent at summarizing content. But here again, it can make mistakes and show you what you want to see. So it’s effective to use content summaries to help get the gist of long papers, for example, but before citing what the AI has shown you, you have to check that it’s an accurate description if what the paper says. (Papers have abstracts, which function as far more effective, presumably human-generated, summaries.)
Why AI is popular for research “AI is not a good tool for primary research.” But secondary research is still essential. This is what we used to call Web research (or if you’re old enough, library research): finding, aggregating, and citing primary content that other people have already created. If you’ve ever searched for a statistic online, you’ve done secondary research. (It’s also, sadly, what people mean when they tell you not to believe the conventional wisdom and to “do your own research,” which boils down to finding stuff that reinforces your existing point of view.)
Here’s how AI changes those steps: “But it’s important to note that effective research means using AI to find sources, not believing what AI tells you as a summary of what “the web” as a whole says.” Vet sources. AI hallucinates sources (I’ve seen it generate links to New York Times articles that weren’t real). Where there are no definitive sources, it’s quite likely to pull information from random Reddit posts or biased sites full of false information. This was always a problem: Web searches had many of the same drawbacks. But in AI summaries, it’s very hard to tell the difference between a number from a legitimate, dependable source or one pulled from a random site or completely made up. This makes it even more imperative that researchers verify the links and content of what they’re finding. Don’t just quote AI, it could very well be wrong.
Vet sources. AI hallucinates sources (I’ve seen it generate links to New York Times articles that weren’t real). Where there are no definitive sources, it’s quite likely to pull information from random Reddit posts or biased sites full of false information. This was always a problem: Web searches had many of the same drawbacks. But in AI summaries, it’s very hard to tell the difference between a number from a legitimate, dependable source or one pulled from a random site or completely made up. This makes it even more imperative that researchers verify the links and content of what they’re finding. Don’t just quote AI, it could very well be wrong. “But here again, it can make mistakes and show you what you want to see.” Analyze and synthesize facts and content. This is the core of what researchers do. It’s a fundamentally human activity, combining insight and judgment with existing content. While you can engage an AI in dialogue to examine different approaches to synthesizing content, it’s a fatal mistake to outsource this intellectual activity to a machine.
Understand what sources say. AI is excellent at summarizing content. But here again, it can make mistakes and show you what you want to see. So it’s effective to use content summaries to help get the gist of long papers, for example, but before citing what the AI has shown you, you have to check that it’s an accurate description if what the paper says. (Papers have abstracts, which function as far more effective, presumably human-generated, summaries.) “While you can engage an AI in dialogue to examine different approaches to synthesizing content, it’s a fatal mistake to outsource this intellectual activity to a machine.” Cite facts and content. This is where you link to what you found. AI can help format the citations, but which ones you include will reflect the quality of your thinking as a researcher.
Understand what sources say. AI is excellent at summarizing content. But here again, it can make mistakes and show you what you want to see. So it’s effective to use content summaries to help get the gist of long papers, for example, but before citing what the AI has shown you, you have to check that it’s an accurate description if what the paper says. (Papers have abstracts, which function as far more effective, presumably human-generated, summaries.) “While you can engage an AI in dialogue to examine different approaches to synthesizing content, it’s a fatal mistake to outsource this intellectual activity to a machine.” Cite facts and content. This is where you link to what you found. AI can help format the citations, but which ones you include will reflect the quality of your thinking as a researcher.
The right way to do research with AI tools “AI tools may increase efficiency, but they also increase risk.” But that doesn’t make AI tools useless. In fact, one doctoral student I talked to said that AI tools had saved her years of work in reviewing existing research content in her field. It’s no coincidence that the heaviest users of AI in my survey had $47,000 higher incomes than the nonusers. Busy, highly paid writers need to be efficient.
In Tools and AI