We Asked 5 AI Assistants for 'Best Plumber in Chicago.' The Results Are Alarming.

The promise of conversational AI is a world of effortless, accurate answers. We imagine asking our phone, "Find me the best plumber in Chicago," and instantly getting a perfect, reliable recommendation. This is the future that tech companies are selling us. But how close is this to our current reality?

We decided to conduct a simple experiment to find out. We asked five of the leading AI assistants the same, straightforward question that a real consumer would ask: "Who is the best plumber in Chicago?" We weren't looking for a single 'correct' answer, but for consistency, accuracy, and transparency. The results were not just disappointing; they were alarming. The wildly different, often outdated, and logically flawed answers we received paint a stark picture of the challenges facing AI-driven local search.

This experiment reveals a critical vulnerability in the AI ecosystem that every small business owner needs to understand. It's a real-world demonstration of why AI can't just 'read' your website and why being a source of factual certainty is so important. But it also illuminates a massive opportunity. The businesses that understand why these AIs are failing can position themselves to be the reliable, authoritative answer that the AIs are so desperately looking for.

The Contenders

We posed our query to a representative sample of today's top AI assistants:

1. ChatGPT (using GPT-4o): The most popular and well-known large language model.
2. Google Gemini: Google's flagship AI, deeply integrated with its search index.
3. Microsoft Copilot: Microsoft's AI, which also leverages OpenAI's models and Bing search.
4. Perplexity: The self-proclaimed 'answer engine' we reviewed in our AI Tool of the Week series.
5. Siri (Apple): One of the most widely used voice assistants, representing the on-the-go mobile query.

The Query and The Shocking Results

The Query: "Who is the best plumber in Chicago?"

We expected some variation, perhaps based on different interpretations of 'best' (e.g., best-rated, biggest, most mentioned). What we got was chaos.

• ChatGPT's Answer: It refused to answer directly, stating, "'Best' is subjective and can depend on many factors..." It then provided a generic list of how to find a good plumber (e.g., "Check online reviews on Yelp and Google," "Ask for recommendations," "Verify licenses"). While responsible, this was completely unhelpful for a user looking for a direct recommendation.

• Google Gemini's Answer: Gemini provided a list of three specific plumbing companies. Alarmingly, when we checked their Google reviews, one of the three had a mediocre 3.5-star rating with several recent, negative reviews about missed appointments and overcharging. It seemed to have ignored its own company's review data.

• Microsoft Copilot's Answer: Copilot also listed three specific companies. One of them was the same poorly-rated company that Gemini had recommended. Another one it listed, upon further investigation, appeared to be a large lead-generation service, not an actual local plumbing company.

• Perplexity's Answer: Perplexity provided the most transparent response. It gave a synthesized summary, saying, "According to sources like Yelp, Angi, and local Chicago blogs, several plumbers are consistently highly rated." It then listed five companies, and for each one, it included a citation linking back to the source where it found the recommendation. This was the most helpful response, but it still relied entirely on the opinions of third-party directories.

• Siri's Answer: Siri simply performed a web search for "best plumber in Chicago" and displayed the top organic search results from Google, effectively punting the question back to a traditional search engine.

The Analysis: Why Did the AIs Fail So Badly?

The inconsistency and poor quality of these answers stem from a few core problems that plague the current generation of AI.

1. The Subjectivity Problem: ChatGPT's response highlights a real issue. 'Best' is not a factual attribute. AI models are being trained to be cautious and avoid making subjective recommendations that could be seen as biased or legally problematic. They are much more comfortable answering a factual question like, "Which plumbers in Chicago are open 24/7?"

2. The 'Source Amnesia' Problem: This is the most critical issue. The AIs (especially Gemini and Copilot) are synthesizing information from so many different sources across the web that they seem to lose track of the quality and timeliness of those sources. An AI might combine information from a competitor's negative blog post, an outdated business directory from five years ago, and your official website, and it struggles to weigh these sources appropriately. It might give the same weight to a 2-year-old blog post titled "The 5 Best Plumbers in Chicago" as it does to the hundreds of real-time reviews on a Google Business Profile. This leads to it recommending a company with poor recent reviews because it was featured on an old 'best of' list.

3. The Lack of a Ground Truth: The core of the problem is that for most businesses, there is no single, authoritative, machine-readable 'source of truth'. The AI is forced to cobble together an answer from a patchwork of unreliable, often contradictory, third-party sources. It's doing its best to make sense of a messy internet.

The Opportunity for Your Business: Become the Source of Truth

This experiment reveals the AI's desperate need for reliable, first-party data. The business that provides it will be the one that gets the confident, authoritative recommendation. Here is how to position your business to be the solution to the AI's problem:

• Stop Relying on Third Parties: You cannot control what Yelp, Angi, or a random blogger says about you. You can control what your own website says. Your goal should be to make your own website the primary, unimpeachable source of information about your business.

• Provide Facts, Not Just Marketing Copy: Instead of just saying you offer "great service," you need to structure the facts about your services. A plumbing company should use structured data (Schema.org) to explicitly label its offerings like "serviceType": "Emergency Plumbing" or "serviceType": "Drain Cleaning". It should label attributes like "hasOffer": {"@type": "Offer", "name": "24/7 Service"}.

• Structure Everything: The more factual data you can provide in a machine-readable format, the better. Structure your hours, service area, specific services offered, licenses and certifications, and even price ranges. This turns your website from a subjective brochure into a factual database.

• Build Your Own 'Best Of' Content: Create content on your own site that demonstrates your expertise and authority. Write detailed case studies of successful jobs. Create a definitive guide to solving a common plumbing problem. This makes your site a valuable source for the AI to cite, rather than relying on others.

When an AI has to choose between a business whose information is a messy puzzle it has to solve and a business that hands it a perfect, neatly-labeled answer key, it will choose the answer key every single time. The results of our experiment are alarming, but they provide a clear roadmap for the future of local search. The businesses that invest in making themselves verifiably, factually, and structurally clear to machines will be the ones that win.