2 Tool Limitations
Building a Hermeneutic of Suspicion: A Critical Evaluation Exercise
Modern chatbots like ChatGPT, Claude, and Gemini are complex hybrids that combine search, generation, and retrieval. Understanding what each component does well and poorly helps us evaluate outputs critically. This table presents prompts that tend to challenge different types of systems. As a group, discuss which tool type would likely struggle with each prompt and why.
Important: These are tendencies, not absolutes. Any given prompt might produce a good answer due to the hybrid nature of modern systems. The goal is to understand underlying limitations, not to memorize rules.
Part 1: Prompts That Challenge Search Engines
Table 6
Part 2: Prompts That Challenge Pure LLMs
Table 7
Part 3: Prompts That Challenge RAG Systems
Table 8
Part 4: Prompts That Challenge Modern Hybrid Systems
Table 9
Key Takeaway
Modern AI chatbots are unpredictable because they're hybrids. The goal isn't to predict behavior perfectly but to develop critical evaluation habits: What kind of information would this require? Where might that information come from? What could go wrong?
