As the NFL season nears and fantasy football drafts kick off, I can only imagine how many people are turning to AI (Claude, ChatGPT, Grok, etc.) for advice on who to pick. Naturally the question comes up: is that AI advice actually good advice? As with most AI topics, the honest answer is it depends.

Trained to look correct, not be correct

Quick caveat before I get into this: I'm making some broad generalizations about AI here. All the big models are trained in slightly different proprietary ways, and there's a decent chance a new breakthrough drops before I even hit publish on this post. So when I say "AI," I mean LLMs (Large Language Models), the chat bots you're used to interacting with like ChatGPT, Claude, and Grok.

So what do I mean by "trained to look correct, not be correct"? Let me tell you a quick story first.

Looks great, but did you verify the answer?

This really happened, though I'm leaving out names. A business owner exported his employees' time sheets and sales data to Excel, then gave an AI chat bot access to it. He was thrilled showing everyone the insights it could pull, things like "who's the top salesperson" or "who had the most gross sales between these two dates." He was so convincing that someone close to us felt pressure to set up the same thing or risk falling behind, despite our warnings.

To settle it, we built a mini replica with just 30 time entries and told him to ask it anything. He started simple: "How much time did this person work?" The AI spit out a confident, clean-looking answer. He figured that settled it, until we asked him to add up the entries by hand. The numbers didn't match. We tried a few more. Same result. Reality set in fast.

Why does this happen

You probably already know the short version: LLMs predict the next token based on patterns in training data, they don't actually "compute" the way a calculator does. Here's why that matters for your fantasy team, not just for spreadsheets.

An LLM doesn't see "1 + 1 = 2" as math, it sees a sequence of tokens, and it's learned that this particular sequence is very likely followed by the token "2." With enough training data, this trick works well for basic arithmetic, which creates a convincing illusion of math ability. But it's closer to a kid memorizing flash cards, just millions of them, than actual reasoning. Math is infinite, so the moment you step outside the flash cards (add up 30 time entries, say, or tally a full season of fantasy points) accuracy starts to slip.

Every so often I test whether AIs have finally cracked adding a string of numbers together. For months, they failed. Recently, one finally got it right. Did it suddenly learn math? I doubt it. What's actually happening is tool use (sometimes called function calling): the model recognizes it's looking at a math problem, hands the numbers off to an actual calculator or code interpreter, and echoes the result back to you. That's a legitimately good hybrid approach when it's used, the problem is you often can't tell whether it kicked in or whether you're looking at a flash-card guess.

Outdated data

The other issue: AIs are trained on a fixed data set, so their knowledge has a cutoff. Some models can search the live web to patch this, but that "skill" doesn't always kick in, and it isn't foolproof.

A good example: Bo Nix (go Broncos) fractured his ankle in the AFC Divisional Round against the Bills. Ask an AI about the next game with a training cutoff before the injury, and if it doesn't think to search for an update, it'll confidently give you an answer that has no idea the Broncos were missing their starting quarterback. Sports move fast, and there are countless examples like this every week during the season. There's also a bias risk: these models train on articles written across the internet, and if the coverage of a player or team skews hyped or overly negative, the AI absorbs that skew right along with the facts.

Final answer: it depends on you

Weighing those risks, if you know next to nothing about football and this is your first fantasy draft, an AI will probably still help you, especially for the basics like "draft running backs early, kickers last." But if you're a fantasy veteran, trust your own instincts over anything an AI tells you that contradicts them.

Tips and tricks

• Ask very specific questions. Keep follow-ups in the same conversation, but start a new one when you switch topics, it helps the AI stay focused.
• Explicitly tell it to look up live data, so it actually uses its web search skill instead of relying on stale training data.
• Cross check different AIs against each other (for example: "Grok, adversarially check these responses from my conversation with Claude" and paste it in).
• Fact check its answers and hand it links showing where it's wrong, that feedback helps it correct course.
• Do your own research, and trust your gut. :)