Why does ChatGPT keep picking answer A?

AI models like ChatGPT have a measurable habit of favoring certain answer positions, so when one writes a multiple choice quiz, it often parks the right answer in slot A. The fix is not a better prompt. It is never letting the AI decide where the answer goes.

I know this because my own trivia game got caught with its answers showing.

The round where every answer was A

My games arcade has an AI trivia game called Bright Spark. Every six hours, an AI writes eight fresh questions, each with four choices, and everyone plays the same round on one shared leaderboard.

Then I looked at one round and my stomach dropped a little. The correct answer was A. For question one. And two. And three. All eight of them.

Think about what that means. Anyone could tap A eight times without reading a single question and land at the top of the board. A trivia game where the trivia doesn’t matter is not a game. It’s a coin with one side.

The set had been written by GPT-4o. I had asked it for each question, its four choices, and the position of the correct one. It wrote the right answer first, every time, and then honestly reported that the right answer was first.

This isn’t a glitch. It has a name.

Researchers who test AI models on multiple choice questions have a name for this: selection bias, sometimes called position bias. Models lean toward particular option letters, and if you shuffle the exact same choices into a new order, the model’s answer can change with them. The content stayed the same. Only the seat changed.

Let’s get a little nerdy for a moment. A language model writes one piece at a time, predicting what most likely comes next. When it is “thinking about” the right answer, the right answer is the most natural thing to write down first. Nobody told it that a good quiz hides the answer. So it does what any of us might do when we’re writing quickly: it leads with the thing it knows.

Why a better prompt didn’t fix it

My first instinct, and maybe yours, was to just tell it: randomize the position of the correct answer. Here’s the problem. A language model has no dice. When you ask it to be random, it produces something that looks random to it, and that still drifts into patterns.

Counting is a weak spot too. When I moved Bright Spark to a newer model, it stopped stacking everything in slot A, but about one question in eight came back with the answer pointing at the wrong choice. The fact was right. The position was off by one. That’s arguably worse, because now a player who knows the answer gets marked wrong.

So I stopped asking the AI to do the one job it is bad at.

The fix: the AI writes, the code places

Here’s the setup Bright Spark runs on now. If you’re building anything with AI generated multiple choice, steal it.

1. Ask for the answer as text, not a letter. The model copies the correct choice word for word. It never says A, B, C, or D.
2. Check the copy. If that text doesn’t exactly match one of the four choices, the question gets thrown out. No guessing.
3. Shuffle with real randomness. The code reorders the choices using the computer’s random number generator, the same kind used for security, not the AI’s idea of random.
4. Find the answer’s new seat yourself. After the shuffle, the code looks up where the correct text landed. That becomes the answer key.
5. Keep the key on the server. Players only ever receive the questions. Grading happens behind the scenes, so peeking at the page gets you nothing.

The beautiful part is that position bias becomes structurally impossible. The AI can prefer slot A all it wants. It never gets to choose a slot.

The bigger lesson for anyone building with AI

This goes way beyond trivia. AI is wonderful at language: writing the question, choosing four believable choices, keeping the tone fun. It is unreliable at anything that has to be truly random, carefully counted, or fair.

So split the work. Let the AI do the words. Hand the dice, the math, and the fairness to plain old code. That applies whether you’re picking a giveaway winner, rotating who gets the next lead, or ordering anything people will compete over.

I didn’t find this in a textbook. I found it because my leaderboard was quietly rewarding people for not reading. Sometimes the most useful lessons show up looking like a bug.

Building something with AI and running into strange patterns like this one? I’d love to hear about it. Come say hi, and keep an eye on the blog for more honest notes from the things I build.