Graphical fidelity increases immersion; gameplay typically tries to create some kind of puzzle-solving flow state in an action / reaction loop. A game can be good with mostly just one or the other, albeit quite different experiences, but the best games have both.
For example, Tetris has terrible "graphics" as such - the thing is entirely abstract, and is almost a pure expression of gameplay. But a game like Skyrim is rather the opposite; it's all about immersing yourself in a role via a character you've created and grown. Meanwhile, the actual tension-creating gameplay mechanics of Skyrim, combat and sneaking, the first is fairly poor, while the second ultimately is overpowered. But those things don't stop it from being a highly enjoyable experience.
Why is it a tradeoff? Because building immersion means emulating reality to a greater and greater extent, and that reality gets further and further away from the abstract core of what makes a gameplay mechanic work well, because reality is complicated and has too many details. You have to build the gameplay in such a way that it doesn't break the spell of immersion; instead of being abstract, it has to be made out of simulated reality components.
For example, Tetris has terrible "graphics" as such - the thing is entirely abstract, and is almost a pure expression of gameplay. But a game like Skyrim is rather the opposite; it's all about immersing yourself in a role via a character you've created and grown. Meanwhile, the actual tension-creating gameplay mechanics of Skyrim, combat and sneaking, the first is fairly poor, while the second ultimately is overpowered. But those things don't stop it from being a highly enjoyable experience.
Why is it a tradeoff? Because building immersion means emulating reality to a greater and greater extent, and that reality gets further and further away from the abstract core of what makes a gameplay mechanic work well, because reality is complicated and has too many details. You have to build the gameplay in such a way that it doesn't break the spell of immersion; instead of being abstract, it has to be made out of simulated reality components.