They can kick you out and tell you if you return you'll be trespassing, but cuffing someone and detaining them in a back room without any legal authority to do so is kidnapping. And depending on how they physically treat you during that, it might be battery or even assault.

I suspect, though, that many/most people in these situations either don't want a run-in with the law, or want to keep a low profile and not make a scene for other reasons, so they'll just go along with it and quietly get out of there.

They can ask you to leave and then hit you with trespassing if you refuse to go, but they can't just slap you in cuffs and throw you in a concrete room in the back of the casino because you're counting cards. Any money they saved on your gambling would be lost in the kidnapping lawsuit afterward. And if they refuse to cash the chips you can add fraud and theft to the charges.