FOUND IT: Himitsuu Noo Otsuu
It might’ve been a oneshot, if not it was a very short manga.
The MC and his girlfriend move to a new town, and the man gets invited by like the mayor or something to hang out with him and other men. There they fuck him (dubcon seems to me).
It continues for a while, then they all stop having sex with him, until MC gets urges and reaches out to the mayor himself to start it up again