Should I just leave him alone?

Even if you are physically attracted to him, disinterest, either verbal or through nonverbal cues such as yawning, crossing his arms or repeatedly looking away, is a good sign that you should leave him alone.