You don't seem like the type to swoon over the first pretty face to come along. Or the seventeenth. Or the eightieth.

I'd call that a challenge. (And be kind of impressed, really.) Doesn't mean anything bad. It does mean that it'll take more than a pretty face for you to be swayed by a person.