I'd have to say warlock, as they can DoT you up so the damage keeps on coming, while their pet and drain can add to it, the drain reducing how fast you can kill them, not to mention they can chain fear you. Trinket is only going to help you once, and diminishing returns don't immediately make worthless a spell that can run for a long time.
Trying to keep them out of range is tough, as they only need a second to get DoTs up if you go inside 30yds, and you'll have to go well out of your own firing range to break a drain.
The easy counter to a warlock is to just focus them down before they spread too many DoTs around. I just wish more players knew how to focus properly (on healers especially), instead of just retaliating against whatever is attacking them.
Warrior's who spam hamstring are an absolute nightmare, but it takes help from the other players to slow the hunter initially, as a good one will give a warrior a wide berth to avoid charges.
Arcane mages could be good (just you don't see many mages, and next to none are arcane), as they can sheep hunter, kite pet down (or just nova the thing), then take on the hunter, although it'll be real close as to whether their potentially higher DPS from uninterruptable arcane volley will cope against another kiting class, when they've lost a lot of that from the frost tree, and probably have to lose a fair bit of stam to get arcane damage up.
[edit] A big reason for me rolling a hunter was to learn how to play against them better.