He's still there. You just have bad luck. That's sort of what BRD is for most people.
When I needed Hedrum, I got the ogre like 8 times. When I needed the ogre, it's all the other ones. When you want Greaves of Withering Despair, you get 7 Enthralled Spheres, 3 Blackveil Capes and 6 Kentic...