I've been able to do it, with an intel onbound. Haven't used 0.87 in a long time, though.
With the screen rotated vertical and the desktop rotated to match, I had
switchres 0
keepaspect 1
screen_aspect 4:5
and the ror, rol, and other rotate options at default. When I had the screen_aspect the 5:4 way, it was very wrong. You get small black bars on the sides.
If you're talking about vertical games on a normal 5:4 screen, same as above except just switch the screen_aspect to 5:4.