I'd be torn too. In the end, I'm not sure I could do a fifth floor walk-up.
FWIW, the place we rented is a third floor walk-up, which I'm slightly excited about (forced exercise! woo!). I just think five might be too much. But your fitter than me.
Are there noise issues on 23rd? I know you like the quietness of your street now.