Whoa whoa whoa, didn't know about the fast travel back thing. Don't get me wrong, I love exploring new areas, and when not running 'back' the same way, it's a great game. For example, I went to that cave to find the horn, which was already stolen, now I have to run all the way back home, the same way I came, so i'll see if the teleport works for that, if so, the game just became much better.
Yeah, when you are outside of a cave (can't do it inside a building or cave) just click on the town/area you got your quest, and it'll ask you if you want to fast travel there. You can fast travel to any location you have uncovered.
You can also push the select button (or back button on 360) and "wait", letting you pick if you want it to be day or night. Not sure if you know this, but just letting you know in case you don't.