Take a break.
The thing is that if you don't, you're just storing it up. There's a good chance that your head just won't be right when you do go and work out. It won't be as good as it could be- if you were really raring to go. And there's always a chance that it bulds up to the point that you get really pissed off and stop for ages.
So maybe you can work out harder and longer, in the long run, if you give yourself those breaks.
You can always look for some sport or game or something that will keep you active in the days you take off.