Well the first novel was published in 1996. Was probably in Martin's mind for years and years before that, possibly decades.
I recall him mentioning he first thought of the White Walkers and the Wall in the 70's, and slowly started work then, but had other projects also. Not 100% so don't quote me on that.
“There are three things all wise men fear: the sea in storm, a night with no moon, and the anger of a gentle man.”