I think it really depends on the country the milk comes from and the treatment of the cows. Milk can be a very good source of nutrition. But a cow that eats other cows as part of their diet or that is fed growth hormones is not a good source for anything, including its milk.
But you are right vitamin d can be received from about 15 minutes of sunlight a day. Protein, phosphorous, and fat are all accessible in other foods, but a non-dairy diet isn't necessary to remain as healthy as possible.