Vegetarians are not against animal products but violence against innocent animals. It is possible to produce milk without harming the animal. Unlike meat, chicken, or fish, the animal is not killed during or after milking. There is nothing wrong with the human use of animals to the extent violence is not caused.
The Creator has created animals for human benefit but with the responsibility on humans to look after and protect them. Milk is a good example of how a wonderful relationship can be created between humans and animals which in fact works both ways. Besides, milk is known to be rich in Calcium and other nutrients.
We see nothing ethically wrong in consuming milk and milk products provided milk is produced humanely.