Williams was born in Rutherford, New Jersey, a community near the city of Paterson. His father was an English immigrant, and his mother was born in Puerto Rico. He attended public school in Rutherford until 1897, then was sent to study at Château de Lancy near Geneva, Switzerland, the Lycée Condorcet in Paris, France, for two years and Horace Mann School in New York City. Then, in 1902, he entered the University of Pennsylvania Medical School. During his time at Penn, Williams befriended Ezra Pound, Hilda Doolittle (best known as H.D.) and the painter Charles Demuth. These friendships supported his growing passion for poetry. He received his M.D. in 1906 and spent the next four years in internships in New York City and in travel and postgraduate studies abroad (e.g., at the University of Leipzig where he studied pediatrics). He returned to Rutherford in 1910 and began his medical practice, which lasted until 1951. Ironically, most of his patients knew little if anything of his writings; instead they viewed him as a doctor who helped deliver over 2,000 of their children into the world.