The phrasal verb meaning to pay, especially reluctantly, is shell out. The exact origins of the phrase are mysterious, but it originated around 1800 in the U.S., probably as an analogy between the removal of money from a pocket or wallet and the removal of beans, seeds, nuts, etc. from their shells. In fact, shell out appeared earlier than 1800 in reference to removing things from shells. Later it developed another sense in medical terminology, where it usually refers to the surgical removal of growths (one might shell out a tumor, for instance, in an effort to save a patient). This last use of the phrase is especially common in late-19th-century medical texts.
A shill is someone who poses as a satisfied customer in order to deceive people into buying a product, and the word also works as a verb meaning to act as a shill. It’s hard to imagine a context where shill out would make sense. That phrase is usually just a misspelling of shell out.
[W]itness the testimony of Mr. Noah, in New York, and others, who prove that the office-holders had to shell out a part of their salary to support Jacksonism and Jackson candidates. [An account of Col. Crockett’s tour to the North and down East, David Crockett (1835)]
“Well,” said he, “shell out my 5 dollars that I put up with that friend of yours, as I can’t find him.” [The Sportsman (1838)]
The fact is, the general public is getting tired of always putting its hand in its pocket and shelling out for the support of people with grievances — real or otherwise. [The Mechanical News, vol. 21 (1892)]
But I’ll come across Con some day, and then I’ll wager I’ll make him shell out what he stole from me. [Two Boy Gold Miners, Frank V. Webster (1937)]
I collected these, and traded them with bookish friends like Fred Muth and Tony Van Liew, and on Saturday mornings in Reading, shelled out dimes not only for the fresh new issues but for the antique treasure. [Getting the Words Out, John Updike (1988)]
What good is it if the city shells out millions of dollars for schools and the children aren’t there? [Washington Post (2012)]