For the idiom meaning to go through in minute detail, fine-toothed comb makes more sense and is more grammatically justifiable than fine-tooth comb. Tooth doesn’t work as an adjective, so it technically can’t modify comb. Yet the latter spelling is far more common than the former.
Examples
Some publications are content to use the questionable form—for example:
I’m suggesting you take a fine-tooth comb and look at your actions, one at a time. [Huffington Post]
Ronny Marshalsea … scoured his house with a fine-tooth comb to fill a giant Ziploc bag with loose change … [Boston Globe]
I went through their lineout with a fine-tooth comb and it is undoubtedly one of the best in the world … [Scotsman]
And others go with the more grammatically justifiable one—for example:
We went through the site with a fine-toothed comb and removed everything that we could … [TechRepublic]
Yet these women look at their bodies with a fine-toothed comb looking for their flaws. [Iowa State Daily]
Well, the 1,000 emails and 3,000 documents from 13 years of correspondence were gone over with a fine-toothed comb. [Telegraph]
In either case, fine-toothed or fine-tooth functions as a phrasal adjective modifying the noun comb, so it does require that hyphen.
