The common phrase a whole nother, formed by splitting the adjective another, makes no sense from a logical or grammatical standpoint, but it is often used informally or to create a colloquial tone in writing. Because it is informal, the phrase might be considered out of place in any type of serious writing. A whole other makes more sense, and there are one-word equivalents, such as different, separate, and unrelated, that are usually better in formal contexts.
Many writers insert an apostrophe in a whole ‘nother, but this would logically indicate that the writer means a whole another, which makes even less sense than a whole nother.
In these examples, the writers use a whole nother (or a whole ‘nother) to create a light, informal tone:
There’s also a gas/electric hybrid, which is a whole ‘nother animal. [STLToday.com]
But that’s on a whole ‘nother level compared to what it has done to Green, the edgy, arty poster boy for reflective, solemn character studies. [Charleston City Paper]
As the pictures up top and at left attest, he was seeing the puck just fine; stopping it, however, was a whole ‘nother matter. [Edmonton Journal]