For skilled pianists, to play a song by ear is to play it after only hearing it, without using sheet music or taking instruction. This is the origin of play by ear in the metaphorical sense. It properly means to do something without instruction or guidance, but it’s often used to mean, simply, improvise.
The idiom is several centuries old, descending from the use of ear as a metonym for one’s natural ability to recognize musical intervals. For example, we might say that Mozart had a great ear because he could play songs after only hearing them. From this, it’s easy to see how play by ear came about.