Alma mater is a compound noun that refers to a school, college, or university that someone previously graduated from. In some cases, it may be used for a school’s song. When referring to the song, it is capitalized Alma Mater, as any song title would be. It is the same if it is the name of an actual college, statue, or other work.
In some instances when alma mater acts as a pronoun, taking the place of a specific college or university name, it is capitalized. But when used generally, it should be lowercase.
In Latin it literally means generous or bountiful mother.
As with most words borrowed from Latin, there is the traditional Latin plural and the English version. The Latin plural for alma mater is almae matres. The English plural is alma maters. Not surprisingly, the English version is vastly preferred.