Browncoats were soldiers who fought for the Independent Planets, who lost to the Union of Allied Planets in the Unification War. They were named after the simple, brown leather trench coats they wore as uniforms. They also wore red scarves around their necks. The simplicity of their gear was greatly because technology and resources were scarce on the Outer Planets. Conversely, the Alliance's soldiers were called "Purple Bellies", because of the purple tinted body armor they wore on their chests.
After the war's end, clothiers made good money dyeing brown coats blue or gray as folks wanted to forget the past and let the past forget them. Those that still "wear the brown" do it on purpose.
Behind the scenes
The term "Browncoat" has also come to apply to fans of the show. By taking agency over the term Browncoat, fans align themselves with the "Independents" as they fight against the Alliance. The Alliance being the FOX industry that stopped airing Firefly after its first season.