It is thought that human beings have been wearing clothing for over 650,000 years, and originally this served a practical value in terms of protection from the elements, as opposed to more modern notions such as modesty.
Clothing then developed past the practical to become a term of status, with some people wearing certain clothes that denoted their authority. From this, then came ideas such as fashion, where clothes extended past their practical application of protection from the elements.
It must be noted that items such as bags, or ladies purses are considered accessories, although they would have developed at around the same time, the first ones being like an Aboriginal 'dilly bag'.
As clothing developed, there came gender distinction, with males and females making and wearing clothes representative of their sexes. This has carried on until modern times, although nowadays there is a blurring of these two sides.
That said, some societies such as Muslim ones still maintain a distinct male and female dress code, and indeed in some states this is enforceable by law. The idea of clothes being worn for status is also still with us, whether it be the Queen of England wearing some form of regal dress, or a businessman wearing a suite to proclaim their position. In many ways then, we are not so different from our ancestors of thousands of years ago!