When I consider possible reasons why farmers' markets and relocalization movements are focusing on food rather than clothing, I come to the obvious reason. You have to eat. You have to wear clothes, too, but you're more likely to run out of food first so it's a good idea to make sure your local supply is assured.
In the more mainstream marketplace, supermarkets, people are paying a premium for local food and organic food. I don't think these people care so much about local supply. I think it has to do with fears about safety.
Food safety is a hot topic but no one is worried about clothing safety in a big way. We worry about what we eat, but not what we wear. I mean, we might worry whether we can afford the right clothes, or whether our clothing will protect us from flames and slippery floors, or whether our clothes make us look fat.
But we don't worry that our clothing will harm our health, that we will suffer anything like those poison-dipped cloaks from mythology. Clothing is perceived as a surface issue, outside us, whereas food is an internal matter. (Yes, I am stealing an idea from a Wendell Berry essay here about people's perceptions of the environment as "outside" and "other." At least I think I am. I read that essay a long time ago.) We are what we eat. The modern-day poisons of clothing are far away. Blue jean dye waste chokes rivers in China, for example, not here.
Yet our skin and our lungs absorb. We wear clothing produced and distributed in ways that are highly parallel to the ways conventional, non-organic food production and distribution are done. I think it's worth thinking about.
When I think about it, my mind bounces around and keeps coming back to how limited my options are and how hard I'd have to work to source or create clothes and household textiles that meet standards comparable to certified organic food. (I know there are items out there, thanks, so you don't have to post links for me. I am sure there are even some nice plain organic clothes out there that aren't plastered with symbols I'd refuse to wear for religious reasons.) I haven't got a silver bullet solution, but I continue to think and spin yarn.