It had been a while since I read Up From Slavery so I'd forgotten that Washington himself explains exactly why the shirt was scratchy:
In the portion of Virginia where I lived it was common to use flax as part of the clothing for the slaves. That part of the flax from which our clothing was made was largely the refuse, which of course was the cheapest and roughest part. I can scarcely imagine any torture, except, perhaps the pulling of a tooth, that is equal to that caused by putting on a new flax shirt for the first time....when I was being forced to wear a new flax shirt, he [Booker's brother] generously agreed to put it on in my stead and wear it for several days, till it was "broken in." Until I had grown to be quite a youth this single garment was all that I wore.There are four points I can add to that. First, the refuse would be made of shorter fibres, and short fibres mean more ends to stick out and prickle. When you have a handful of flax to process, you separate the tow which is the shorter fibre out from the longer flax strands called line flax. Tow is used for lower grades of cloth. Second, tow comes from the lower end of the plant stem and even if you look at line flax that runs the whole length of the plant, the end that came from the root is also the coarser, thicker end. Third, any linen cloth is going to be stiff at the start and get softer with wear. When I need linen cloth, I buy the pre-softened sort by the yard or I buy second-hand dresses that have been washed many times and I cut them down to what I need. Fourth, there are many steps along the way to finished linen cloth where inept or insufficient handling could lead to a coarse, inferior product altogether, not just in the refuse part. For example, you could grow the plants too far apart allowing more side branches and thus harvest a higher proportion of short fibres than you ought. Pull the plants out of the ground too late after the tender stage (sort of like zucchini) and get coarse stalks. Ret the stalks for too little time so pectin doesn't break down to release the fibres, and get strands that won't separate from the boon. Neglect to break and scutch away all the woody boon, and the strands will be lumpy. Misalign the tow fibres before spinning when hackling or when dressing the distaff, and produce rough inconsistent yarn instead of smooth. Spin flax dry in dry weather without applying water or saliva as you go and yarn will be hairy instead of smooth. Skip softening and bleaching after weaving and the cloth will be stiff.
Certainly, however, the main thing is that the scratchy shirt was made with waste flax fibres. From Frances Lousia Goodrich's Mountain Handspun we find corroboration of the same practice of dressing small boys in scratchy linen shirts and the same explanation:
In the old times much flax was raised and worked in the mountains [Appalachia]....The tow, or refuse from the scotching, and the better tow left after the heckling, was also spun for making coarse sheets or bedticking and for shirts for little boys, who found it very scratchy until several washings had rid it of the sticks.One of the most common forms of flax available to handspinners to buy today is tow roving. The other forms are fine flax roving and flax stricks. I can't imagine ever wanting to weave yarn made from tow nor spinning it really. Sackcloth and ashes stuff.
At the demo, another spinner had set out on display a skein of linen yarn she'd spun as well as a flax strick which is a twisted bundle of line flax. Everyone commented how the strick looked like hair. One woman asked if flax was what the rich people wore, meaning back in the middle ages. I had fun astonishing her by saying, "You are wearing what the rich people wore," meaning cotton. And very likely not even then, I don't think, not in Europe.
Flax is supposed to take less work by hand than cotton to separate out the completely unusable bits. Flax is supposed to be less work than cotton to get the fibres in shape and arranged for spinning. It is work, just not as much work. Flax needs less twist to be spun into yarn than cotton because of the length difference and structure, so there's less work there too. That gave flax an edge historically along with its wider growing range at greater latitudes in places with short spring days.
I didn't have any tow at the demonstration to show how much of a different grade of flax it is, but I did say, "You saw me combing wool and throwing the short fibres on the floor because I didn't want to spin it. That's sort of like what Booker T. Washington's shirt was made from. They shouldn't have done that, it was wrong, but they did."
I got asked if my wool combs were a cotton gin, which kind of threw me, because I was working with Romney wool which has a staple length many times longer than cotton and also because as far as I know a cotton gin is machine-powered not a hand tool. I've seen one. I don't know if I gave a good answer, except to say that I was working with wool, and you can get seeds out of cotton by hand by using something that looks like a large pasta roller if the seeds are smooth and not hairy.
The upshot is, if you don't know about spinning flax and cotton, you don't know, and that's okay. If you don't run into these things in your daily life, there's no reason you should. But it's good to know, to actually have a literal grasp on the tools and materials that shaped world history.