While by no means complete, consider the following long roads taken from concern, to hazard, to action:
Asbestos was first called into question in 1898, when the Chief Inspector of Factories in the UK reported to Parliament that the "sharp, glass like nature of (asbestos dust) particles" when mixed with air "have been found to be injurious, as might have been expected." The first cases of asbestos deaths in factories in Britain were confirmed in 1906. While claims were filed in the US starting in 1918, and many shipbuilders suffered from asbestos exposure during WWII, asbestos was pulled off the US market in the mid-1980s.
Mercury enjoyed an exceedingly long day in the sun as a cure-all in medicine (starting in the 16th century), to felt production from fur (in the mid-18th century), to the main ingredient in dental fillings in the early 19th century. While mercury was identified as a neurological poison 200 years ago, its various suns are setting slowly. Banned from the US felting industry in 1941, it is only now being eliminated from hospitals by organizations like Health Care Without Harm and in state-by-state legislation.
The slowest setter of them of all may be lead, with its toxicity recognized for thousands of years, as indicated by ancient Greek and Roman accounts of lead as a brain and blood toxin. But remediating houses (and not necessarily historic ones) for lead paint is a current activity, as is discovering that inflatable vinyl "bounce houses" (see "The Cost of Greening" post) are delivering lead to the young population most susceptible to its hazardous effects.
I'm going to venture a guess. Perhaps our response time to toxins may be somehow proportionate to the course of the human disease or environmental disaster, which may take years or generations to manifest. "Not enough evidence" is often what is cited when there is no immediate danger, even though we're smart enough to extrapolate the evidence that is there.
None of us take the long view easily perhaps, but somehow we have to get better at it. That, or simply embrace what's called the Precautionary Principle: not waiting for scientific certainty before taking action against serious threats to human or environmental health.