When I was barefoot, as I often was, people tried to discourage me, on many occasions, saying that not wearing shoes is egregiously dangerous and I'd surely get hurt, from all the sticks and stones and broken glass. I ignored such panic, knowing that, obviously, the true injury rate is not more than with shoes at any statistically significant level.
Then, one day, I got the largest contiguous blisters I'd ever seen. On both feet.
As it often turns out in Reality, the most accurate answer lies well between people's commonly-taken extreme positions. Walking barefoot in arbitrary, outdoor environments is dangerous, at a manageable, partly avoidable level, for different reasons than many think.
Rough surfaces (like concrete sidewalks) are not a problem. You need to slowly get used to them first, but when there's nothing unusual on them, they're like any other safe surface.
Broken glass, in public walking places, is rare. I've seen it perhaps twice in months. Maybe it's more common where these people who worry about it are? Look where you're going a bit and you can step around it.
The real remaining danger, that far too few talk about, is burns. In warmer weather, the concrete sidewalks — and, much more so, the streets' tarmac — gets brutally hot. I attribute the blisters to exactly this issue. If you notice the ground being painfully hot, stay on grass, or lighter-coloured (and hence less heat-absorbing) pavement.
Independent of the weather is a more mild and frequent "danger": being seen as crazy, or disturbing people, as amusingly happened to a local high school's football team when I walked past their campus at the end of their practice.