At what stage should we collect references?

I work for a large organisation who deals with a lot of recruitment for a diverse range of jobs, some of which involve fraud potential, or contact with children or vulnerable adults.

We currently collect references for all shortlisted candidates prior to the interview.
My view is that this creates a lot of waste and we should make conditional offers of employment and collect the references then from the suitable candidate only, but my recruiting managers don’t like the idea. They want to ratify their choice at interview stage, have concerns over refusing someone if a poor reference is received and think the risks are high.

What do other organisations do, and how does it work for them? Do you agree with me, and if so help me convince the managers!! If not, put me straight on this.

Many thanks
Sarah Lister