Flagged Smart Group Loads Slowly

I think I may have figured out where the problem is. I had it sorted by date accessed. When I change it to sort by name, it speeds up. Not sure why this is. If I change it back it slows down again.