By design, we should be able to safely clean up orphaned data using only local information (unless we are not sure if we're a shard or not). Keep thing simple by only refreshing if our local information is unset (i.e. needMetadataFor() is false).
Previously was a ticket tracking error messages:
"failed to refresh shard metadata: cannot refresh metadata for foo.bar before shard name has been set"