Some people are particularly emotionally sensitive to others. If you’re one of those people then you might find that as soon as you call someone to mind in meditation you have some sort of feeling response towards them.
That’s great! It’s quite an asset to have such emotional responsiveness. That quality will make it easier for you to develop lovingkindness in this meditation practice.
The point of this stage of the meditation is not to find someone that you feel absolutely nothing for, but to take someone who is a representative of that huge mass of people that you don’t regard as being a personal friend and at the same time don’t have any sense of conflict with.
So if you’re concerned that you can’t find a neutral person then it’s OK. As long as that person is not a friend or an “enemy” then they’ll do. Just pick someone you don’t have any particularly strong emotions towards — either positively or negatively.
You can pick anyone that you hardly know — perhaps an acquaintance or someone who works in a store that you visit, or a library. Or perhaps you could pick someone that you pass regularly on the street or who works in the same building as you.
For the purposes of this stage of the meditation practice that sort of person will do just fine.