Not surprising, perhaps, when a third of Chinese Hongkongers don't want to sit next to members of other ethnic groups on public transport, live next door to them, or have their children as classmates to their own children, according to a survey by the NGO Hong Kong Unison.
Is the beautiful game really powerful enough to overcome that kind of prejudice?
Since 2016, a "refugee" football team - All Black FC - has been testing the theory as its players try to integrate, and build a better image for Africans in Hong Kong.
The team's founder Medard Privat Koya is a former professional footballer from the Central African Republic. While his first players were almost all African asylum seekers, the squad has expanded to welcome Chinese locals and other ethnic minorities.
"Football can bring all of us together," he reckons. "We can bring ideas, share experience and support each other."
It's also a constructive use of time - which is valuable because asylum seekers aren't allowed to work in Hong Kong, even if they have lived in the city for years.
Many of the team are unable to work legally in Hong Kong
Darius, an asylum seeker from Togo, arrived in Hong Kong five years ago. He's the team captain, and says football gives him a sense of purpose as they are not allowed to work.
"It is really hard. It makes people feel they are dying slowly because you spend five or seven years of your youth life waiting. You really start wondering what's are you going to become in the future?
"After all that time with no work, or training, what kind of job are you going to do? Who would hire you?" says Darius.
When he became part of this team, it gave him hope.
"There is a group of people that I belong to," he says. "We fight for something together. People will respect us based on our skills."
And for the team's most talented players, the distraction can lead to something bigger. An exceptional few are chosen to represent local football clubs, and can then apply for visas to work legally in Hong Kong.
Captain Darius (centre) and founder Medard (right) want the team to be a source of support for players
'People take pictures and cover their noses'
Medard is encouraged to see locals and other non-Africans joining All Black FC. It shows that their values are respected, he says.
That's an unusual feeling for some of the team.
Darius says it's difficult for immigrants to rent a flat in the city as many landlords look down on foreigners.
Solomon Nyassi, a 26-year-old Gambian, tells BBC Chinese that on his first day in Hong Kong, he saw some people using their hands to cover their noses when they passed by him.
Sometimes, he says, he walks down the street, and Chinese tourists surround him and ask for pictures, which makes him feel "embarrassed".
Solomon met his girlfriend, 20-year-old Hong Kong woman Louise Chan at an All Black FC match. She believes African men are more "mature" and "self-motivated", and her family support the pair's relationship. But some people's reactions have shocked her.
"Hong Kong people are so closed-minded," says Louise, who owns a clothes shop. "Some keep sending articles to me to tell the bad rumours about African people. I feel very disappointed."
Footballer Solomon and his girlfriend Louise have faced prejudice from some quarters
Prejudice can show its face on the pitch, too.
Darius says some Hong Kong locals think Africans play football "violently". Some have shouted foul language at the team.
"We always tell our players to keep disciplined," the captain says. "No matter what has happened, we just need to keep it calm. We can't let them label us."
'I don't see race'
Doug Tze, a 34-year-old Hong Kong man, has been part of All Black FC since July 2017.
He says he didn't feel totally welcome at first, and blames the African players' previous poor treatment at the hands of Hongkongers.
But when he showed some effort and commitment, Doug says they became good friends who now hang out together.
"I don't see race; I just see good players," he says.
All Black FC in action against Biu Chun Rangers, who play in the Hong Kong Premier League
Kevin Fung, a college football player who recently played All Black FC in a tournament, believes the African players could motivate Hong Kong natives.
"They are passionate, focused and work hard. But Hong Kong players give up easily," Kevin says.
"Hong Kong people focus on other work and study. They only spare their little leisure time for football."
For that reason, he doubts football can really achieve racial harmony in Hong Kong.
"After all, local football is not a popular activity," he notes. "In Hong Kong, there are only a few competitions. Official games are not broadcast widely. No-one really cares (about local football). Solving racial problems through football is doomed to be difficult."
All Black FC understand that too. So to further their goal of achieving social change, the players sometimes do voluntary work like visiting old peoples' homes to connect with the wider community.
"It changes the locals' mindset," says Darius.
"Maybe we are not benefitted directly today. But it may make a path for the next generation so that ethnic minorities, asylum seekers and refugees will be seen differently in future.
By Lam Cho Wai