The match will benefit the Sentebale charity that Harry set up with Prince Seeiso of Lesotho for impoverished children in that African nation.
"The prince and I both lost our mothers when we were very young," Harry told a pre-match VIP gathering. "We set up Sentebale in their memory, and because my mother loved this city, it makes this occasion all the more poignant for me."
Harry is on his first official trip to the United States — a two-day New York visit.
The prince arrived on Governors Island at about 12:30 p.m. on a police launch.
Under a sunny sky, hundreds of people gathered in a white tent for a champagne reception and lunch, many wearing lavish hats topped with flowers and feathers.
Harry was casual in a navy blue blazer, an open-collared shirt and white jeans, and loafers.
Members of the public were invited to watch the polo match, in which Harry would face off against the Argentine heartthrob Nacho Figueras.
Mike Hallman, visiting from Cary, N.C., decided to take his family to the match. "My kids have never seen polo before," he said. "It's pretty exciting. I have never seen a prince."
Another spectator who had never seen a polo match was LL Cool J.
"Hopefully, it'll be quite good," Harry told the rapper during the reception.
"Are you going to win?" the rapper asked.
"Mmmm. I don't know. Hopefully it's fixed," joked Harry.
Other celebrities at the match included actresses Chloe Sevigny and Kate Hudson and designer Marc Jacobs.
On Saturday morning, Harry and Seesio toured Harlem's Children Zone, a community organization that offers families social and educational services. They chatted with 14 students working on math in preparation for a Regents Exam.
"Who's the best pupil?" he asked the class of ninth-graders. "I was always the worst!"
The two princes also competed in an obstacle course; Harry lost, throwing his arms in the air and yelling, "Yeah!" to the cheering kids.
The younger son of Prince Charles and Princess Diana is third in line to the British throne, after his father and older brother, William.
Dubbed "the party prince," Harry is a much-watched member of the royal family, regularly making newspaper headlines with his party escapades and brushes with scandal. Several years ago, he apologized for wearing a Nazi swastika armband to a friend's costume party.
A different Harry is emerging on his U.S. visit, which started on Friday morning with a prayerful stop at the site of the Sept. 11 terror attack. There, he spent about 15 minutes quietly speaking to a half-dozen relatives of 9/11 victims.
Harry then attached a wreath to a chain-link fence overlooking the Sept. 11 memorial under construction at ground zero, bowing his head in silence for a few minutes. He also visited the firehouse across the street that houses Engine 10 and Ladder 10, which lost five members on Sept. 11, talking and laughing with firefighters there.
Later Friday, Harry formally named the British Memorial Garden in Hanover Square downtown to honor the 67 British victims of the terrorist attack and he visited Manhattan's Veterans Affairs Medical Center, touring a clinic that treats veterans with post-traumatic stress disorder and the prosthetics facilities.
On Saturday, Harry's thoughts turned to children who are impoverished or suffering from AIDS — a cause for which his late mother worked passionately.
Harry is scheduled to leave for England on Saturday evening.
"It's been wonderful, it's been a whirlwind," he said on Saturday. "I haven't had a chance to let the jet lag set in, and it's time to go already."
Related Content from AccessHollywood.com:
VIEW THE PHOTOS: Week's Best Star Sightings, Feb. 21 - 27, 2009