Cancellara finished the 9.6-mile time trial in 19 minutes, 32 seconds, beating 2007 Tour winner Alberto Contador by 18 seconds to capture the yellow jersey.
Contador, returning to the Tour for the first time since his victory, led four Astana riders into the top 10. Andreas Kloden of Germany was fourth, American Levi Leipheimer was sixth and rival Armstrong was 40 seconds back in 10th.
"I felt pretty good," Armstrong said after taking the early lead as the 18th rider out of 180 to complete the course along winding and hilly roads in the Mediterranean principality of Monaco. "I was a little all over the place. ... It was up, flat, it was down, it was a technical course. But I think overall, I felt good."
Armstrong's lead lasted until Germany's Tony Martin outpaced the 37-year-old Texan by 7 seconds only 15 riders later.
Others also soon bettered Armstrong's time, including Leipheimer and Liquigas rider Roman Kreuziger.
"Kreuziger just moved into 2nd. I raced with his dad! Haha," Armstrong said on his Twitter account.
Cancellara, of the Saxo Bank team, will wear the overall race leader's yellow jersey for Sunday's second stage — a 116.2-mile ride across plains from Monaco to Brignoles, France.
Indicating his power, Cancellara overtook Giro d'Italia winner Denis Menchov — who left 90 seconds earlier down the start ramp in the time-trial — just yards from the finish line.
Bradley Wiggins, who won gold at the Beijing Olympics in individual pursuit, finished third, 19 seconds behind.
Armstrong, who came out of retirement this winter, is competing in his first Tour four years after the last of his record seven victories in cycling's premier race. He said he didn't have any "big illusions" that he'd win.
"I didn't expect to win or to take the jersey. I didn't expect a super, super performance," Armstrong said. "Was I focused enough? Yeah, I think so — I was nervous, which is logical with the years away. I didn't feel necessarily comfortable.
"It's been a long time since I've had that emotion of being on the start ramp at the Tour."