In 1801,Thomas Jefferson was the first person to be sworn in as President in Washington, D.C., the location chosen for the permanent capital and the site of all but a handful of Inaugural ceremonies. Jefferson showed his taste for simplicity by going on foot to the Capitol for the oath taking ceremony. Chief Justice John Marshall administered the oath of office to Jefferson, the first of five presidents elect he would induct. Concerning the speech that Jefferson delivered, it was a 1721 word speech, which he gave in the U.S. Capitol's Senate chamber. After the ceremonies, Jefferson returned to his boardinghouse for dinner.

However, in 1805 after his second Inauguration, Jefferson rode on horseback from the Capitol to the President's House (the name then used for the White House). Amid music and a spontaneous gathering of mechanics from the nearby Navy Yard, a procession of people gathered. This was the start of what has grown into today's Inaugural Parade.

More Info: en.wikipedia.org