The transfer of the US embassy from Tel Aviv had incensed Palestinians, who already claim eastern Jerusalem as the capital of a future Palestinian state.
After the deadliest day of violence killed 55 and wounded 2,700 Palestinians, Israeli Prime Minister Benjamin Netanyahu said his military acted in self-defence against Gaza's Islamist rulers, Hamas, who he said wanted to destroy Israel, the media reported.
The violence on Monday coincided with the dedication ceremony for the US embassy. The transfer of the US embassy from Tel Aviv had incensed Palestinians, who already claim eastern Jerusalem as the capital of a future Palestinian state.
Netanyahu defended his military, saying: "Every country has an obligation to defend its borders.
"The Hamas terrorist organisation declares its intention to destroy Israel and sends thousands to breach the border fence in order to achieve this goal. We will continue to act with determination to protect our sovereignty and our citizens."
Israel said some 40,000 Palestinians had taken part in the "violent riots" at 13 locations along the Gaza Strip security fence, the BBC reported.
Palestinians hurled stones and incendiary devices, while the Israeli military used tear gas and live fire from snipers, the BBC reported.
An Israel Defence Forces (IDF) spokesman said soldiers fired on those engaged in "terrorist activity and not on demonstrators, who were dispersed by usual means such as tear gas and according to the rules of engagement".
Announcing three days of mourning, Palestinian Authority President Mahmoud Abbas said: "Today once again, the massacres against our people continue."
Palestinians were demonstrating on Monday as they have been for six weeks as part of a protest, orchestrated by Hamas, called the "Great March of Return".
However, Monday's protests, and more planned for Tuesday, were the culmination that mark the anniversary of Israel's creation in 1948, and what Palestinians term the 'Nakba' or Catastrophe.
It refers to the hundreds of thousands of Palestinians, who subsequently fled their homes or were displaced in the war that followed, the BBC said.
There were clashes between Israeli police and protesters who raised Palestinian flags outside the new embassy in Jerusalem and several protesters were detained.
Hamas said the border protests were being stepped up for Tuesday.