Difference between で(de) and に(ni)

What is the difference between で(de)andに(ni)? They both are used after place words.
で(de) is a location marker which indicates the location where something is happening.
For example,
“Watashi wa resutoran de shokuji wo shimashita.” means “I had a meal at a restaurant.”, and で(de) works as “at“.
In the sentence “Watashi wa toshokan de hon wo yomimashita.”, which means “I read a book in the library.”, で(de) works as “in“.
に(ni) is a destination/direction marker which indicates the direction of the action.
In the sentence “Watashi wa ashita Nihon ni ikimasu.”, which means ” I will go to Japan tomorrow.”, に(ni) works as “to“. This is the destination of the trip.
In the sentence “Watashi wa Sumisu-san ni tegami wo kakimashita.”,which means “I wrote a letter to Mr. Smith.”, に(ni) works as “to”. This is the direction of the action(writing) in this case.
However, there are some exceptions. In the sentences such as “-ga arimasu/imasu.”, “ni” is used to indicate the location.
There is a restaurant on the 2nd floor. → 2kai ni resutoran ga arimasu.
I live in Canada. → Watashi wa Canda ni sundeimasu.