Get closer to your Dot. When using your Dot to make phone calls, it becomes a speaker phone. If you were using a regular speaker phone, you probably wouldn't be across the room. You'd be 2 or 3 feet away at most. And just like all speakerphones, if there is a lot of background noise on the other end (TV, radio, driving, other people, etc.), they will not hear you unless you speak louder than that background noise.
Since most Echo devices have a microphone array that is able to determine where you are in the room, you should face the Dot as you speak. If you turn your head, the microphone array will not pick up your voice from the direction that it thinks you were speaking from.