Sounds like the Pi is the determining factor on where the audio output is going to be. I assume you have the Pi because it's being used as a media player. It is likely that the Pi decides what speaker to use as its output. If it decides the TV's speaker is the output device, then the Echo becomes in input device. If the TV is not part of this setup, then automatically both input and output are with the Echo since it's the only device. Because there's something between your devices (namely the Pi) and the speaker, it's not the Echo that decides where the audio output is. The device in the middle (Pi) is where your troubleshooting should be focused on.
How does the Pi decide what output device to use? I can only guess that it depends if the TV is on or off, if it's plugged into power, how many hours it has been off, how many hours or days since the TV was last used with the Pi, what software/media player you are using on the Pi (and its settings), or any number of other scenarios. But the troubleshooting must start with the Pi.