This might work: –
Since you have only one unit, set it up to "recognize" your voice only. Once that's done, go into the settings and pick your calendar. In settings you will also find an option to pick a calendar for "unrecognized voices" use this to pick the other calendar.