I see them as a package deal so yes they both have to go.
Now as much as I think Polian is a complete and utter douchebag he was a good GM for the Colts from 1998-2006 but now? He's a liability and his antics aren't tolerable anymore with the team in shambles. There are other GM's the Colts can get that are out there and unlike Polian are hungry and willing to prove something and not rest on past success and arrogance.
This team needs to change drastically otherwise it will continue to get worse(hard to believe but yes it can)