You might be right on both points, especially the latter. Ophelia and Brenna both acknowledge David's cruel behavior towards Sterling; perhaps the family had become dysfunctional in the four years since Sterling left.
Sudden thought: maybe David and Ophelia grew apart after Sterling left, and David turned to Drew for "companionship"?
All possibilities, but I wouldn't read much into the phrase "taken from me/us". It means someone's
life was taken, not necessarily that their
body was taken. It's a common expression when talking about people who died young, whether "taken" by another person (e.g. a drunk driver) or simply "taken" by God (as in the case of an illness).
I'm thinking something along these lines...