Hi, BoyFromNowhere, and welcome! I do agree with what you're saying about how unfair the relationship is to Alma (and Lureen, and Bobby, and Jenny, and Alma Jr. . . . ). Ennis and Jack's families certainly don't get these men's full attention, and so they also have less satisfying, less loving lives than they deserve. But I think the movie's point--as you also seem to imply here--is that Ennis and Jack themselves are treated unfairly by those around them, too. If society weren't so homophobic--if Ennis in particular hadn't internalized those feelings and fears from his own family--perhaps they could have lived the lives together that they wanted to, the lives that they DESERVED to live. :\'(
To maybe get a little more back on topic here, I'd also add that these fishing trips let them have their true lives together if only for brief periods of time. Jack really did turn out to be Ennis's "fishing buddy" in the end.