Fish

Fish are all over this poem. They start at the aquarium and make their way to the very last stanza. Though they may not always act as a symbol for the black infantrymen or slaves, it's a pretty prominent role for them.

What makes the fish/infantrymen symbolism so tricky is that Lowell makes it so inexact. We can never point to one spot in the poem and say, there, that's the key to unlock this crazy comparison that he's made. But he hedges around it so often, and nudges at it, that it's impossible to ignore. As slippery and shifting as this symbolism is, it's present, and it's important to the big picture of this poem. Let's track the fish at the beginning of the poem and at the end, and see how the symbolism develops or changes.

• Line 8: We're introduced to the "cowed, compliant fish." Cowed means that these fish have been intimidated or forced to submit, and compliant, in this case, means obeying to an excessive degree. These are characteristics familiar to slaves—they've been forced against their will to obey whoever has enslaved them. "Cowed" and "compliant" are also characteristics of soldiers. They're supposed to follow their leader, and although soldiers aren't enslaved, they are expected to comply with any orders from higher-ups. 
• Lines 61–62: Okie dokie, this one takes some sleuth-work to unravel. We first saw the bubbles coming from the fishes' mouths at the beginning of the poem when the speaker daydreams about watching them behind the aquarium glass. If we accept that the fish are symbols for the infantrymen (and let's definitely do that, so we can makes some sense of this image here), then Colonel Shaw is riding on their bubble. Weird. But, in an even weirder way, this makes sense. Shaw fought with, and was kept afloat by, his men. Instead of riding a horse like the monument shows, Lowell imagines Shaw on a figurative bubble—something much more fragile, for sure, but also something that was provided to him by the African American soldiers under his command. 
• Lines 66–67: The most important thing to note from these lines is the "savage servility" that is still around in the early to mid-1960s in Boston (when this poem came out). The "savage servility" should bring us back to the earlier description of the "cowed, compliant" fish. People, Lowell is commenting, possess the same servile attitude as the fish, because they have been so long oppressed, and that oppression still persists.