Phalanx

[PMBardunias]

Surley they had spears protruding from the sides and possible the back like the front? Basically like a square formation?

You must think beyond the single unit. While each component part might be 16 ranks x 16 files, if you approach from the front you have a more or less continuous line for hundreds of meters. If you approach from the flank you face 16 men. You then easily surround them and "roll up" the whole line.

Attacking the rear of an already frontally engaged has the immediate effect of limiting their mobility. Ancient battles swayed back and forth as one side pushed the other back, then the other rallied. You could tell who was winning by the way the dust cloud angled. The inability to give ground was one of the worst things that could happen, you end up crushed into too small a space and men cannot fight properly. Also, the order of men from front rank to read was designed to put the best fighters in front and the steadies, perhaps oldest, men in the rear. Reversing this order is problematic.

To me any good general would try to maximise on this i.e send men to attack the sides of the Phalanx.

They did, Spartans in particular did or attempted this many times. The problem is that a phalanx does not easily lend itself to the type of small unit tactical flexibility needed to pull off such a manouver. If you bend your line around your enemies flank, often your line on the right extended beyond theirs on the left, then you just expose the flank and rear of your own troops to any enemy troops on the phalanx's flank.