Areas of Effect
Instead of keeping track of everyone's exact physical location on 2D or 3D space, people are in abstract ranks and in "mêlée groups". Each spell or damage attack instead of targeting a specific damage "shape" (like a cone or a line) is translated to a number of targets instead. For example, a fireball hits up to four targets. Add a d3 if they are described to be standing unusually tightly and subtract a d3 if they are standing unusually far apart.
These targets can be in separate mêlée groups, but before targeting anyone in a new group, you need to target everyone in the previous group.
Another way to phrase that exact same rule is that you can only have one group where you "spare" people from being hit.
Let's say our beloved heroes Alice, Bob, Carol and Ted are fighting the horror villains Skeletor, Dracula, Mumm-Ra, and Jace Beleren. Alice is fighting Skeletor, Bob and Carol are both in a big group with Dracula and Mumm-Ra, and Jace stands alone, prepping to sculpt someone's mind.
Ted is about to unleash a dangerous six-target spell. He decides that he wants to hit all four of the horror villains. He isn't an evoker so he doesn't have Spell Sculpt (a special ability to spare some of your friends) so some of his own friends might get a li'l bit burninated, too.
He can spare either Alice, or Bob and Carol, since he can only spare people from one group.
He decides to spare Bob and Carol, which means Jace, Skeletor, Alice, Dracula, and Mumm-Ra all need to make saves because they are being put on blast. That's only five targets, but that's fine, he doesn't have to use all six targets.
A mental model for how this "sparing" works is that you are only blasting the edge of the group. If you are hitting multiple groups, you've got to blast through every group completely until you get to that last group where you can be sparing.
Let's say Ted wants to follow-up with a two-target spell, another blast spell on the villains. What are his options?
- Blasting Jace plus any one other villain? Sure. The danger of standing alone. You'll get hit by a blasting spell that hits the edge of another group.
- Blasting Dracula and Mumm-Ra? Great. They're in the same group.
- Blasting Skeletor and Mumm-Ra? Not possible. He'd need to spare people from two groups (Alice in Skeletor's group, or Dracula, Bob, and Carol in Mumm-Ra's group). You can only spare people in one group.
There is a math formula, first seen in the 5e DMG, for converting from the various "shapes" like cube or sphere to a plain number of targets. For most shapes, you divide by 5 but round up (ceiling). For cones, it's by 10 instead (so Burning Hands is a two-target spell) and for lines it's by 30 (so Lightning Bolt is a four-target spell). Again, if it's established that the targets are unusally tightly clumped or spread far apart, add or subtract a d3. Having been using this system since 2014, the spells kinda quickly become known for how many targets they hit as opposed to how what shape it is. For example, I don't know by heart what shape the three-target spell Thunderwave has, I'd have to look it up (I did, and it's a 15 foot cube).