The question arose last week: how do you design around/against people being idiots and jerks? “You can’t fix stupid.” There is no 100% solution, because some people really are that dumb and others will go to great lengths as griefers, but there are better and worse designs in terms of the behavior they reward. If the system rewards pro-social behavior, it promotes harmony. If the player must make sacrifices to help others, you will see destructively selfish if not predatory behavior. Economics in two words: “incentives matter.”
For example, consider Marks of Triumph in The Lord of the Rings Online™: Shadows of Angmar™. The epic quest chain is a big feature for LotRO, but it was punctuated with instances that demanded full groups. If most of the population had completed them all, how did newer players and alts get through the epics? You asked someone to repeat one. Repeating one was a way to help friends, but you got jack for it. Your friends had to give something up, and you would not meet new people unless someone was a very charitable stranger (or, lucky day, you find a few people who need it, a couple of whom have charitable friends). Game update: repeating one of those instances began to award (once per five days) a Mark of Triumph; accumulate several Marks to barter for various rewards. The rewards were rather nice for when they were released. Pro-social behavior increased.
Because of how Marks were awarded, you did not need someone new to repeat the quest. This has the further benefit of letting you repeat older content without completely sacrificing character advancement, and developers want players to pay for recycled content. The downside is that it is more efficient to get a level-capped group and cycle through all the Mark instances rather than actually helping near-cap players on their first run-through. On balance, however, Marks increased pro-social behavior more than they inhibited it.
Looting is a major source of drama and anti-social behavior. Did you know that you cannot ninja-loot in City of Heroes? Not that there is much that you could ninja-loot, but there are not even corpses to loot. Rewards just instantly appear in each character’s inventory. So one person randomly gets the reward from the raid boss? No, everyone gets one. There are drawbacks to every system, particularly given the adjustments necessary to adapt it to another game, but there are ways around all the problems you assume.
Consider the related problem that there is nothing for you in that instance. Alice wants to run the caverns for her tanking trinket (best in slot!), Bob needs the mountain for his shoulders, and Carl is grinding crabs by the lake in case one drops the resource he needs. They could help each other, but that involves giving up their individual progress: pro-social behavior requires an individual cost. Or we could set up a different system for how you get “best in slot” that will not give people competing demands, such as a token/badge system that lets you adventure anywhere and then exchange for your rewards. Of course, that will encourage grinding in the most time-efficient area (drawback).
LotRO has an interesting mix on those loot systems. The top-tier armor involves tokens that you can earn in sets of dungeons, so you are not bound to any one. Freedom! When Mines of Moria™ was younger, you saw people who would never go to instance X again because they had their radiance pieces from there. So the armor rewards are mostly solved, and the weapon rewards are on a similar system, but all the other slots are on the same system in which the best tank ring drops here and the best DPS bracelet drops there. Is that a compromise between pro-socialization and encouraging people not to focus on the most time-efficient dungeon?
Consider the DPS meter. On one hand, it can be useful in assessing your performance and seeing where the group could improve. It frequently becomes its own mini-game at the expense of good tactics, with DPS encouraged to deal as much damage as possible to as many targets as possible as quickly as possible. Sometimes that works well, sometimes that causes wipes, and the tanks and healers frequently want to choke people. Also consider the gap between what is easy to measure and what helps the group.
Consider achievements for doing various absurd things. Someone on your Team Fortress 2 server is probably trying for one at any given moment. That Medic is not healing anyone because he is hunting for Scouts. That Heavy is just standing in a corridor spamming G in the hopes that someone will walk into a taunt kill. You see similar things where there are rewards for suffering; I left characters intentionally getting chain-stunned overnight for City of Heroes badges, and people also farm damage there. You get perverse results if the tank equivalent of the DPS meter measures damage taken and effectively punishes having high defenses (damage prevention is not measured).
Consider achievements for working together. The best of them will be for normal cooperation as opposed to situations you would perversely work to create. The TF2 Engineer achievements have lots of good examples. Identify how you want people to work together and place a bounty on it.
We could also consider pro-social design from the group perspective, rather than the individual. Consider how FFXI approaches (approached?) level ranges in groups. Everyone needed to be in a very narrow band, or else there were severe penalties. “Ding!” “Grats! /kick” Consider how CoX approaches level ranges in groups. Everyone is automatically the same level, which is exactly the right level for the instance. Group with anyone anytime: perfect. (We have discussed the drawbacks of the CoX instancing/scaling at length, and we will surely do so again.)
Really, pick any mechanic that sets the interests of the individual against the interests of the group, rather than aligning the two. People respond to incentives. If people are rewarded for leeching, stealing, or griefing, it will happen. Yes, some people will always be schmucks, but it will happen a lot less if you do not pay them for it.