Honestly, for a game that doesn't have open PvP, I think SWG did a pretty good job (combat balance aside, faction rewards being crap aside, etc. I'm talking the combat engagement situation -- and I'm talking pre-NGE, I don't know if it changed after that or not).
Holy crap this got long... if you know SWG's system, feel free to skip down to the next bold stuff.
Basically how it worked was, there's Empire and Rebels for the two sides... you PICK which side you want to be on, or if PvP isn't your thing at all, just don't even pick a side. If you don't pick a side, you can't attack rebel or imperial NPCs or players. If you do pick a side, you can attack the other side's NPCs. If you do attack the NPCs, you will become attackable by the other side's players. Otherwise if you talk to a recruiter NPC, you can choose to flag yourself for PvP. If you declared yourself via a faction NPC, it took like an hour to go away if you wanted to turn it off, so you couldn't hop in and out of being flagged to PvP (although there was a grouping 'exploit/bug' thing to go around this, but that was not ideal, so I'll dismiss that for the sake of describing the ideal setup)
Personally I prefer open PvP, but the way the SWG system of flagging worked way back when it launched was pretty decent. Not to mention there is full communication between all players, and you can SWITCH teams any time you want to (well you have to grind rep... when your reb and your rebel rep goes up, your imp rep goes down double). So if you decide your faction is full of asshats, spend a week grinding, swap teams and then go after em. Also you could purchase faction at a cost, via a smuggler, which was a pretty cool... "lore" type way to get it done quick, although it was spendy. Of course, since that game actually had community, switching sides was a major deal for some of the servers stronger players or guilds. It could have a huge effect on the server. Really I just hate being pinned down to one side.
And then for such a "crafting/non-combat profession" intensive game, I was actually really impressed that they thought to add guild wars to the game. You could declare war with other guilds, making all of that guild's members constantly flagged to your guild for PvP, and vice versa. The coup degrace of guild wars was to camp someone so hard they de-tagged to get out of it. If you could accomplish that, you truly kicked the shit out of some other guild, if they couldn't bring back-up to get the guy uncamped. If WoW had a guild-war system so that alliance and horde could go at it both cross-faction and inter-faction in a world-PvP setup, I might think more of the PvP system in the game, even though it wouldn't change THAT much, it could provide some much needed rivalries in the game between people who could talk shit or congratulate each other.
Hell, even duels there mattered more. The deathblow system was truly a pretty cool way to have it set up. Once you dropped someones hitpoints to 0, they were "incapacitated" and just laid there on the ground for anywhere from 10-30 seconds while they regened to 0. (they would go to l ike -100 hp or somethin) and you would have to walk up to melee range, click their corpse and choose deathblow to finish them off... (or you could macro it) But even in duels you could kill someone. Or let them live... and in PvP, even in heavy ranged combat, only riflemen could finish someone off from range, and medics/doctors could drag bodies to pull them out of the fray if they got incapped.
Now... SWG had a ton of other problems that I won't get into, mostly balance issues and shit like that, not to mention a horrible relationship between the devs and their players... but honestly, some of the things they did in PvP were pretty cool.
And they only really had a minor decay penalty for dying, which could be negated by purchasing insurance... so really it was kind of like dying to a mob in WoW... *but* the game was pretty much impossible to play without buffs, which increased your stat pools GREATLY. Some poeple might have said that was a bad thing, but I liked it, because in PvP, with two buffed players going at it, the guy who died was gonna be out of the fight for anywhere from 5 minutes to an hour while he got fresh buffs. (depending on location and finding a buff doctor) Not to mention they cost you at least 15 minutes worth of cash grinding as well.
So every death cost you maybe 15-30 minutes worth of cash from grinding, plus maybe another 15-60 of travel and hunting for buffing characters.
It wasn't super harsh, but at the same time, it kept people from zerging too much (except CMs for those of you keeping track, but again, that's a balance thing
)
But yeah.... Prefferably I'd like a fully open PvP system. But if it wasn't open, I'd like to see something like SWG's...
**Pickable, switchable teams, with some inconvenience in switching, but not making it impossible to switch
**PvP flagging that takes a lil work and doesn't allow people to just jump in and out of fights at will (have to find an NPC in a major city to get in and out of being flagged, and can't drop flag instantly)
**Sizable, but not horrendous death penalty that
takes someone out of the battle for a while
**And the option between killing and just beating someone, at all times, death even from duels, and leniency even out in the world... both good, not needed, but it was kinda cool thinking back on it.
**Communication between all factions!!! (god I hate that about WoW) Nothing makes the PvP better than being able to stir shit up with someone and create a hatred rivalry, or being able to tell someone "Hey great fight" and become friendly rivals. The reason I like Halo better than PC fps's, is because I never play Halo online. It's more fun to be ribbing the guys sitting next to you while you play it. PvP in MMOs is the same way IMO. It's way more fun when you can talk a lil trash or go "Oh damn, you got me good there, nice one." or whatever.
*******NO INSTANCES********
And do not take this as an endorsement to play SWG... Stay the hell away from that game. SOE is shit and they ruined it.