There seem to be different understandings about the mechanics of reducing magic damage, so I thought I'd try to gather everything together here. There are basically 4 ways to reduce magic damage:
1) Resists
People have had great success tanking Tiamat and such with fire resistance setups. 300+ seems to be the magic number to get full resists on most elemental attacks. However, the amount of damage taken varies randomly, and sometimes you can still get hit for full damage.
2) Magic Defense Bonus
Multiplies magic damage by 100/(100+Total MDB). For instance, MDB+12 from WHM37 subjob reduces Fafnir's Hurricane Wing from 368 to 368*100/112=328.
Because MDB is a divisor, it has diminishing returns. If a magic attack normally does 100 damage, MDB+10 will reduce it to 90, but MDB+20 will only reduce it to 83.
If you have a Barelement spell of the appropriate element cast by a WHM with Barspell merits and/or Blessed Briault, you get +2 MDB per merit and +5 MDB from Briault.
MDB does not affect Mijin Gakure damage.
MDB does not cap.
Shell does not give MDB.
3) -Magic Damage Taken
Reduces magic damage directly. Listed percentages on gear are actually an approximation for a fraction of /256. Magic damage is multiplied by (256-Total -Magic Damage Taken)/256.
Shell factors into this term: Shell II is -36/256, Shell IV -56, Shellra V -62, -64 or -66.
Like physical damage reduction this term caps at -128/256 damage, but MDB and resists can continue to reduce damage once that cap is reached. This is most significant for Aegis PLD, since Aegis+Shellra V.2 is enough to cap this term without any additional -magic damage gear.
This term does affect Mijin Gakure damage. I've read several posts saying that Shell doesn't affect Mijin, but in Dynamis-Beaucedine on Tuesday there were two Mijins, and the tank party I was in had the damage reduced exactly by the amounts above. On the first I had Shell IV and the others had Shell V, and the damage we took varied accordingly (base 1093, Shell IV 853, Shellra V.1 828). Furthermore, since Shell is reported to factor in to the same term as -Magic Damage Taken gear, it wouldn't make much sense for Mijin to be affected by gear but not Shell.
(Aside: Diabolos seems to take 1/8 damage from Mijin for some reason, has anyone else noticed this?)
4) INT
As Genosync pointed out below, adding INT reduces magic damage from Elemental Magic spells, and sometimes from other sources of magic damage, notably Gear/Chariot AoE WS in Salvage.
References:
http://wiki.ffxiclopedia.org/Calculating_Magic_Damage
http://www.bluegartr.com/forum/viewtopic.php?p=457670
http://www.bluegartr.com/forum/viewtopic.php?p=432694
http://www.bluegartr.com/forum/viewtopic.php?p=376144
Edit: Barspell update
Edit: INT
Edit: minor updates
Edit: corrected shellra V numbers