Ok, one hour later and some reading later.
I changed my machine.config and web.config to allow override trust trust level.
Then, even without using your custom medium trust, if I set the trust level to high, it still didn't work. With full trust, it worked. A couple of testing and switching...