Marijuana and alcohol? that's cool. You know, as long as you are being cautious. But hardcore drugs? I don't buy it that the person "only does it during the weekends".
There are a lot risks, like someone mentioned previously, such as getting your kids taken away from you. But if the person thinks it's ok and it's not endangering the kids life, then... ok?
You just have to remember you're responsible for your kids life.