1. The biggest "downside" is the tax implications resulting from a "early withdrawal" from your 401K.
2. It is FAR BETTER to use a home equity loan since the mortgage interest that accrues on a home equity loan is TAX DEDUCTABLE which will lower your tax burden which is better than having to PAY TAXES if you do a "early withdrawal" from your 401K.
3. RECOMMENDATION: If you don't already have one, you should consider a home equity line of credit credit card.
I had one for years, using it like a normal credit card, and got the use of the yearly tax deduction from the mortgage interest that accrued.
Hope this helps you.