If you have a trap door to your attic or crawlspace, it could be a chink in your insulation armor. Make sure you weather strip and insulate it.
To insulate a basement or crawlspace trap, cut and glue rigid foam insulation to the underside of the door where it fits between the floor joists. One 2″ sheet will give you R10; use two for better insulation. You can glue the panels to the board and to each other with an adhesive designed for this purpose, which comes in a caulking gun. It takes some time to set up, so don’t put stress on it for at least 24 hours.
You can also sandwich the insulation between the door and a sheet of thin plywood, bolted together.
Do the same to attic trap doors. Depending on which way the trap door swings, you’ll insulate differently. If it opens up into the attic, you may be able to cut the insulation to overlap the door.