Sleep is a highly regimented process, progressing in cycles through the night. You may think this is the only important cycle in your sleep habits, but in reality, routine matters in more ways than simply your circadian rhythm. The body likes consistency, and that means at bedtime too.
When you go to bed at radically different times every night and wake up at variable times in the morning, your body doesn't know what to expect. This lack of consistency can make it far harder to relax, settle down and get ready for bed — when your brain isn't expecting to sleep, getting into a nighttime mindset can be a challenge.
Even if you have an inconsistent schedule, do your best to go to bed at the same time every night and wake up at the same time every morning. When your body is used to getting to sleep, you'll have an easier time shuttting down and getting some shut-eye.