Hi Courtney -- thanks for writing!
Without knowing the details of the programs at your University, it's difficult to offer specific advice, but I'm happy to suggest ways to think about this. I believe that your major should reflect your interests and be the best representation of the courses you want to take. So the first thing to do is go through your school's course catalog and identify all the courses that you would really love to take.
Having done that, do the analysis: how many of those will help you finish your math degree? How many would be appropriate for a computer engineering degree? In other words, in the absence of requirements -- would your passion take you deeper into math or more into computer engineering?
Then look at the other side -- given the requirements of the two majors, how many courses that don't make your "courses I want to take" list would you have to take? The last thing you want to do is spend your final two years in college taking classes that you don't want to take.
So, depending on how that analysis goes, you will either decide to stay in math and pursue CE on your own, or you'll switch to CE. I'm assuming that if you switch to CE, you'll not have a lot of flexibility in what you do, so let's talk about what it might look like to continue in Math but learn some CE. How many course slots do you have open to take classes you want? Are you allowed to cross register in the school of engineering to take the classes you want? Do you have the prerequisites to do so? If you can take a lot of the classes you want while finishing your math degree, excellent! If not, then you might explore possibilities for online courses that help you develop the skills/background you need.
I hope this is useful!