What’s the best way to manage third party libraries in C++ especially when you’re not just dealing with software but several hardware?

I usually just store each library in its own sub module that gets rebuilt each update/pull, but this is started to get crazy as my project gets larger that it is not scaling.