Continuous integration with home libraries

We have a project like coresystem that uses several built-in libraries, all in different versions.

The build configuration for the server system is configured to reference libraries with specific version numbers, for example, coresytem 2.3.4 uses the abc-version-1.2.3 and def-version-3.4.5 libraries.

These libraries change frequently at the same time as the server system. And not necessarily the same set of libraries change with every kernel version.

How do we handle continuous delivery in this case. We currently have to constantly change the build configuration.