The normal way MSP projects are done is by using static linking. This means, the library code is handled (almost) like all other project code. The result is a monolithic binary file that contains all together.
This is the way that gives most compact code, most efficient usage of flash and ram, and as well as causes least trouble.
What you want to do is more like dynamic linking. That means the application knows of the existence of the library and which functions it provides but the library code is completely outside the application.
With an OS that provides memory management and can assign separate address ranges for both, this is quite easy. On an MSP this is rather difficult.
For the code, the best is to have the library have a jumptable to all its functions. The table is located at a fixed memory location. The application then uses a header file that defines function pointers. This adds the overhead of a jump to each library call, but allows application and library to be completely independent except for the jumptable.
More difficult is the handling of the ram space. If the library requires static or global variables, the space required for them needs to be excluded from the application ram space.
Next step is the boot code. Of course the library (if written in C) needs some boot code for initializing the globals. Yet it has no main(). You need your own init code.
Best would be to make all globals uninitialized and write a lib_init function that has to be called by the application and which does the required init of the global vars. Makes the source code a bit less readable, but shouldn’t be a big problem. You can’t initialize local static vars this way, though.
The biggest chunk is if the library required interrupts and has ISRs. In this case, you’ll need to use some double-jump buffering, so all interrupt vectors point to the library area, and those which aren’t required by the library itself are routed to the application. This can be done by moving the app vector table to a different location, and then doing indirect jumps on this table when an interrupt is called. This increases interrupt latency by some clock cycles.
This way, the reset vector can point to the library init function, which at its end calls the application reset vector.
Well, it isn’t easy, but doable.