Jamie,
A typical reason for the prefetch abort is a double-bit ECC error. Do you enable ECC checking for the flash? If so, you do need to fill the entire available 3MB flash. Any unused flash needs to be filled with a value that could act as a trap if the CPU ever starts executing code fetched from those "unused" locations. One suggested value to use as a fill is 0xEEFF3300.
Please also make sure that you program the ECC locations along with the main program memory.
Regards, Sunil