High-resolution magnetostriction measurement of $Delta L/Lsim10^{-6}$ at a speed of 5 MHz is performed, using optical filter method as the detection scheme for the fiber Bragg grating (FBG) based strain monitor is performed under 35-millisecond pulsed high magnetic fields up to 45 T at 2.2 K. The resolution of magnetostriction is about the same order as the conventionally reported value from FBG based magnetostriction measurement systems for millisecond pulsed magnetic fields. The measurement speed is $sim$100 times the conventional ones. Present system can be a faster alternative for the conventional FBG based magnetostriction measurement system for millisecond pulsed high magnetic fields.