I've always been a bit puzzled as to why theses things are so firmly fitted. I can understand tight, but chuffin' tight seems a bit extreme as there is a grub screw to keep it located.
Don't be tempted to hit it along the outside edge. You will break it.