As stated above, there should be a little spring clip. Be very careful. The plastic is brittle, and the shaft rusts. Don't pry very hard (one guess how I know that!). If you can, try getting some WD-40 or the like on it, and then instead of immediately pulling it off, use a socket or something to drive it a little further down on the shaft to break it free. Then try moving it upward a little. If you repeat that process enough times, it will usually eventually slide off without breaking.