I do not necessarily believe you need to accelerate his learning (as you state in your question). Little ones like this tend to do very well acquiring knowledge quickly all on their own! Offer him a variety of toys and books, as well as fun learning experiences -- just like you have been doing. A good book on this topic is Bringing Out the Best by Saunders and Espeland (Free Spirit Publishing). It focuses on gifted children under the age of seven years. Best wishes.
The best way to determine intellectual giftedness is through formal educational assessment testing.
My four-and-a-half-year-old son is reading, adding, subtracting, and memorizing a lot of different things. Is this normal for his age, or may he be gifted? If so, what should my next step be to accelerate on this?
A four-year-old who is already reading and doing primary level mathematics is indeed advanced. Though giftedness can certainly be observed, the best way to definitely determine intellectual giftedness is through formal educational assessment testing. If you are in the process of planning for kindergarten, between now and summer would be a good time for testing. The Wechsler Preschool and Primary Scale of Intelligence would give you an IQ score plus a breakdown of your son's areas of academic skill strengths and weaknesses. It is important that he have a reasonably good attention span (and be in good health the day/days of the testing) for the results to be accurate. Be sure to ask the psychologist in advance if she has experience testing young children. If private testing is too expensive, you can consider contacting a local university with a psychology department for competent but possibly less expensive testing.