I'm pretty sure that gifs are simply a collection of complete images, while most videos you watch are encoded in a more efficient way. Most video codecs contain all the information for the first frame, and then information for how to change that frame to the next one. (Every once in a while an entire frame is loaded.)

As far as what makes gifs large or small, I think it depends on the format of the individual frames and the framerate. Basically if you have unoptimized images going at a high frame rate, it will take some time to load them all. Ideally you want to optimize the images and choose a conservative but acceptable framerate.

EDIT: Reading below, I realize I'm wrong about gifs, they do use a compression algorithm that prevents loading every single frame independently.