Why is Electricity & Magnetism difficult to understand?

I'll have to admit that I struggled with it a little bit.  I think much of the trouble is related to intuition, just like you said!!  Since we can't see electric charges (unless they jump a gap like a spark, or lightning), and we can't see electric or magnetic fields, we don't have experience in dealing with them everyday, and they don't make quite as much sense as kinematics.  You've been throwing baseballs etc. for many years, and you understand the trajectories they form, you understand that when you throw harder they go faster, higher, farther... it makes intuitive sense.

With electrical phenomenon, for most people it's a new set of thought processes.  Most people do not go around measuring the magnetic field at the center of a solenoid, etc.  But there are things that you can relate, for example, scuffing your feet on the rug, or petting the cat causes a transfer and related accumulation of charge, which causes a potential difference (voltage), which causes a sudden charge flow (current) when you touch a doorknob, etc.

I got better at this subject with a little practice, but I agree, circuits are easier.

Good luck!!