Explain Like I'm 5 (ELI5)

I’m really good at math and I have a decent grasp of computer science. I understand that multiplying two prime numbers to get a huge number is easy, but checking out if a huge number has only two prime factors is a monumental task for a computer. What I don’t get is how this is used for encryption and coding and decoding messages. I keep reading about this in books and they keep talking about how one side is the key or whatever but they never really explained how it all works. Every book seems to love explaining the whole large-numbers-take-a-lot-of-time-to-factorise concept but not how it actually works in encryption. I understand basic message coding–switch around the alphabet, add steps that changes a message into a mess of letters; then the recipient has to do all those steps backwards to change it back. How do prime numbers and huge numbers fit into this? How does knowing a pair of factors enable me to code a message and how does knowing the product enable my recipient to decode it?