Is it more difficult to forgive yourself if the person you hurt doesn't forgive you
Yes, it is more difficult. While you can forgive yourself while the person you hurt doesn't forgive you, it's relatively more difficult than it would be if they did. However, I must add onto that and say, you can still not forgive yourself for something the person you hurt does forgive you for. It all depends on what the situation is, and what state you're in, mentally. If you accidentally break someone's arm, for example (I've been in some tough situations) even if the person whose arm is now broken (thanks to you) forgives you, it can be astoundingly difficult to forgive yourself.