You can generally cross compile across architectures, but there can be instruction or build nuances that can cause hiccups. Also, you need to build everything against the target architecture as well, not just the kernel.