[I have taken the liberty of splitting this post into two parts, moving the historical explanation to a new thread, Project Enigma (historical background), along with some comments of my own, so as not to hijack the thread, leaving the part that's actually about this challenge. -bh]
What do you expect applicants to publish? Software? A list of coded messages?
Perhaps you could make a list of, say, 30 messages, and ask applicants to code them and publish the codes in random order; such that you (and others) may try to discover which code corresponds to which original message?
And I think you should draft some rules, like not allowing certain advanced protocols, such as RSA, Diffie-Hellman or DES/AES, or use of commercial encryption services. How about limiting the number of vanilla Snap! blocks to be used encrypting a/o decrypting?