The History and Mathematics of Codes and Code Breaking

Tag: frequency analysis (Page 1 of 2)

You Don’t Need to Be Trained to Be Professional

With sophisticated and detailed research on statistics, al-Kindī invented his system of cryptanalysis, later known as the frequency analysis. It’s not surprised that he was considered as the greatest scientist in the ninth century when many disciplines including mathematics, statistics and linguistics that are well developed today were still in their rudimentary stages; thus his research was undoubtedly remarkable. More importantly, the public, especially amateur cryptanalysts were not educated as well as those in today’s education system; few people were likely to have the opportunity to master, or even learn mathematics. Therefore those attempting to decipher encrypted messages had to depend merely on al-Kindī’s approach. However, things have changed.

Today’s schooling system guarantees the educational opportunity for almost everyone to learn basic mathematics and statistics, and to have personal perspectives on languages. As long as we comprehend the linguistic rules, do the math and then go through the process of trial and error, all of us can crack codes in our own ways.

The easy access to various resources and information should not be understated as well. Simply searching ‘cryptography’ on the Internet, even high school students are probably capable of decrypting substitution cipher. For those who want to dedicate more time and truly dive into code breaking, thousands of books will be available if they are willing to reach them. Learning new theories and grasping the nature of any discipline without instructions are no longer the missions impossible; in other words, self-study is relatively more feasible than ever before.

Generally, the simple truth is, we are getting more knowledgeable and everything is getting more accessible. Though code breaking is hard, the actual barrier for amateur cryptanalyst today may be their willingness to reach the resources and their persistence as well as patience when deciphering codes with complex encryption.

Modern Cryptanalysis Geniuses?

In The Code Book, by Simon Singh, the discovery of cryptanalysis is discussed. It is explained that without a strong background in core disciplines, cryptanalysis is impossible to achieve. Mathematics, statistics, and linguistics are vital in the development of many methods, such as frequency analysis. Earlier civilizations lacked a certain amount of efficiency in these fields, and that is why cryptanalysis was not discovered until around A.D. 750. The discovery was made in the Islamic civilization during a time when the arts and sciences began to explode with breakthroughs. It took years to become masters of cryptoanalysis, like they were.

It is not uncommon in modern society that an amateur cryptanalyst is able to crack a simple substitution cipher using the same method of frequency analysis, without ever being formally taught it. Are all modern cryptanalysts just naturally born geniuses in the art of code breaking? No, that's not likely. Instead, our society has just developed significantly since the year A.D. 750. People are given a better foundation when growing up in the arts and sciences. Mathematics, statistics, and linguistics are taught to all people, in varying degrees of course, but these fields are still emphasized. Civilization has advanced so far that many people have a basic understanding of subjects, that very few people knew about in ancient times. This is the main reason why amateur cryptanalysts can decipher substitution ciphers with out extensive training. To people in today's day and age it is just a "logical" way to attack enciphered messages, opposed to the people in ancient times who worked tirelessly to unlock the secrets of mathematics.

Easier to Learn, or Easier to Access?

I believe that, while a high level of scholarship was required to develop the frequency analysis approach, it is not critical to the use of this approach. When the world was new to this subject--when it had just discovered ciphers and keys and cryptanalysis--all of the knowledge was completely new. It was the cutting edge, so not many people understood it yet. It was essential to attain a high level of education to comprehend the mysteries of cryptology. However, with the modern education system, and modern technology, people have the information necessary more readily available. People can access the "mathematics, statistics, and linguistics" necessary to equip themselves for code making and codebreaking. Also, the easy access means that the information surrounds the human population. We have billions of pieces of data sitting at our fingertips, just waiting in that ever-present "cloud." Because of this access, and as a result of the heightened academic expectations, "amateur" cryptanalysts can use previously lengthy and difficult methods of analysis with much more ease. The civilization has reached a "sufficiently sophisticated level of scholarship in several disciplines," and therefore the people of that civilization may achieve the same accomplishments which the Islamic civilization discovered. However, as a result of the constant inundation of information prevalent in our society, and the resultant size of the body of common knowledge, amateur cryptanalysts can now use approaches such as frequency analysis, which was so arduously sought out, without any formal training.

A Developing Familiarity Throughout History

Fundamental development in the disciplinary topics of mathematics, statistics, and linguistics was procured from a comparative ground-level hundreds of years ago as opposed to what we have unearthed today. The advantages and resources currently available to the vast public are, of course, the most they have ever been in history. This goes without saying. "Discovering" tactics to break codes and ciphers that were once considered the most advanced techniques by exceptional cryptanalysts is certainly not as easy a task without the long history of code breaking (in the colloquial meaning of the term) that had come before us.

The ability to learn methods such as frequency analysis from a quick Google search is much less arduous a task than inventing them without any previous notion of such a possibility. Even assuming that today's amateur cryptanalysts aren't explicitly searching "how to's" from public databases, the idea of frequency analysis and any analogous general form of use is very comfortable and familiar. Perhaps teachers from grade school distributed puzzles aimed to unscramble words and phrases or your classmate used a simple cipher as a way to ask out their prom date. Experiencing or seeing a number of similar events throughout our lives inevitably ingrains the technique somewhere in the back of our minds, at least implicitly.

Noting the above, it is truly incredible to acknowledge how commonplace once incredible and cutting-edge discoveries are considered in the present day. This will always be observed, even beyond subjects regarding cryptography, as a natural progression of time.

Cryptanalysis: From Complexity to Common Knowledge

Each an art form of its own, cryptanalysis and cryptography demonstrate opposing counterparts focused on accomplishing the same common goal—the understanding of a hidden message. These two techniques highlight the competitive battle between codemakers and codebreakers. Although cryptography requires a distinct level of skill and secrecy, the practice of cryptanalysis encounters even greater obstacles as the codebreaker must determine the meaning of the hidden message as well as the technique necessary to break it. Arguably, the mastery of one skill can lead to an expertise in the other as the making of a complex cipher derives the further logic and creativity necessary to uncover these intricate codes.

Singh cites the frequency analysis technique as an “innocuous observation” by Muslim cryptanalysts that became “the first great breakthrough in cryptanalysis” (17). Nowadays, this code-breaking method is quickly and easily used by first-time cryptanalysts with no previous instruction, almost as if by second nature. While the frequency analysis technique was undoubtedly a major breakthrough in the seventh century, the vast amount of education and technology provided to our society today allows this method to become an obvious first step towards discovering the unknown.

As time has gone on and technology has expanded, the human mind has reached a common intelligence almost unimaginable even one hundred short years ago. Education has taught us to not only focus on how to put things together, but also on how to take them apart. Practices such as cryptanalysis have become more applicable to the average man as common knowledge typically requires an understanding of both how and why things work. The fifteenth century Western world is a prime example of the human tendency to discover how something functions as immediately after cryptography was introduced “already there were individuals attempting to destroy this security” (27).

While in the past cryptanalysis was labeled as an expertise only accessible to those in higher society with the finest education, its ability to be understood by even the most amateur cryptanalyst emphasizes the incredible expansion of knowledge in our society today.

The Infinite Struggle Between Makers and Breakers

The continuous tug-of-war between cryptographers and cryptanalysts has triggered the expansion and prominence of cryptography in the world today. Just as quickly as new encryption methods are developed, new decryption strategies are implemented, creating a state of equilibrium in cryptography.

The art of secret writing dates back to Herodotus (Singh 3) and has had various uses in history, including military communication and murder plots. In more recent years, the explosion of technology has propelled the expansion and necessity of cryptography to the point where it has an effect, either direct or indirect, on every individual. Cryptography has a range of important functions, such as keeping bank account information private and ensuring that government information is kept secret.

Cryptanalysis was not invented until the Islamic golden age under the Abbasid caliphate (Singh 14). According to Singh, cryptanalysis could not be invented until a civilization had reached a sufficiently sophisticated level of scholarship in several disciplines, including mathematics, statistics, and linguistics (15). The Muslim civilization provided an ideal cradle for cryptanalysis because the Muslims were educated in a variety of fields and pursued knowledge in all of its forms (Singh 15).

This breadth of knowledge associated with the Muslim civilization is valued in most modern society today. Most individuals are at least somewhat proficient in an assortment of disciplines. Primary education sets the field by educating children about a variety of different subjects in order to equip them with a large intellectual toolset. Many secondary education institutions stress a liberal education in which students obtain an educational background in the humanities as well as the math and science fields.

Because of this, it is no longer necessary to undergo formal education in cryptanalysis in order to use elementary strategies such as frequency analysis. With the broad education that modern citizens receive, combined with the availability of abundant resources, especially those on the Internet, many of the older, simpler codes can be decrypted by the layperson. However, modern encryptions are much more complex than those of the past and may require serious study.

During the past few centuries, cryptography has exploded due to the expansion of technology. Cryptography affects numerous fields ranging from government information to banking, and brings in knowledge from a variety of areas including mathematics, logic, linguistics, and statistics. Modern education allows us to engage in the world of cryptography even as the complexity and number of cryptographic techniques increase exponentially.


The part of Cryptonomicon that caught my attention was Lawrence Waterhouse's attempt to solve the cipher, or "mathematical exercise" given to him and others by Commander Schoen. Schoen writes out the cipher, a list of 5 groups each with 5 numbers. The numbers are either 1 or 2 digits. Waterhouse instantly recognizes that the greatest number provided is 25, thus he assumes that the numbers must represent the letters in the alphabet. He then decides to run a frequency analysis test using the numbers on the board and realizes the number 18 occurs 6 times. Waterhouse then makes an assumption that the number 18 must be the letter E so he mentally substitutes the letter E into the cipher. Next, Waterhouse observes that the opening 4 numbers are '19 17 17 19'. He then knows that if 19 is a vowel, 17 must be a consonant or vice versa, and since 19 is twice as common in the cipher, he assumes that 19 is a vowel (specifically A). Waterhouse then proceeds by using the context of the message as provided by Commander Schoen earlier. Schoen had said that the message was intended for a naval officer. Using this, Waterhouse was able to guess that the first word was ATTACK. Immediately, he saw the rest of the cipher decrypt before his eyes so he stood up and stated his findings emphatically. The message read: "Attack Pearl Harbor December Seven"

The methods Waterhouse used to solve the cipher reminded me of the discussion we had earlier than year about the use of intuition to solve puzzles. Waterhouse used problem solving techniques and pure logic to decrypt the cipher. By assuming the first word was 'ATTACK' and recognizing that the numbers represented individual letters in the first place, Waterhouse demonstrated the problem solving techniques we have all acquired through basic learning.

The Accessibility of Knowledge

In Simon Singh’s The Code Book, he states “Cryptanalysis could not be invented until a civilization has reached a sufficiently sophisticated level of scholarship.”  If such a sophisticated level of scholarship was needed to invent cryptanalysis, and cryptanalysts were considered to be of ‘higher’ scholarship, why are amateurs capable of decoding ciphers themselves?

Is the world becoming more intelligent? That I do not know.  But it is obvious that knowledge is spreading and learning is occurring more vigorously throughout the world than in the past.  As children grow, they are expected to conquer challenges using the mind, not their fists, and grow mentally as they progress through the education system.  They are expected to continue their path of learning to higher degrees in high school and college, and expand on their critical thinking abilities.  Therefore, society's expectations have furthered amateur cryptanalysts ability to decode messages.

Frequency analysis is a very complex approach to cracking a code.  Yet our minds are capable of applying frequency analysis subconsciously when the opportunity presents itself.  For example, in the word game, Hangman, our mind subconsciously guesses letters that are frequently seen in words (vowels, specific consonants).  Ultimately, people can only apply “on their own” approaches if they have been exposed to higher levels of critical thinking.  Education and the accessibility of knowledge provide the world with these abilities to critically think and solve problems. Amateur cryptanalysts, though untrained, can accredit their code breaking skills to the education they’ve received from such an early age.

Cryptanalysis Evolving Over Time

In Simon Singh’s book, The Code Book, it is expressed that “Cryptanalysis could not be invented until a civilization had reached a sufficiently sophisticated level of scholarship in several disciplines, including mathematics, statistics, and linguistics.” I think this is a very true statement because cryptanalysis involves a level of mathematics and linguistics that wasn’t readily available to many people in early civilizations. The argument that amateur cryptanalysts intuitively use frequency analysis on substitution ciphers and therefore a level of scholarship isn’t needed to solve ciphers is truly invalid in my mind. One needs to realize how different modern civilization is compared to the other time periods we are comparing it to. Along with the civilization needing to reach certain levels of scholarship in mathematics, statistics, and linguistics, they also needed to have to time, energy, and willpower to sit down and try to solve ciphers. In the time of Queen Elizabeth and Queen Mary, the normal folk were illiterate and were working very hard to just have enough food to survive. If they came across a note that looked like a code, they first of all wouldn’t have enough leisure time to try to break the code if they wanted to and they also wouldn’t be literate so they wouldn’t even know where to begin.

In conclusion, the level of education in the modern era is light years ahead of what it was many hundred years ago. One example I heard in class that made it very clear to me was that a hundred years ago, a scholar that was very specialized and on a career path towards engineering or something like that would be the only person who would even considering studying physics. In today’s world, almost every high school graduate has taken a full course in physics.

Cryptanalysis in time

The level of knowledge required by cryptanalysts to cipher and understand codes has remained relatively the same throughout history. The difference is that this breadth of knowledge is easily developed and acquired in this day in age. Today’s society demands a higher tier of critical thinking in everyday activities due to the advancements in technology and progression of arithmetic and language. An eight year old kid nowadays has the knowledge that a seasoned cryptanalyst had at his disposal long ago. Much like the standard of life, the standard of knowledge has risen exponentially. People involuntarily and unknowingly find themselves in the midst of ciphers and codes every day, therefore peoples’ minds have become conditioned habitually to decode and decipher messages in puzzles and word games.

The knowledge and use of frequency analysis in the alphabet is a more modern practice in cryptanalysis and developed because people are exposed to puzzles, riddles, and word games at an earlier developmental stage. In the end, humans are more frequently immersed in codes and ciphers because of the advancements in thinking and technology. This allows a higher level of cryptanalysis to be performed by ordinary people only due to the fact that they have been subliminally exposed to it their entire lives. People now just have a larger pool of knowledge and experience to pull from, giving them the upper hand in cryptanalysis.

Page 1 of 2

Powered by WordPress & Theme by Anders Norén