Cryptography

The History and Mathematics of Codes and Code Breaking

Tag: frequency analysis Page 1 of 2

How All Things Advance and Progress

While it can be determined that the evolution of cryptography and cryptanalysis is a result of a high level of academic and scholarly progression, the ability to analyze codes and ciphers does not necessarily have to come from that level of scholarship. In fact, amateur cryptanalysts are fully capable of analyzing and deciphering codes without much experience or training. It is fully possible that any normal person is able to solve these simple codes through trial and error.

In order to explain why the development of frequency analysis was so complicated and challenging and why in contrast, solving these substitution ciphers are seemingly so easy and simple, I want to use a simple example, or in fact any invention that we use practically today. One of the earliest and most profound inventions in human history is the wheel. The wheel took about 300 years to develop, yet today it is one of the simplest mechanisms that all of humanity understands. Similarly, Thomas Edison's invention of the lightbulb took one thousand attempts to master, yet today it is manufactured at an insane rate along with thousands of other inventions that probably took weeks, months, and even years to develop. In just the same manner, the people that developed frequency analysis may have spent hours and days trying to come up with the essence of frequency analysis, yet those that practice today have it down with ease due to the length of time for which it has existed.

Throughout time since the creation of frequency analysis, the human race has advanced in knowledge and logic and has since made the profound the simple. For this reason, even the most basic and amateur cryptanalyst has the ability to effectively decipher a substitution cipher "on their own".

The Achilles Heel of Mary Queen of Scots: A Weak Cipher

Arnie: Why is a weak cipher worse than no cipher at all?

The cipher that Mary Queen of Scots used in this chapter was able to be broken, and in this case, having no cipher at all would have been better than the weak one that was used. He says that because they believed their communication was secure, the Queen and her accomplices became too complacent. The contents of their letters were far more incriminating because the conspirators believed that even if the letters were found, they would most likely just look like gibberish. With frequency analysis, even a somewhat strong cipher can be cracked over time if someone has the right resources, which the Queen of England most definitely did. If Mary had just used cryptic language that was vague and concealed the letters in the same manner, even if they were found, they would have been much less incriminating and she would most likely not have been sentenced with the death penalty. Because of her complacency and her blind trust in the cipher she was using, she let down her guard, and this ultimately led to her demise. This is what Singh means by the fact that sometimes a bad cipher is worse than no cipher at all.

I think that the same thing could be said about passwords on the internet today. If something has a weak password it may be worse than having no password at all. If there is a hacker trying to get your data, they are probably more likely to try and hack into password protected websites, because that is where more sensitive information is normally stored. If your password is "12345678," it may be worse than having the same information on a non password protected website because hackers may be less likely to look there. I think even in the modern era, the idea that no cipher is better than a weak one is still applicable in some senses.

You Don’t Need to Be Trained to Be Professional

With sophisticated and detailed research on statistics, al-Kindī invented his system of cryptanalysis, later known as the frequency analysis. It’s not surprised that he was considered as the greatest scientist in the ninth century when many disciplines including mathematics, statistics and linguistics that are well developed today were still in their rudimentary stages; thus his research was undoubtedly remarkable. More importantly, the public, especially amateur cryptanalysts were not educated as well as those in today’s education system; few people were likely to have the opportunity to master, or even learn mathematics. Therefore those attempting to decipher encrypted messages had to depend merely on al-Kindī’s approach. However, things have changed.

Today’s schooling system guarantees the educational opportunity for almost everyone to learn basic mathematics and statistics, and to have personal perspectives on languages. As long as we comprehend the linguistic rules, do the math and then go through the process of trial and error, all of us can crack codes in our own ways.

The easy access to various resources and information should not be understated as well. Simply searching ‘cryptography’ on the Internet, even high school students are probably capable of decrypting substitution cipher. For those who want to dedicate more time and truly dive into code breaking, thousands of books will be available if they are willing to reach them. Learning new theories and grasping the nature of any discipline without instructions are no longer the missions impossible; in other words, self-study is relatively more feasible than ever before.

Generally, the simple truth is, we are getting more knowledgeable and everything is getting more accessible. Though code breaking is hard, the actual barrier for amateur cryptanalyst today may be their willingness to reach the resources and their persistence as well as patience when deciphering codes with complex encryption.

Modern Cryptanalysis Geniuses?

In The Code Book, by Simon Singh, the discovery of cryptanalysis is discussed. It is explained that without a strong background in core disciplines, cryptanalysis is impossible to achieve. Mathematics, statistics, and linguistics are vital in the development of many methods, such as frequency analysis. Earlier civilizations lacked a certain amount of efficiency in these fields, and that is why cryptanalysis was not discovered until around A.D. 750. The discovery was made in the Islamic civilization during a time when the arts and sciences began to explode with breakthroughs. It took years to become masters of cryptoanalysis, like they were.

It is not uncommon in modern society that an amateur cryptanalyst is able to crack a simple substitution cipher using the same method of frequency analysis, without ever being formally taught it. Are all modern cryptanalysts just naturally born geniuses in the art of code breaking? No, that's not likely. Instead, our society has just developed significantly since the year A.D. 750. People are given a better foundation when growing up in the arts and sciences. Mathematics, statistics, and linguistics are taught to all people, in varying degrees of course, but these fields are still emphasized. Civilization has advanced so far that many people have a basic understanding of subjects, that very few people knew about in ancient times. This is the main reason why amateur cryptanalysts can decipher substitution ciphers with out extensive training. To people in today's day and age it is just a "logical" way to attack enciphered messages, opposed to the people in ancient times who worked tirelessly to unlock the secrets of mathematics.

Easier to Learn, or Easier to Access?

I believe that, while a high level of scholarship was required to develop the frequency analysis approach, it is not critical to the use of this approach. When the world was new to this subject--when it had just discovered ciphers and keys and cryptanalysis--all of the knowledge was completely new. It was the cutting edge, so not many people understood it yet. It was essential to attain a high level of education to comprehend the mysteries of cryptology. However, with the modern education system, and modern technology, people have the information necessary more readily available. People can access the "mathematics, statistics, and linguistics" necessary to equip themselves for code making and codebreaking. Also, the easy access means that the information surrounds the human population. We have billions of pieces of data sitting at our fingertips, just waiting in that ever-present "cloud." Because of this access, and as a result of the heightened academic expectations, "amateur" cryptanalysts can use previously lengthy and difficult methods of analysis with much more ease. The civilization has reached a "sufficiently sophisticated level of scholarship in several disciplines," and therefore the people of that civilization may achieve the same accomplishments which the Islamic civilization discovered. However, as a result of the constant inundation of information prevalent in our society, and the resultant size of the body of common knowledge, amateur cryptanalysts can now use approaches such as frequency analysis, which was so arduously sought out, without any formal training.

A Developing Familiarity Throughout History

Fundamental development in the disciplinary topics of mathematics, statistics, and linguistics was procured from a comparative ground-level hundreds of years ago as opposed to what we have unearthed today. The advantages and resources currently available to the vast public are, of course, the most they have ever been in history. This goes without saying. "Discovering" tactics to break codes and ciphers that were once considered the most advanced techniques by exceptional cryptanalysts is certainly not as easy a task without the long history of code breaking (in the colloquial meaning of the term) that had come before us.

The ability to learn methods such as frequency analysis from a quick Google search is much less arduous a task than inventing them without any previous notion of such a possibility. Even assuming that today's amateur cryptanalysts aren't explicitly searching "how to's" from public databases, the idea of frequency analysis and any analogous general form of use is very comfortable and familiar. Perhaps teachers from grade school distributed puzzles aimed to unscramble words and phrases or your classmate used a simple cipher as a way to ask out their prom date. Experiencing or seeing a number of similar events throughout our lives inevitably ingrains the technique somewhere in the back of our minds, at least implicitly.

Noting the above, it is truly incredible to acknowledge how commonplace once incredible and cutting-edge discoveries are considered in the present day. This will always be observed, even beyond subjects regarding cryptography, as a natural progression of time.

Cryptanalysis: From Complexity to Common Knowledge

Each an art form of its own, cryptanalysis and cryptography demonstrate opposing counterparts focused on accomplishing the same common goal—the understanding of a hidden message. These two techniques highlight the competitive battle between codemakers and codebreakers. Although cryptography requires a distinct level of skill and secrecy, the practice of cryptanalysis encounters even greater obstacles as the codebreaker must determine the meaning of the hidden message as well as the technique necessary to break it. Arguably, the mastery of one skill can lead to an expertise in the other as the making of a complex cipher derives the further logic and creativity necessary to uncover these intricate codes.

Singh cites the frequency analysis technique as an “innocuous observation” by Muslim cryptanalysts that became “the first great breakthrough in cryptanalysis” (17). Nowadays, this code-breaking method is quickly and easily used by first-time cryptanalysts with no previous instruction, almost as if by second nature. While the frequency analysis technique was undoubtedly a major breakthrough in the seventh century, the vast amount of education and technology provided to our society today allows this method to become an obvious first step towards discovering the unknown.

As time has gone on and technology has expanded, the human mind has reached a common intelligence almost unimaginable even one hundred short years ago. Education has taught us to not only focus on how to put things together, but also on how to take them apart. Practices such as cryptanalysis have become more applicable to the average man as common knowledge typically requires an understanding of both how and why things work. The fifteenth century Western world is a prime example of the human tendency to discover how something functions as immediately after cryptography was introduced “already there were individuals attempting to destroy this security” (27).

While in the past cryptanalysis was labeled as an expertise only accessible to those in higher society with the finest education, its ability to be understood by even the most amateur cryptanalyst emphasizes the incredible expansion of knowledge in our society today.

The Infinite Struggle Between Makers and Breakers

The continuous tug-of-war between cryptographers and cryptanalysts has triggered the expansion and prominence of cryptography in the world today. Just as quickly as new encryption methods are developed, new decryption strategies are implemented, creating a state of equilibrium in cryptography.

The art of secret writing dates back to Herodotus (Singh 3) and has had various uses in history, including military communication and murder plots. In more recent years, the explosion of technology has propelled the expansion and necessity of cryptography to the point where it has an effect, either direct or indirect, on every individual. Cryptography has a range of important functions, such as keeping bank account information private and ensuring that government information is kept secret.

Cryptanalysis was not invented until the Islamic golden age under the Abbasid caliphate (Singh 14). According to Singh, cryptanalysis could not be invented until a civilization had reached a sufficiently sophisticated level of scholarship in several disciplines, including mathematics, statistics, and linguistics (15). The Muslim civilization provided an ideal cradle for cryptanalysis because the Muslims were educated in a variety of fields and pursued knowledge in all of its forms (Singh 15).

This breadth of knowledge associated with the Muslim civilization is valued in most modern society today. Most individuals are at least somewhat proficient in an assortment of disciplines. Primary education sets the field by educating children about a variety of different subjects in order to equip them with a large intellectual toolset. Many secondary education institutions stress a liberal education in which students obtain an educational background in the humanities as well as the math and science fields.

Because of this, it is no longer necessary to undergo formal education in cryptanalysis in order to use elementary strategies such as frequency analysis. With the broad education that modern citizens receive, combined with the availability of abundant resources, especially those on the Internet, many of the older, simpler codes can be decrypted by the layperson. However, modern encryptions are much more complex than those of the past and may require serious study.

During the past few centuries, cryptography has exploded due to the expansion of technology. Cryptography affects numerous fields ranging from government information to banking, and brings in knowledge from a variety of areas including mathematics, logic, linguistics, and statistics. Modern education allows us to engage in the world of cryptography even as the complexity and number of cryptographic techniques increase exponentially.

Intuition

The part of Cryptonomicon that caught my attention was Lawrence Waterhouse's attempt to solve the cipher, or "mathematical exercise" given to him and others by Commander Schoen. Schoen writes out the cipher, a list of 5 groups each with 5 numbers. The numbers are either 1 or 2 digits. Waterhouse instantly recognizes that the greatest number provided is 25, thus he assumes that the numbers must represent the letters in the alphabet. He then decides to run a frequency analysis test using the numbers on the board and realizes the number 18 occurs 6 times. Waterhouse then makes an assumption that the number 18 must be the letter E so he mentally substitutes the letter E into the cipher. Next, Waterhouse observes that the opening 4 numbers are '19 17 17 19'. He then knows that if 19 is a vowel, 17 must be a consonant or vice versa, and since 19 is twice as common in the cipher, he assumes that 19 is a vowel (specifically A). Waterhouse then proceeds by using the context of the message as provided by Commander Schoen earlier. Schoen had said that the message was intended for a naval officer. Using this, Waterhouse was able to guess that the first word was ATTACK. Immediately, he saw the rest of the cipher decrypt before his eyes so he stood up and stated his findings emphatically. The message read: "Attack Pearl Harbor December Seven"

The methods Waterhouse used to solve the cipher reminded me of the discussion we had earlier than year about the use of intuition to solve puzzles. Waterhouse used problem solving techniques and pure logic to decrypt the cipher. By assuming the first word was 'ATTACK' and recognizing that the numbers represented individual letters in the first place, Waterhouse demonstrated the problem solving techniques we have all acquired through basic learning.

The Accessibility of Knowledge

In Simon Singh’s The Code Book, he states “Cryptanalysis could not be invented until a civilization has reached a sufficiently sophisticated level of scholarship.”  If such a sophisticated level of scholarship was needed to invent cryptanalysis, and cryptanalysts were considered to be of ‘higher’ scholarship, why are amateurs capable of decoding ciphers themselves?

Is the world becoming more intelligent? That I do not know.  But it is obvious that knowledge is spreading and learning is occurring more vigorously throughout the world than in the past.  As children grow, they are expected to conquer challenges using the mind, not their fists, and grow mentally as they progress through the education system.  They are expected to continue their path of learning to higher degrees in high school and college, and expand on their critical thinking abilities.  Therefore, society's expectations have furthered amateur cryptanalysts ability to decode messages.

Frequency analysis is a very complex approach to cracking a code.  Yet our minds are capable of applying frequency analysis subconsciously when the opportunity presents itself.  For example, in the word game, Hangman, our mind subconsciously guesses letters that are frequently seen in words (vowels, specific consonants).  Ultimately, people can only apply “on their own” approaches if they have been exposed to higher levels of critical thinking.  Education and the accessibility of knowledge provide the world with these abilities to critically think and solve problems. Amateur cryptanalysts, though untrained, can accredit their code breaking skills to the education they’ve received from such an early age.

Page 1 of 2

Powered by WordPress & Theme by Anders Norén