Benjy Weinberger

10 Tech Concepts Everyone Should Know

Thu, 16 May 2013 15:37:00 -0700

If you work in the tech industry then your daily conversations are littered with tech terms. You’ll probably have at least a vague idea of what these mean, but if you’re not in a technical role it’s sometimes hard to put these concepts and buzzwords in precise context.

In this post I’ll briefly explain ten basic terms that engineers use every day. Whatever your role in the tech industry, you’ll benefit from knowing exactly what these mean.

Brevity will require me to leave many important details out. If you’d like me to elaborate further, or if there are other concepts you’d like explained, let me know! I’ll be happy to write another post in this vein in the future.

1. API

Like any other work we do, software needs to be organized. We typically organize software into distinct modules, each responsible for a different task.

These components often need to talk to each other. For example, a DISPLAY module that displays a web page can send a URL to a FETCHER module that pulls web pages from the internet and returns the contents of those pages.

An Application Programming Interface (API) is a formal specification of how one software module interacts with another. For example, the API of the FETCHER component might be something like:

content = FETCHER.fetch(URL)

Meaning, “pass a URL into my ‘fetch’ function, and I will return the content of the web page at that URL”. The DISPLAY component can use this, but so can any other module that needs to fetch web pages.

In other words, APIs offer a simple, standardized way to provide functionality, without requiring a lot of intricate coordination. I explained APIs in detail in a previous post.

2. Technology Stack

Every engineering team works on unique problems: making tweets flow, mining check-ins for recommendations, sharing photos.

However most engineering projects also have many problems in common: how to store and retrieve data efficiently, how to serve web pages over the network, how to handle user logins and so on.

It would be a huge waste of resources to have every engineering team solve those common problems over and over again, re-inventing the wheel each time. Instead, we rely on standard components to do this general-purpose work. The set of standard components we choose are our technology stack.

A common example of a technology stack is the LAMP stack: Linux for the operating system, Apache for the web server, MySQL for the database and PHP (or Python) for the server coding environment.

3. DNS

Computers love numbers, so servers on a network locate and identify each other using their IP addresses, which are dot-separated numbers, like 107.23.22.73. These are difficult for humans to work with, so servers are given human-readable domain names, like www.foursquare.com.

The Domain Name Service (DNS) is a distributed global directory that converts human-readable domain names into machine-friendly IP addresses. When you type www.foursquare.com into your browser’s address bar, the browser contacts a DNS server to ask it to translate that name into an IP address, and then sends the original request to that IP address.

4. Open Source

Software is written by developers in a high-level programming language like C++, Java or Python, and then converted into low-level machine instructions that computers understand. You can use software without access to the original source code from which it was created, so in conventional commercial software projects, the source code is proprietary and secret.

Open source projects, however, make their source code publicly available. This allows users to read the code in order to debug the software, to modify and improve it, and to re-use parts of the code for other purposes.

Open source software has the distinct advantage of being free, both “as in speech” and “as in beer”. It’s this second type of free-ness that is most appealing: why pay for proprietary software if there’s a free open-source alternative that’s good enough?

Prominent open source projects include: The Linux operating system, the MySQL database server, the Apache web server, the Hadoop data analysis system and more.

Open source projects encourage sharing and inspiration. Most open source projects encourage people to contribute their changes back to the project, for everyone else to enjoy.

5. Machine Learning

Computers are very good at doing mundane tasks, like sorting and searching, much faster than humans can. Whenever there’s a straightforward algorithm for arriving at a solution, the computer’s brute force can apply that algorithm with amazing speed.

However there are tasks that humans are much better than computers at, such as converting speech to text, or identifying visible objects. These are tasks for which there are no known straightforward algorithms.

Humans, of course, aren’t born with these abilities, but learn them early in life, doing poorly at first but then improving bit by bit. So one approach to getting computers to be able to perform these tasks is to devise algorithms that simulate learning.

These Machine learning algorithms infer general rules from a set of examples, in a manner superficially similar to human learning. They are useful for finding approximate solutions to those problems for which there are no known straightforward algorithms. Siri, for example, is the result of a machine learning algorithm that approximates human understanding of speech.

6. Version Control

In any engineering organization, multiple people collaborate on the same set of source code files. There is a constant danger of people stomping on each others’ work: If Alice and Bob work on the same file, unbeknownst to each other, then if Bob saves his work right after Alice does he’ll overwrite her changes.

Version control systems store source code and manage their versioning. They prohibit conflicting updates from multiple people. When Bob tries to save his work, the system will tell him that Alice has updated the file since he last read it, and force Bob to read Alice’s version and re-apply his changes on top of that. The system provides file-merging tools to make this re-applying of changes easy.

Version control systems also store all previous versions of each file. This allows developers to make progress while still being able to debug servers running older versions of the code. Developers can temporarily “roll back” to an older, more stable version of the code, if a new version turns out to be buggy.

Git is a commonly used version control system, and github is a SaaS version of git.

7. Algorithm

An algorithm is like a recipe: it’s a list of step-by-step instructions that can be unambiguously and mindlessly followed by a computer. Algorithms are implemented by writing out those instructions in a particular programming language. All software development is, at its core, implementation of algorithms.

Some algorithms produce an always-correct output, such as a sorted list of numbers. Others produce an estimated output that may only be approximately correct, such as a transcription of a voicemail message. Machine learning algorithms (see above) are examples of this class of approximating algorithms.

Some algorithms are deterministic: they produce the same result each time they are run on the same input. Others are probabilistic: they produce a different result each time they are run on the same input. An example of a probabilistic algorithm is one that simulates a coin tossing game.

I discuss algorithms in detail here and here.

8. Client / Server

When two computers interact over a network, the client initiates the interaction by sending a request to the server. The server performs some task and returns the results to the client.

This is easiest to understand by analogy:

Say, for example, your sister phones you and asks if you can give her a ride to the airport. You say you’ll pick her up at 6. In this scenario your sister is the client, making the request, and you are the server, responding to it.

Now say you were planning to go to a movie with a friend, so you put your sister on hold while you phone the friend and ask if he minds going to the later show. He doesn’t mind, so you return to the call with your sister and tell her you’ll pick her up at 6.

In the first conversation, your sister is the client and you are the server. In the second conversation, you are the client and your friend is the server. Note that in a single “roundtrip” you were acting as the server in one interaction and the client in another.

Web browsers and mobile devices are often referred to colloquially as “the clients”, because they are the first initiators of network requests, and so are always clients, never servers.

9. UNIX / Linux

An operating system is software that manages computer hardware and bridges between the machine and the programs that run on it. Two very familiar OSes are Microsoft Windows and Mac OS X (I discuss operating systems in detail here and here).

Long before Bill Gates and Steve Jobs came along, however, there was UNIX.

UNIX is an operating system first developed at Bell Labs in the early 1970s. It’s notable because many of its innovations strongly influenced the design of later operating systems, all the way to the present day.

Linux is an open source UNIX-like operating system that has been in continuous development since 1991. Unlike Windows or OS X, Linux is free. This means that companies can install it on all their datacenter machines without it costing any money. And since companies can have hundreds, thousands or even tens of thousands of datacenter servers, this leads to huge savings, and helps explain why Linux is so phenomenally popular.

10. Distributed Systems

Large-scale services like Google and Facebook have so much data, and serve so many requests, that no single server can possibly handle it all. A distributed system is one that uses multiple computers, connected by a network, to perform a task or provide a service.

The user of such a system has no idea of the details: how many computers are involved, how they’re connected and what each one does. The system appears to the outside like one entity, even though internally it’s composed of many parts.

Distributed systems provide two advantages: they can handle far more data and far more traffic than any single machine, and they can be far more reliable: if a single machine in a distributed system fails, other machines step in to take over the broken machine’s work. A distributed system is a reliable whole built out of unreliable parts.

Is a CS Degree Worth It?

Tue, 30 Apr 2013 11:01:00 -0700

As ReadWrite noted recently, learning how to code has never been hotter. Tech-education startups like Treehouse and Codeacademy are booming; Non-profits such as code.org are working to make programming education available in schools; Informal workshops abound, and more and more non-engineering tech industry insiders are getting in on the action.

Anyone with access to the internet now has within reach an incredible array of informal programming education resources, including many specifically targeted at women (a demographic still underrepresented in CS departments). With that, and the availability of simple web application frameworks such as Ruby on Rails, you can learn to create basic but real webapps in just a few weeks. What was once the province of the nerdy few is now available to many.

Should I go back to school?

Given all this, you might wonder if an academic Computer Science degree still has as much value as it used to.

Indeed, ReadWrite recently published this infographic under the question-begging headline “Learning to Code: Get a Degree, or Just Teach Yourself?”

However there is a rather large difference between “coding” and “software engineering” (*). To use a rough and imperfect analogy:

coding : construction work :: software engineering : architecture

After a few weeks or months of informal coding instruction you can acquire the skills to put together a small-scale, simple webapp, or to implement a small, isolated piece of logic. But you’ll have only limited ability to design a large-scale system, to analyze performance, to implement algorithms or to choose appropriate data structures. Those abilities rely on a rigorous theoretical background.

Coding vs. Computer Science

In fact, that ReadWrite headline is misleading, because you don’t get a degree in order to “learn how to code” (**). You get a degree in order to learn computer science.

Undergraduate and graduate CS students learn algorithms, data structures, operating systems, computational complexity, programming language theory, machine learning and more, not to mention the math needed to properly master these topics: advanced calculus, linear algebra, mathematical logic, set and group theory, combinatorics and more.

This rigorous theoretical background gives a graduate the knowledge and experience necessary to solve a wide range of problems, and to reason about them from first principles.

In fact, in some CS departments, coding is taught as an afterthought. For example, at my alma mater, Hebrew University in Israel, we had no required coding courses at all. We were taught C and C++ by the TAs of various classes just so we’d be able to code up our coursework.

Some people can learn CS, or at least enough of it to be very successful software engineers, informally. I know people who have done this. But such autodidacts are the exception that proves the rule.

Then why learn to code?

If a CS degree is so important to a software engineering career, we might ask the opposite question: is there any value in learning how to code informally?

Absolutely there is!

First of all, if you work in tech but are not an engineer, a basic grasp of coding will improve your understanding of your own industry. VCs, tech journalists and bloggers, in particular, can benefit greatly from leveraging their existing affinity for tech into learning how to code. A little programming experience helps you understand which technology problems are hard and which are easy, which is an important distinction to make when investing in, or reporting on, technology.

But coding skills will also benefit people far outside the tech industry.

In the past two decades computers became ubiquitous in homes and workplaces. In the past five years they’ve become commonplace in our pockets, in the form of smartphones. In the next decade they’ll enmesh even deeper: in our cars, our household appliances, our eyewear…

Increasingly, understanding computers will become important to understanding society. Computer literacy will be the skill underlying all others in the next few decades, and learning how to code is by far the best way to acquire this skill.

Going inside the black box

To use an analogy: you can drive a car reasonably well without knowing how it works. You press the gas pedal, it accelerates; You press the brake it stops. But if you want to be a really good driver, able to drive fast yet safely in non-ideal conditions, it’s vital to know a little bit about how cars work, in order to understand, say, the relationship between engine torque and acceleration, or road conditions and braking.

Similarly, if you don’t know how to code, then you’re mostly treating a computer as a black box: if I do X then I get result Y. But as we all know, sometimes you get Z instead, and boy are those times frustrating. If you know how to code you can reason about why the computer behaves the way it does. You’ll get a hell of a lot more out of using a computer if you understand it at that level.

Very few people earn a Mathematics degree, but we still expect everyone to be able to do basic mental arithmetic. Similarly, not everyone can or should get a CS degree, but everyone can and should be able to do a little bit of coding.

(*) Full disclosure- the author of this post has a CS degree, and is therefore predisposed to believe that it’s valuable…

(**) The headline is also misleading for another reason: the article itself talks only about web design, which is a different discipline than software development.

CS101 part 6: Operating Systems pt. 2

Mon, 22 Apr 2013 10:28:00 -0700

My previous CS101 post explained what operating systems are, and what services they provide. This post offers a quick tour through some basic operating systems concepts, and explains in more detail how the OS provides certain services.

I’ll describe in turn how the OS manages each of the four basic resources: CPU, disk, memory and network. You’ll probably have heard of some of the concepts before, but not known exactly what they referred to. So now you’ll know!

CPU

A running instance of a program is called a process. A computer runs many processes at the same time.

Modern CPUs have multiple cores. Think of a core as a mini-processor within the main processing unit. If a CPU has 8 cores, it can perform 8 computations simultaneously.

Programs take advantage of cores by having multiple threads of execution. A thread is a sequence of computation in a process that can run concurrently with other threads in the same process or in other processes. The various processes running on a machine ask the OS to schedule their threads for execution, and the OS allocates processing time fairly to all the threads in all the processes.

Typically there might be hundreds of threads vying for 8 cores. The OS controls the time-sharing, so that each thread runs for a while, then stops, freeing up a core so that another thread can run. This all happens so rapidly that programs appear to execute smoothly, even though in reality they, or rather their threads, are constantly stopping and starting.

Disk

The OS organizes data on disk into files. A file is a logical piece of data: your tax return in PDF format, or “She Bangs the Drums” by The Stone Roses in MP3 format (yes, I was at Coachella last weekend…)

Files are organized into directories, and directories are ordered hierarchically in a filesystem. This much is familiar to anyone who’s used a computer.

However the file/directory structure is just a convenience, and has little to do with how the data is actually stored on the physical disk. The filesystem carves up each file into fixed-size blocks, scatters those blocks all over the physical disk, and remembers where they all are and how to reassemble them in order. There is no meaning to “files” or “directories” at the physical level - just data blocks.

Thus, programs (and humans) can access disk using the convenient and simple “file/directory” idiom, while the OS, via the filesystem, provides the low-level disk management.

The OS also provides page caching - commonly or recently used data blocks are kept in memory, as well as on disk, so that they can be read much faster than if they had to be fetched from physical disk. This greatly improves the performance of the filesystem in many cases.

Note that “disk” here means not just rotating disks but also solid-state drives, which are increasingly replacing those old-fashioned spinning magnetic platters.

Memory

Memory is written to and read from via addresses. Addresses are numbers that point to memory locations: location 0, 1, 2 … etc. Each addressable location stores a single byte of data, so a machine with 4GB of RAM has around 4 billion addresses.

That may sound like a lot, but it isn’t. RAM is often the most scarce resource on a computer, and many processes must share it. The OS mediates the memory needs of different processes using the virtual memory system.

Virtual memory allows processes to pretend that they each have their own private memory address space, in other words, each process has its own location 0, 1, 2 … etc. The address space of each process is divided into pages and the OS assigns these virtual pages to address pages in real physical memory. The OS then handles the translation between the virtual and physical addresses.

For example, for process A, virtual locations 0-999 might be stored in physical locations 8000-8999, while virtual locations 1000-1999 might be stored in physical locations 5000-5999. At the same time, physical locations 6000-6999 might store locations 0-999 of process B.

Virtual memory has several benefits:

Multiple processes can share physical memory without stepping on each others’ toes.
It’s much simpler to write programs with the assumption of exclusive access to a contiguous address space.
If demand for memory exceeds supply, the OS can swap less frequently used data out to disk, freeing up physical memory for a process that needs it. When the original process whose data was swapped out needs it again, the OS reads it back in from disk, possibly to an entirely different physical memory page.

Swapping is a neat trick, but disk access is slow, so excessive use of swapping can slow the computer down tremendously. Swapping is typically more useful on desktop computers than on servers, where predictable performance is often crucial.

Network

Many processes need to communicate with other computers, either locally over a local network in an office or datacenter, or globally over the Internet. However a computer only has a fixed amount of network bandwidth. As usual, the OS mediates fair access to this shared resource.

A process sends a stream of data to the OS, instructing it to send it to some remote computer. In the almost universally used Internet Protocol (IP), the OS breaks the data into packets, and sends those over the network, to a remote machine identified by an IP address, such as 143.43.1.22.

The OS at the far end receives those packets and reassembles the original data. The commonly used Transmission Control Protocol (TCP) is a scheme for managing the flow of packets from a single sending process to a single receiving process. It numbers packets so they can be assembled in order on the receiving end, and provides a mechanism for confirming received packets or re-sending missing ones.

The sending OS transmits packets sent by multiple processes on its machine, to multiple destinations. The receiving OS handles packets sent from multiple sources, to multiple processes on its machine. But a machine typically only has a single IP address. So how does a receiving OS know which process is supposed to receive each incoming packet?

The answer is that each packet is sent to a specific port on the remote machine. A port is just a number that identifies the ultimate address within the machine. It’s like an apartment number within a large condo building that has a single street address.

So for example, when a sender sends data to 143.43.1.22:12954, it means “send this packet to the process on machine 143.43.1.22 that is listening on port 12954”. Processes can listen on any port, but there are various conventions. For example, web servers typically listen on port 80.

In a future post we’ll discuss networking in more detail, and describe the building blocks that make up the modern Internet.

CS101 part 5: Operating Systems (pt. 1)

Tue, 09 Apr 2013 17:19:00 -0700

It’s been a while, but CS101 now resumes…

In a previous post I mentioned that modern computer systems consist of layer upon layer of increasingly complex building blocks. In this post I’ll talk briefly about the most basic of these building blocks: the operating system.

An operating system (OS) is a piece of software that provides a set of common services to all the other software running on a computer. These services primarily involve managing shared resources, notably CPU, disk, memory and network.

Two classes of OS dominate the desktop world: Microsoft Windows, and UNIX-like OSes. UNIX was an OS originally developed over 40 years ago at Bell Labs. It inspired a host of descendants, and its design lives on to this day, including in popular OSes such as Linux, FreeBSD and Mac OS X.

Why are OS services important? For two main reasons: interface and coordination.

System Interface

Computers and their components are complicated beasts. At the lowest level, directly controlling hardware is a very intricate business.

For example, to write some data to disk you have to find some free space on the disk. But you’re unlikely to find enough consecutive free space for all your data, so you have to break the data up into chunks, and store each chunk at a different disk location. Then you have to remember where all those chunks are, so you can restore the original data later by reading the chunks from the right locations in the right order.

The author of a program doesn’t want to deal with this level of detail directly. How tedious that would be! Instead, the OS provides a simplified interface to the disk. A program says to the OS “save this data to disk for me”, and the OS does the gruntwork of chunking up the data and placing it on the physical disk.

The OS provides similar simplified interfaces to memory, CPU, network and other resources. It allows programs to worry only about their specific logic (e.g., a browser can worry about how to display a web page) and not about the mundane but complicated tasks that all programs have in common.

Coordination

On a modern computer, many programs run at the same time. For example, right now, on my macbook, I’m running Chrome, iTunes, Finder, Dropbox and Preview. All these programs need CPU resources in order to do computation, disk resources in order to load and save data, and memory resources as a workspace. Some also need network resources in order to talk to the outside world.

Imagine if the author of each of these programs had to know about all the other programs, and had to coordinate with them directly in order not to overwrite another program’s data or read another program’s network messages? It would be like driving on a crowded road with no traffic rules, so that to safely change lanes, pass or enter an intersection you’d have to coordinate directly with every other driver on the road.

Just as we have traffic rules, signs, signals and cops, a common layer that allow cars to flow without every driver talking to all other drivers, computers have the OS, a common layer that carves up resources fairly between programs and takes care of coordinating access to those resources, so that programs don’t step on each others’ toes.

OS Layers

An OS typically consists of a kernel and a set of applications. The kernel is the heart of the OS. It provides the basic resource and hardware management facilities we’re discussing. The applications are standardized tools to allow humans to interact with the OS conveniently.

An example of such an application is a shell. A shell is a prompt where humans can type commands that interact directly with the OS. For example, if you’re on a Mac, go to Applications > Utilities > Terminal.app and double-click on it. That terminal window is running a shell.

Type pwd (short for ‘print working directory’) and hit enter. The shell will display the directory it’s currently ‘in’. Now type ls (ell-ess) and hit enter. You’ll get a list of all the contents of that directory.

pwd and ls are applications, as is the shell you ran them in. They access information about the filesytem by making calls into the kernel.

Different OSes strike different balances between the kernel and the applications. Some do almost all work in the kernel, while others do as little as possible in the kernel and defer other work to higher layers.

In my next post we’ll dive into how OSes manage the four main resources in a computer: CPU, RAM, disk and network.

April Foolishness

Mon, 01 Apr 2013 08:23:00 -0700

A few weeks ago @shanley published this post about the superficiality of what passes for “company culture” in much of Silicon Valley.

In the spirit of the April fool’s day grinch, I’m going to add another one: Culture is not about telling semi-clever lies on your corporate blog once a year.

Every year, come April 1st, dozens of tech companies, from Google down to the smallest startups, post oh-so-hilarious faux press releases about some obviously absurd product move or feature launch. You can imagine the self-satisfied giggles at the brainstorming sessions in the marketing department. The trouble is, no one else is really laughing.

April fool’s jokes were always a fairly annoying phenomenon. Very few are truly funny (this is still the gold standard, and it’s over 50 years old). A good joke has to take the reader in at first, but reveal itself on a second reading. If it doesn’t then you’re just making stuff up, and there’s nothing clever about that.

Google was an early tech-industry proponent of this tiresome tradition, with the obvious goal of demonstrating how cool and funky and non-corporate they are. And now so many companies follow suit, and Google themselves now have so many different April fool’s jokes, that it’s difficult, and ultimately pointless, to keep track of them all (happy to say that my employer doesn’t participate in this nonsense.)

Please, please, make it stop. It’s not funny, or clever, just slavish and forced. It doesn’t make you look like a cool place to work, it just makes you look like you’re trying too hard.

Men, It's On Us Now

Sat, 23 Mar 2013 09:06:00 -0700

The Adria Richards/PyCon/SendGrid affair has made me sick to my stomach. Any decent human being should be outraged and sad at how low so many people in the wider tech world can sink.

I’m not going to express an opinion on the original incident. How far out of line were those guys? Did Adria overreact? Could she have handled it better? I don’t know, I wasn’t there, and it doesn’t matter any more.

What does matter is that the only people who are happy right now are the trolls: the hateful, misogynistic sickos who’ve spent the last few days making horrific threats of physical and sexual violence against Adria.

The rights and wrongs of the case are, for now, irrelevant. The opportunity for a reasonable conversation about them has passed, and may never return. Any discussion of them against this background of hate becomes an implicit debate on whether Adria “brought this on herself”, and none of us should fall into that trap.

This is where SendGrid failed so badly: it may have been the right call to let Adria go (and I have no opinion on the matter), but doing so under threat merely capitulated to the worst elements on the internet. Whatever mistakes you think she’s made, Adria Richards belongs to the sane, productive side of tech, and we should be circling the wagons around her, not cutting her loose.

Women in the tech industry are getting the message loud and clear: if you step out of line (or even if you do nothing at all) the trollosphere will destroy you at will, and your company will abandon you to your fate. As John Koetsier of VentureBeat put it:

What do you do now if you are a woman in technology and you feel harassed or abused and want to shine a light on it, but now see this prominent woman totally abandoned by her company?

I’ll tell you what you do, unless you’re a saint or a hero. You shut up. You put your head down. You grin and bear it, because it’s a man’s world. And you leave, eventually, for a better place.

So now it’s up to us, the sane male population of the tech world. If we want a diverse, welcoming industry that benefits from the talents and perspectives of women, we’ll have to be the ones to make it safe for them to participate. We must push back against exclusionary language, sexist banter, brogramming and all the other issues, large and small, that determine whether women are welcome and respected at our companies, conferences, open-source projects and online communities.

We can’t expect women to go through this alone. And if we don’t fight to keep and increase female participation in all parts of the tech industry, and women abandon it, we’ll all be poorer for it.

CS vs. SWE

Mon, 04 Mar 2013 14:03:00 -0800

Someone asked me at a bar gathering yesterday: “Is computer science really a hard science? Isn’t it more like engineering?” I had at that point had one too many drinks to give a coherent answer. Plus, nothing kills small talk faster than bringing mathematics into the conversation…

But it is actually a good question. What is the distinction between computer science and software engineering?

A Foot in Each Camp

As you might have guessed, there’s no clear boundary between the two concepts. University computer science departments, for example, deal with both theory and with practical application. Within computer science, each subfield relies on varying amounts of each.

For example, the field of computer graphics relies on advances in both the mathematical theory of computational geometry, and in the practical development of hardware and software for very fast image processing and rendering.

As another example, the field of programming language research deals with both the formal mathematical properties of programming languages and with the pros and cons of their use by human programmers.

Computer Science Without Computers

At the extreme we have theoretical computer science. This subfield consists of those aspects of computing that would be just as interesting and valid as a branch of mathematics even if computers didn’t exist.

This isn’t as silly as it sounds: Theoretical CS has its foundations in the work of John von Neumann, Alan Turing, Alonzo Church, Claude Shannon and others in the 1930s and 1940s. Their work predates modern computers, and indeed they were all considered mathematicians at the time, the term “computer science” not having been coined yet.

Theoretical computer science deals with the basic properties of information and computation. It answers questions such as “are there uncomputable mathematical statements?”, “how efficiently can various computations be performed?”, “how much information can be represented by a certain number of symbols?”

The answers to these questions don’t depend on a particular computer design. In fact, like the rest of mathematics, they are valid even in a universe without actual computers. Or humans, for that matter.

So why is this considered a computer science discipline? Because real-world computers, as we know them, could not exist without it. Theoretical CS is the foundation of all other branches of computer science, including the most applied ones. It doesn’t require them, but they sure as hell require it.

A Mathematics Machine

And this, in the end, is why computers are so fascinating to me and to many other math nerds. Beyond its use as a tool for creating and consuming information, a computer is a real-world manifestation of a complex mathematical concept.

Granted, that’s hardly a unique property. A hula hoop is a real-world manifestation of the geometric concept of a circle. But the theory of computation is complex and interacts with many other branches of mathematics, one way or another. On some level a computer is math, is logic, rendered in silicon and metal. And that, I think, is why I have been captivated by computers for as long as I can remember.

Respecting Failure

Mon, 18 Feb 2013 07:54:00 -0800

Alexia Tsotsis just published an important post on TechCrunch about an insidious side of startup culture, one she refers to as “the cult of success”.

In the startup world we pay lip service to risk all the time. No concept is more hallowed or hyped in Silicon Valley than “entrepreneurship”, and the defining feature of entrepreneurship is risk.

You Can’t Spell Risk Without Failure

Risk, by definition, implies frequent failure. Yet while we recognize the value of failure in principle, when presented with an individual instance of it, one involving a specific startup and actual people, we too often respond with snark and schadenfreude. TechCrunch, and the rest of the tech press, are not immune from this, as Alexia admits. Too often, how the ups and downs of a startup get covered depends more on how close the founders and investors are to the “cool kids” rather than on the merits or the long view.

What’s an appropriate response to failure, then? Should we celebrate it, as we do success?

I believe not. If the word “failure” is to mean anything, then it’s not something to be celebrated. We can, however, respect failure.

Entrepreneurship and risk aren’t to be lauded for their own sake, either. Anyone can take risks: to paraphrase Dawkins, however many ways there may be of being successful, it is certain that there are vastly more ways of being a failure. Silicon Valley is about taking smart risks.

Smart Risks and Virtuous Entrepreneurship

How should we define a smart risk? Clearly we can’t do so retroactively, by whether a venture succeeded. That just brings us right back to where we started.

We can, however, evaluate a startup using the old-fashioned virtues that have underpinned enterprise since long before Silicon Valley.

Hard work: are the founders and employees dedicated?
Execution: is the startup consistently and rapidly moving towards its goal?
Professionalism: Are the founders and employees good at what they do?
Respect: Do the founders treat their investors and employees as true partners, and do they deserve the trust of their users?
Focus: Is the startup keeping its eye on the ball at all times?
Inspiration: are the founders, investors, employees and users excited?
Vision: Does the startup have a north star goal, no matter how unlikely, for two, five or ten years out?
Noble goals: is the startup trying to make some corner of our world even a little bit better? Or are they pandering to baser human qualities just to make money?

If a startup scores well on these values, then it deserves our respect and admiration whether it succeeds or fails. And by the same token, if a startup doesn’t live up to enough of these values then we should be leery of it even if it happens to “succeed” by some limited definition of success.

Respecting Failure Without Celebrating It

If we focus too greatly on outcomes, then lauding entrepreneurship becomes nothing more than empty lip service. But if we’re still to tell the wheat from the chaff we can’t glamorize failure for its own sake either.

Entrepreneurs, who’ve been so driven towards success, aren’t likely to take much solace in being told that failure is fine. Although the intention of that sentiment may be kind, it actually undermines everything a founder works for.

Instead, if we focus on virtue, we have a solid footing on which to respect founders, employees and investors, while they’re in mid-venture, regardless of the outcome.

If a venture ultimately fails, this respect gives the entrepreneurs something to be proud of and build on for the future. And the knowledge that they’re respected for what they’re doing now, and not for some murky future outcome, may lift some of the enormous mental and emotional pressure of entrepreneurship. Because, as Alexia puts it, “killing it” isn’t worth it.

Intelligent Geometry

Thu, 17 Jan 2013 17:37:00 -0800

Religious dogma sometimes leads to absurd anti-scientific stances: intelligent design, young earth creationism, faith healing, and ridiculous conclusions about reproductive health.

If you want to revert to the 19th century, you can leave it at that. But a religious political party in Israel are now aiming much further back: United Torah Judaism, a fundamentalist ultra-orthodox party, are running this ad in the campaign for next week’s knesset (parliament) election:

For those who don’t read Hebrew, here’s a translation:

EUCLID

No, it’s not the name of a medicine, it’s a Greek mathematician your son will be forced to study, instead of the mishna.

What’s really important when we send them to the cheder? Values, good conduct, fear of god, studying the torah day and night, from the earliest age. A government without Judaism will force your son to learn non-religious studies. You must protect the next generation. We all must. We are all haredim.

I know a lot of kids who’d be happy to take up religion if it would get them out of math class. But rolling the science clock back to 300 BC? That’s a bold move.

And did you notice how the ad only refers to ‘sons’? Either they’re not concerned about girls poisoning their minds with such dangerous ideas as the parallel postulate and the pythagorean theorem, or they consider it a given that girls shouldn’t be properly educated…

CS101 part 4: Software

Tue, 15 Jan 2013 13:52:00 -0800

You suffered admirably through my necessary but dense preliminary discussions of boolean logic, binary arithmetic and memory hierarchy. Now comes the payoff - a series of posts about things you’ve actually heard of. First up: software.

I’m sure you have at least a rough idea of what hardware and software are. In fact, if you’re reading this, you probably know a lot of people who write software for a living. But you may be wondering what it means to “write software” or “run a program”. Or you may still marvel at how it is that we can make a pile of electronic circuits into some magical device that can show us pictures of kittens on skateboards. Read on to find out more!

Instruction Sets

As you no doubt recall from a previous post, computers implement very basic logic and arithmetic using electronic circuits that operate on numbers represented as sequences of ‘bits’ - 0s and 1s. These circuits are known as hardware, to emphasize that they are a physical thing, built out of tiny pathways etched in silicon, that behave in certain useful ways when you run electrical current through them.

An instruction set consists of all the basic logic and arithmetic operations that are available in a particular processor’s circuits. Examples of such basic instructions are: “add the number in memory location A to the number in memory location B, storing the result in B” or “if the number in memory location A is zero, perform the next instruction, otherwise jump to this other instruction”.

These basic instructions are the smallest building blocks of all computer programs, just as written characters are the smallest building blocks of all books. And different types of processor (e.g., an Intel Core i5 vs. a Qualcomm Snapdragon S4) have different instruction sets, just as different cultures have different writing systems.

CISC vs. RISC

The two most common instruction sets in use today are that of the x86 family of processors, used primarily in desktop and laptop computers, and the ARM family of processors, used primarily in mobile devices.

x86 is an example of a CISC (Complex Instruction Set Computing) instruction set: it contains over a thousand different instructions. ARM is an example of a RISC (Reduced Instruction Set Computing) instruction set: it contains only a few dozen different instructions.

Note that this doesn’t make ARM a weaker or less capable processor. It’s just a different hardware design, analogous to the fact that English is written with 26 characters while Chinese is written with thousands of them, but both are equally expressive.

Hardware and Software

A computer program is a sequence of these basic instructions. It tells the computer how to combine these elementary operations in order to perform complex computation. Instructions combine into computer programs just as letters of the alphabet combine to make books.

Running a program is when the computer takes each instruction in a program in turn and activates the hardware circuit it corresponds to. If the program is constructed properly, the result of all those hardware circuits firing will be something useful. Or kittens on skateboards, who knows…

The term software is just a synonym for “computer program”, implying that it’s something logical, as distinct from the physical hardware.

War and PC

The analogy to books is instructive: a printing press knows how to typeset individual letters. It is, quite literally, hardware. In this analogy, War and Peace is software. The bridge between the two is the English alphabet: it exists in both worlds, as a physical thing that can be typeset and as a logical thing that can be used to construct words, sentences, paragraphs, chapters, and eventually whole works of literature.

Similarly, the instruction set is the bridge between hardware and software. Each instruction lives in both worlds: as a building block of the software, and as a representative of a physical circuit. The software is a recipe for how to combine hardware operations, and that recipe is written in their common language - the instruction set.

One important difference from literature, though, is that while novels are linear, intended to be read in a straight line from start to finish, programs are not. Some instructions cause execution to “jump” to a different part of the program, depending on various conditions, rather like those choose-your-own-adventure books from childhood. This has the interesting and useful effect that a running program doesn’t have to end: it can “loop” over the same instructions over and over again. For example, a web server program loops over and over again on instructions that take an incoming browser request for a URL and return the contents of the requested page.

Grains of Sand

Useful computer programs typically consist of hundreds of millions of instructions. Computers can execute these instructions very, very quickly. The processor clock rate (those ‘gigahertz’ numbers you may be familiar with) is the rate at which the processor can execute instructions. For example, a 1.6 GHz processor can execute 1.6 billion (that’s billion, with a ‘b’) instructions per second. As you can imagine, this degree of complexity is intractable to humans.

Composing programs directly from the instruction set is like building a skyscraper out of individual grains of sand. It’s completely impractical to write any but the most trivial programs this way.

Instead, modern computer systems consist of layer upon layer of increasingly complex building blocks, each constructed from simpler units, all the way down to individual instructions. Software engineers write programs using these building blocks, but in the end, everything converts back down to millions and millions of instructions, each causing a circuit to fire for a tiny billionth of a second.

In my next few CS 101 posts, we’ll discuss some of these building blocks: firmware, drivers, operating systems, programming languages and more.

Depression and the Highly Logical Mind

Mon, 14 Jan 2013 11:39:00 -0800

I was saddened to learn of the tragic death, by his own hand, of Aaron Swartz. That a prominent member of my community, our community, the tech community, is gone forever is sad. That he was lost so young is tragic. That he took his own life is a horror.

Aaron Swartz wrote openly about having depression (I don’t like to use “depressed” as an adjective; It’s a disease that you have, not a thing that you are.) And many of the outpourings of grief and support after his death recognized in it a shared experience. The relatability of Aaron’s struggle made the tech world feel even more like a community, as it did after Ilya Zhitomirskiy’s tragic suicide in 2011. For, while depression is dreadful for anyone, it may have a uniquely pernicious effect on highly logical minds.

Math and Computer Science require the kind of abstract, logical, inner-focused thinking associated with introversion (in the Myers-Briggs sense, not the social sense; Many MBPT introverts are social extroverts). And there appears to be a correlation between introversion and depression. Perhaps it’s due to a physiological connection between scientific tendencies and vulnerable brain chemistry. But it may also be because depression has one symptom, unique among all illnesses, that tugs at the seams of logic: it makes you believe, contrary to all available data, that you can never recover from it.

Depression is a mind-parasite, burrowing into your consciousness. Like any good parasite, it must protect its nest. This trick that it plays on you, convincing you that you can never be rid of it, is the uniquely perverse way it keeps itself alive. Nerdy readers may appreciate the relevance of this quote:

“Only the enemy shows you where you are weak. Only the enemy tells you where he is strong.”

Someone whose toolkit for dealing with the world consists of logic and reason, ideals and abstractions, may have particularly weak defenses against this trickster disease.

You realize that it’s lying to you, that there are treatments, that that things aren’t objectively as bad as they feel. But you know, on some level deeper than logic, that there is no point, no hope and no future. And to encounter, maybe for the first time, the hard limits of rationality, to realize that there’s a part of your mind that can override the logical world view that is the core of your identity, may leave you feeling particularly helpless and hopeless.

You can’t rationalize depression away, a fact that people who’ve never suffered from it find hard to comprehend. But if someone you care about is struggling with it, and it’s likely that someone is, you can help them find a new way to access their mind.

Tell them that you care about them and appreciate them and are glad to have them in your life. Show them that you enjoy being around them and that you love them. And above all, spend time with them. Give them glimpses of an alternate future, one in which they are secure, happy and loved, tear away the lies that depression needs in order to survive, and in that sunlight it will wither.

CS101 part 3: Memory

Mon, 12 Nov 2012 14:33:00 -0800

In my first two CS101 posts (here and here) we discussed the basic electronic circuits used to compute logic conditions and basic arithmetic. At the end of my last post I alluded to a third element we need before we can construct something worth calling a ‘computer’. That element is memory.

Memory gives us the ability to have the current computation be influenced by the result of past computation. It’s what allows us to compose sequences of basic operations to produce ad-hoc, complex computations. These sequences of instructions are called programs.

Computers need memory to store two kinds of data: the programs - the sequences of instructions that define the computations - and the inputs and outputs of those computations. Both types of information are, as always with computation, reducible to long sequences of 0s and 1s. So the basic building block of computer memory is a way to store a 0 or a 1.

Durability

As you recall, electronic circuits can have two states, represented by the presence or absence of electrical current on a wire, and we can call these states “1” and “0”. A memory bit can be constructed using such a circuit, as long as it has the additional property of stability. That is, it must be able to sustain whatever state it’s in. So, if it has current (representing a 1) it has to continue to have current until we deliberately tell it otherwise, and vice versa. It can’t, for example, just let the current decay on its own, thus spontaneously turning a 1 into a 0.

There is, in fact, a way to construct these memory circuits, using the basic boolean components discussed in my first CS 101 post. Unfortunately, however, these circuits maintain their value, 0 or 1, only as long as there is current available. In other words, the memory loses all its state when you switch the power off. Outside of the most basic calculations, this causes a huge problem - how do we keep important information around for a long time?

The answer is to save the data to long-term, durable memory, so that even when the power goes off, the information isn’t lost. An example of long-term memory is a hard drive, where the information is stored on spinning platters coated with a magnetic material divided into many tiny areas, each set up so that its magnetic field points in one direction or another, representing a 0 or a 1 respectively. The material on the platter doesn’t require power to sustain its magnetic field, and so the data is preserved even without power. Then, when power returns we can copy the data from the disk back into the electronic memory, and continue with computation.

Latency and Bandwidth

You may well be thinking, “why even bother with electronic memory? why not use durable memory directly?”

The problem is that durable memory (hard drives, solid-state drives and so on) are incredibly slow in comparison with electronic memory. There are two manifestations of slowness, when it comes to memory: latency and bandwidth.

Latency is a measure of how long it takes to find the data. Or, in other words, how long it takes to retrieve the first bit we need. For example: RAM has latency on the order of 100 nanoseconds. Hard drive latency, however, is on the order of 10 milliseconds, because a hard drive has moving parts that have to be positioned mechanically. Finding a piece of data on disk is 100,000x slower than main memory! That is a huge difference.

Bandwidth is a measure of how long it takes, once we’ve found the data, to retrieve it all. Or, in other words, how many bits we can transfer into or out of memory per second. RAM has bandwidth on the order of 4 gigabytes/sec, while hard drives have bandwidth on the order of 40 megabytes/sec, which is 100x slower. Not quite as huge a difference as with latency, but still very significant in some cases.

A Snail’s Pace

The concepts of latency and bandwidth are relevant not just to memory, but to any case where we move data, the obvious example being networks. And indeed in many cases a computer on a network can use another computer as a form of memory: sending data over the network for a receiving computer to store, and then requesting the data back over the network later.

An extreme and evocative illustration of the difference between latency and bandwidth is that of the “snail pulling a chariot of DVDs” meme. That photo originated from a tongue-in-cheek experiment based on a computation similar to the following:

Say our “memory” is a box of DVDs with data burned on them, and for whatever reason, we decide that it’s a good idea to retrieve data by having a snail pull DVDs along.

An african land snail moves at a pace of about 0.5 inches per second. The snail+chariot is about 7.5 inches long, so if we send snails out single-file we can send a new one every 7.5/0.5 = 15 seconds. This means we can send out four chariots, or eight DVDs, a minute.

A common DVD has a capacity of around 4 GB, which means the snail system can deliver 8 x 4 = 32 GB / minute, which is about 500 MB/sec. This is a huge amount of bandwidth! By contrast, a fast broadband connection for home internet might be 10 MB/sec. And if we replace those DVDs with blu-ray discs, which can have a capacity of 40GB or more, we’re looking at 5 GB/sec of bandwidth, which is faster than pretty much any network out there, all on the backs of snails.

The latency of the snail system, however, is absurdly slow. If you’re just a mile away from the origin of the data, you’ll wait 36 hours for the first bit of data to arrive.

The Memory Hierarchy

Another big difference between different forms of memory is price. Hard drives are significantly cheaper per GB than RAM, even though the prices of both keep dropping dramatically (I’m not clear on what the current cost of snails is).

Therefore modern computers have an entire hierarchy of memory components, to provide various different tradeoffs between price, size, latency, bandwidth and durability. These range from tiny chunks of very fast memory embedded on the CPU itself, to RAM-based main memory, to SSDs and hard drives. Even tape drives are still used sometimes for backup. Like the snails, they have very high latency but also very high bandwidth, which is appropriate when you want to dump a large amount of data out efficiently, but don’t mind the restoration process being clunky.

Typically, the faster the memory (in latency and/or bandwidth) the less of it you have, due to cost or engineering constraints. So data needs to move up and down the hierarchy, from slow/cheap/durable memory to fast/expensive/volatile memory, as needed, and then demoted back down that hierarchy when possible to free up room in the fast memory for other data.

A holy grail of hardware is a memory subsystem with the performance of RAM, but the durability and price of disk. But until such a thing is invented, a good part of the art of building systems, such as those that power your favorite web sites, is that of managing the tradeoffs between the advantages and disadvantages of these various types of memory. Great systems are characterized by their ability to shunt data around in clever ways in order to maximize speed without compromising data safety if the power goes out in the datacenter.

Growth, Jobs and Silicon Valley

Mon, 22 Oct 2012 17:59:00 -0700

A non-technical, political post today (I did warn readers there would be some).

The overriding theme of the upcoming presidential election is jobs, with the candidates sparring aggressively over who can create more of them. Unemployment is still high across the country, and finding work, especially if you’re a new college grad, is a daunting challenge.

And yet here in Silicon Valley companies can’t hire fast enough. If you’re a new CS grad you’ll have no trouble at all getting several job offers to choose from. Your starting salary, on day one of your career, might be higher than the current salary of either of your parents. And if you’re an experienced engineer, designer or product manager then the sky is pretty much the limit.

This raises tempting questions: Can the presidential candidates learn from Silicon Valley’s success? Does the tech industry point the way towards a nationwide economic renaissance of job growth? Is what’s good for startups good for America? And if so, do the political endorsements of tech luminaries carry extra weight?

Innovation and Growth

Many people in Silicon Valley believe that the answer to all these questions is yes.

For example, several movers-and-shakers in the startup world, including Reid Hoffman, Dave Morin, Drew Houston and Craig Newmark, recently recorded a series of videos promoting innovation as the driver of economic recovery, and announcing their support for President Obama as the right leader to increase innovation and parlay it into economic growth.

As another example, Paul Graham recently posted his informal presidential election survey of startup community leaders, who apparently, by a 2-1 margin, favor President Obama over Mitt Romney, as does Graham himself (and as do I).

Graham writes:

Encouraging economic growth is a big theme, perhaps the biggest theme, in the current presidential campaign. Here in Silicon Valley, economic growth is something we understand well. Growth is the defining quality of a startup, and the leaders of the startup community are arguably the world’s leading experts in making it happen.

These claims of expertise in innovation and growth are valid. All these folks have experience with both, and their opinions unquestionably carry significant weight within the tech community. But whether that expertise carries over to the rest of the economy is a more nuanced question.

Growth vs. Jobs

Silicon Valley does indeed lead the pack in innovation and in capital growth, but the true big theme in this presidential campaign is jobs. And Silicon Valley is, by some measures, a poor job creator.

The following chart (1) shows that capital in the technology sector yields far fewer jobs per $ than capital in other sectors:

As you can see, the big tech industry darlings are near the bottom of the list, along with the oil companies. And Google’s numbers would be even lower if not for the Motorola Mobility acquisition, which added around 20000 employees for “only” $12.5B.

I didn’t include data for privately-held startups, as I don’t know them with any precision. But if you plug in whatever numbers you estimate for Twitter, Foursquare or a host of other startups, you’ll likely get similar ratios. For example, a guesstimate of $9B for Twitter’s market cap and 1400 for its number of employees gives a ratio of 155. And using 400/$3B for Square we get a ratio of 133.

By this metric, at least, Silicon Valley, while good at growing capital, is bad at growing jobs. If I had $1B to invest, and my goal was to create profit, tech would be a good place to put it. But if my goal was to create as many jobs as possible, investing in tech might give me poor returns.

Obviously this is not to say that the tech industry doesn’t provide immense value. It continues to revolutionize how we produce, consume and locate information, how we interact with people and places, how we shop and how we do business. But these transformations are due to Silicon Valley’s expertise in technology and innovation, not its expertise in capital growth. The latter may have little applicability outside the rarified air of SoMa and Menlo Park.

Inequality

You can debate whether employees-per-market-cap is a valid metric, but it’s hard to argue with the fact that, while the nation needs millions of low- and medium-skilled jobs all across the country, especially in the Midwest and the South, the tech industry produces small numbers of highly-skilled jobs, mostly concentrated in the San Francisco bay area. Apple and Google aren’t even making iPhones and Android handsets in the US, which could have created large numbers of manufacturing jobs in states that most need them.

The true economic leitmotif of the past 20 years and more hasn’t been growth, or even jobs, but inequality. Increasingly huge amounts of wealth have been concentrating in fewer and fewer hands, while everyone else, at best, struggles to stay in the same place, and at worst plunges into poverty. The tide has been rising, but instead of lifting all boats it’s been flooding the basements.

Silicon Valley is no exception when it comes to this hollowing out of the economy: the wealth created here is concentrated firmly in the hands of a lucky few (myself included). We’re rewarded handsomely for happening to have skills for which demand greatly outstrips supply. But the impact on the rest of the area economy has been mixed. For example, tech wealth has driven house prices and rents in San Francisco to such heights that many people vital to the city’s fabric, the people who make SF the world-class city it is, are being priced out. (2)

I’m not saying that Silicon Valley is evil, or that the way it allocates wealth is wrong. The tech industry has created value by providing incredible new products and services to consumers worldwide. And in any case, to steal a great phrase from Fred Wilson: you can park your outrage at the door of capitalism. But there’s a happy medium between socialist redistribution and a world of Eloi and Morlocks. If you want to build an economy in which everyone thrives, instead of just the select few with world-class educations and unrivaled access to opportunity, Silicon Valley is not an ideal model.

(1) Click here for original spreadsheet. Data retrieved from Google Finance on Oct 14th. Companies chosen from lists of largest market caps, lists of largest non-state employers and leading companies in various sectors. And Whole Foods, because I was hungry when I wrote this.

(2) See David Talbot’s recent article in San Francisco magazine for more on this, and other effects of the tech industry’s rise on the rest of the city.

Hashing Redux: Proving What You Know

Tue, 09 Oct 2012 22:53:00 -0700

Remember my post about hashing passwords? I recently encountered a tweet exchange between two of my co-workers that used hashing in a novel and nerdy way.

It started with this:

Jorge is posting a quote from Minnesota congresswoman and amateur conspiracy theorist Michelle Bachmann, helpfully providing a link to the source. But he’s also added a mysterious string of letters and numbers in parentheses. Why? Let’s find out.

Harry replies with:

Pointing out to Jorge that the source is satirical, and that he’s been duped into believing that this is a real quote. And, to be honest, given Congresswoman Bachmann’s propensity for fanciful conspiracy theories, anyone might have fallen for that. Or would they?

Jorge could have tried to save face by tweeting “of course I knew it was satire, he he…” And we would have been forgiven for thinking “yeah right… if you knew that you would have said so in the original tweet”. Jorge could have then claimed “well, but I didn’t want to ruin the tweet by giving the game away”, but that would have sounded defensive and lame.

Instead, Jorge tweeted this:

What the hell does that mean?

Saving Face via md5

If you’re on a mac right now, open up a terminal (Applications -> Utilities -> Terminal) and enter this:

md5 -s “Yes, I know this is satire.”

Hit enter. Does the result look familiar? Yep, it’s that strange string of letters and numbers that Jorge added to his original tweet. md5 is a type of hash - “a way of jumbling up text in a way that can’t be unjumbled”, as I put it in my old post.

What Jorge has cleverly done is use md5 to prove that he knew this was satire all along, without saying so explicitly and ruining the tweet. Not just claim, mind you, but prove.

The logic is as follows: “That hash I provided in my original tweet? That is the hash of the sentence ‘Yes, I know this is satire.’ And since hashes are not reversible, there was no way for me to use a random string and then later reverse-engineer it into a face-saving sentence. Therefore I must have derived that hash from that sentence when I was writing the tweet.”

Basically, Jorge did say “Yes I know this is satire” in the original tweet, but he did it in a clever way.

Not Giving the Game Away

A consequence of the fact that hashes aren’t reversible is that no one, no matter how nerdy, could have figured out what sentence Jorge used to derive the hash until he told them. So publishing that hash didn’t ruin the original tweet by giving the game away.

This tweet exchange gives a small taste of the powerful applicability of computer science to so many real-life problems. This example may seem trivial, but here’s a more significant one:

Keeping Secrets

Imagine, for example, that you’re a secret agent. A fellow agent, whom you’ve never met, approaches you and asks “hey, did you get that new top-secret document? I wanted to discuss it with you.” Sounds reasonable, right?

But wait! Maybe this person isn’t an agent after all. Maybe she’s a journalist, or worse, an enemy spy, trying to get something out of you by claiming to know more than she does. Pretending to have insider information is a classic form of social engineering.

So you say to the alleged agent, “tell you what, to prove to me that you have access to the top-secret document, send it to me.” But she, being, or at least pretending to be, a responsible agent, says “don’t be daft^*, I don’t know if you yourself are allowed to read it”. What an impasse!

Then, remembering the CS classes you took in college before you went to secret agent school, you suggest this: “OK, let’s each compute the hash of the document, write it on a note, and exchange the notes.” Now, if you both give each other the same hash you can each be certain that the other has access to the document, and you can discuss it without fear of revealing anything to an unauthorized person. And if not - time to call security: you’ve uncovered an impostor without actually revealing any information to them.

This example shows that the ability to retroactively prove that you know something, without revealing that thing up front, is incredibly powerful.

* Apparently this is a British secret agency…

Spanner and Planet-Scale Services

Tue, 25 Sep 2012 06:34:00 -0700

Google recently published a much-anticipated paper on Spanner, their globally-distributed database. Spanner is the new pinnacle of Google’s technology stack, supplanting Bigtable at the cutting-edge of scalable databases. The authors will present the paper at the OSDI (Operating Systems Design and Implementation) ‘12 conference in October.

Google’s OSDI papers tend to be seminal: the MapReduce paper from OSDI ‘04 sparked the open-source Hadoop project, which in turn powered the industry-wide “big data” trend. And the Bigtable paper from OSDI ‘06 sparked a host of similar systems (HBase, Cassandra, MongoDB and others) that power services such as Facebook, Twitter, Foursquare and many more. Will Spanner spark a trend in the same way? It’s too soon to say, but there are arguments in both directions.

Spanner offers several novel and highly valuable features. Most of these are too technical to describe in this blog (which is primarily aimed at non-engineers). But one feature worth mentioning is that Spanner manages data across multiple datacenters. It is, in other words, the first truly planet-scale system.

Previously, systems were managed at the level of, at most, a single datacenter. Some companies can run for a while out of a single datacenter, but beyond a certain size you really need multiple datacenters, at different locations around the world, for two reasons:

Resilience: If an entire datacenter is down, say because it loses power or network connectivity, you can still keep your service up from another location.
Speed: One of the major contributors to slowness of a global service is the time it takes for data to traverse all the network connections between a user and a far-away datacenter. The closer you put the servers to a user, the better their experience. So, for example, you might serve West Coast users from a datacenter in California, but Asian users from a datacenter in Japan.

Managing services across multiple datacenters has always been a technical challenge. You have to make sure that the data in all locations is consistent, and that you can switch seamlessly between them as needed. One of Spanner’s biggest features is that it takes care of all this complexity for you. It’s hard to overstate how useful this is. This sort of service would be a huge benefit to large-scale services such as Yahoo!, Facebook, Twitter, Foursquare and others.

However there are also reasons to think that Spanner may remain a rarified, Google-only technology for several years:

Complexity: Spanner is mind-bogglingly intricate. It took some of the top names in distributed systems years to develop, and the paper, though detailed, was still forced to omit many crucial details for the sake of brevity.
Technology stack: Spanner relies to a great extent on the rest of Google’s technology stack. Open-source equivalents of parts of this stack exist, but there are still gaps, and even the parts that exist are relatively immature.
Hardware: Spanner relies on two forms of time reference: GPS and atomic clocks, to synchronize operations between remote locations. This requires specialized, non-commodity hardware that is not widely available and with which the open-source community has little or no experience.

This last point is especially interesting. The uncertainty of computer clocks and the limits they place on distributed systems are well-studied but too complex to explain here. I will perhaps dedicate a future post to the issue. But suffice it to say that the requirement of specialized clock hardware in datacenters is a major barrier to open-source implementations of Spanner.

Nonetheless, I expect the paper to spark debate and inspire the open-source community to create even bigger and better systems, for the benefit of companies large and small across the industry. And that can only be a good thing.

CS101 part 2: Binary Numbers

Tue, 18 Sep 2012 09:20:00 -0700

In my debut CS101 post I discussed Boolean algebra, and how any Boolean function can be modeled by a real-life electronic circuit. In this post we’ll explore how this fact provides computers with the ability to do arithmetic.

Natural Numbers

To do arithmetic, you first need to be able to represent numbers. The most basic numbers are the natural numbers, the whole numbers we use for counting things. To write down a counting number we can simply represent it as a list of ‘things’ of that size. Say we pick the hash symbol # to represent a ‘thing’, then we can write down the counting numbers as #, ##, ###, ####, ##### and so on. Of course we’d also need some symbol, like 0, to represent “no things”.

Decimal Digits

The problem with this representation is that it’s very unwieldy. It becomes impractical to represent more than a small number of things this way. The solution is to assign shorter symbols, or ‘digits’ to the various counting numbers. So, for example, we use ‘3’ to mean ### and ‘7’ to mean #######. These digits are much shorter and more convenient. The problem now is that there are infinite counting numbers but only so many symbols we can assign, write and remember, so it’s impractical to have a unique digit symbol for every number. The solution is simple: have only a few digits, and combine them to form larger numbers. Because we have ten fingers (digitus in Latin), and counting on fingers is a ubiquitous habit, most human societies settled on ten digit symbols, that we write as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

So what happens when we want to represent more than 9 things? We count off groups of ten and use one digit to represent how many such groups we had, and then another digit to represent the remainder. This allows us to represent up to 9 tens and a remainder of 9, or 99. And if we want to represent more than 99 things? We add a third digit to count groups of ‘ten groups of ten’, which we call ‘a hundred’. And so on. This is what we refer to as ‘base ten’ representation.

Computers Don’t Have Fingers

This is all presumably intuitive and well-understood by all readers. But the crucial point to recognize is that the choice of ten digits, 0-9, is purely cultural. There is nothing fundamental or ‘mathematical’ about it. This observation is important because electronic circuits don’t have fingers. They have little wires that can either contain current or not. So electrical circuits can fairly straightforwardly represent only two states. Can we do useful math with only two states? Why yes we can!

Binary Numbers

Just as we were able to represent any natural number using the digits 0-9, so can we represent them using just the two digits 0-1, in a similar way. So 0 means ‘no items’, 1 means #, and now we have to carry to another digit, so 10 means ##, 11 means ###, 100 means #### and so on. This base two, or binary, representation of numbers is the cornerstone of digital (there’s that word digit again) computation. Note that these are the same counting numbers as before. We’re just writing them differently. In one notation ############# is 13, and in the other it’s 1101, but these are just two different ways of representing “the number of hash symbols in #############”. We’re really used to seeing that written ‘13’, but it’s still an arbitrary notation. ‘1101’ is just as valid. To avoid confusion between the two representations, we’ll suffix binary numbers with a ‘b’ in the remainder of this post. So, e.g., 13 = 1101b.

In binary notation the two digits 0 and 1 and referred to as ‘Binary digITs’, or ‘bits’. By convention, bits are commonly grouped together in eights, called ‘bytes’. Each byte can represent a number in the range 0b-11111111b, which is 0-255 in decimal notation. You’re probably used to thinking of the powers of ten: 10, 100, 1000 as round numbers. But in binary, the powers of two: 2, 4, 8, 16 etc. are the round numbers. To a computer scientist 1024 (two to the power of ten) is a round number and 1000 is not. After all, in binary 1024=10000000000b, while 1000=111110100b, which is not particularly round-looking.

Arithmetic Circuits

As we saw in the previous post, Boolean algebra has two states we called ‘false’ and ‘true’. But in a mathematical context it’s more useful to call them 0 and 1. And just as we were able to build physical circuits to compute Boolean functions (functions with inputs that are either ‘false’ or ‘true’) so can we build circuits to compute functions with inputs that are either 0 or 1. Which means that basic mathematical operations on natural numbers, such as adding them together, can be implemented by physical electronic circuits! For example, a circuit to add two eight-bit numbers together would have eight input wires to represent the first number, eight input wires to represent the second number, and eight output wires to represent the result. We set the input bits to be 1 or 0 by passing current only through the appropriate input wires, and we measure the 1s and 0s of the result by checking which output wires have current and which don’t.

It’s a Little Trickier Than That

We’ve seen how, thanks to binary representation, it’s possible to build physical electronic circuits that can do simple math on whole numbers. However there remain a great many complications. Some of these are:

What should happen when a computation overflows the number of output wires? In our example above, 8 wires is enough to represent the binary numbers 0b-11111111b, which are 0-255 in decimal. But suppose we add 10010110b + 11001000b, which in decimal is 150 + 200. That would be 350, which is 101011110b, requiring nine digits, while our circuit only has eight output wires.
How to represent non-counting numbers: negative numbers, fractions, and irrational numbers such as pi.
How to perform more complicated mathematical operations, such as division and taking roots.

Explaining these is beyond the scope of this post, but I may address them in a future post, if there’s enough interest.

For now, at least, we’ve established that it’s possible to build physical electronic circuits that can not only compute logical statements, as we saw last week, but also count and do basic arithmetic, all through the magic of binary numbers.

There’s one more element that we need to put in place before we can really call our circuit a ‘computer’, and we’ll talk about that in the next post.

CS101 part 1: Boolean Algebra

Thu, 13 Sep 2012 08:05:00 -0700

Several people have suggested that I write a series of “CS101” posts explaining computer science and software engineering fundamentals, unrelated to anything in the news cycle. This idea appealed to my inner didact, so I’ll try a few posts along those lines and see what responses I get. Feel free to comment and/or suggest topics.

Hopefully, reading these posts will provide non-engineers in the tech industry with a better sense of what goes on under the hood, of what their engineering colleagues do, and of why software engineering is such a fascinating, intricate discipline. To any engineers reading this: you’ll have to forgive my simplification of many concepts, in the interest of clarity and brevity.

I’ll kick off, appropriately, by discussing the lowest level concept in Computer Science…

Boolean Algebra

Boolean algebra, first developed by 19th century mathematician George Boole, is a mathematical formalization of intuitive logic. It provides a formal framework for reasoning about relationships between mathematical quantities that are limited to two possible values: true and false, often denoted as 1 and 0, respectively. In particular, Boolean algebra allows us to reason about operations on truth values.

The three basic operations on truth values are:

NOT: NOT(true) is false, and NOT(false) is true.
OR: OR(X, Y) is true if either X or Y are true, and false otherwise.
AND: AND(X, Y) is true if both X and Y are true, and false otherwise.

Note how intuitive these definitions are. For example, we all understand intuitively that the statement “X is a programming language and Y is programming language” is only true if the statements “X is a programming language” and “Y is a programming language” are each independenly true. So for X=Java and Y=Ruby, the statement is true, but for X=Java and Y=Spanish the statement is false

The basic Boolean operations can be combined, e.g., we can formalize the intuitive statement “either X is false and Y is true, or Z is false” as OR(AND(NOT(X), Y), NOT(Z)). Boolean algebra provides rules for manipulating these expressions, much as regular high-school algebra does for expressions involving numbers. Such expressions are known as Boolean functions: They take one or more truth values as input, and output a single truth value. Crucially:

Any Boolean function can be written as some combination of the basic Boolean operations.

So far so good, but why is Boolean algebra so central to computer science? Because it provides the bridge between the abstract world of mathematics and the physical implementation of computer hardware.

Boolean Hardware

Electronic circuits consist of components connected by wires. We can assign a truth value to a wire in an electronic circuit: true if there is electrical current on that wire and false if there isn’t.

Now it turns out to be possible to construct circuits that implement each of the three basic Boolean operations. For example, it’s possible to build a little circuit that has two wires going in and one coming out, and in which current comes out only if both incoming wires have current. This little contraption is a real life AND circuit: a physical thing that implements the abstract mathematical concept of Boolean AND. Similar circuits can be built to implement OR and NOT.

Such circuits can be combined, by connecting the output wire of one to an input wire of another so that current can flow from one to the other. As we saw above, any Boolean function is some combination of the 3 basic operations, and therefore:

Any Boolean function can be implemented as an electronic circuit.

This is a remarkable engineering fact. Given any Boolean function, we can build hardware that can actually compute it! If we want to find out what the function’s output is for a given set of inputs, we simply apply current to the wires representing the inputs that are true, apply no current to the wires representing the inputs that are false, and measure the current on the output wire.

Now Repeat Billions of Times

The advent of transistors and, later, integrated circuits have made it cheap and easy to mass-produce phenomenally complicated circuits, consisting of many millions of AND, OR and NOT operations, modeling similarly large and complicated Boolean functions. These circuits form the basis of real-life computers.

In the next post in the series we’ll examine how these circuits are used to provide elementary computing capabilities.

Posts To Resume Shortly

Tue, 11 Sep 2012 22:30:07 -0700

Sorry for the long hiatus. I was on a much-needed vacation in Australia, Vanuatu and New Caledonia. I will resume regular updates shortly.

A Reliable Whole From Unreliable Parts

Sat, 18 Aug 2012 18:07:00 -0700

Wired Enterprise recently published an excellent article about how Google revolutionized distributed systems, creating the technologies that power not only Google, but much of the rest of the internet. The article gives some much-deserved limelight to Jeff Dean and Sanjay Ghemawat, accurately describing them as “two of the most important software engineers of the internet age — and two of the most underappreciated”. I might also add, “two of the most unassuming”.

I won’t belabor points already made in that article, but it’s worth emphasizing the key insight that underlies most of that groundbreaking work:

Building Internet-scale systems is the art of constructing a reliable whole from unreliable parts.

These days this may sound obvious to most engineers. But it’s easy to forget that in the first dot-com era, the “big iron” approach was far more common.

Back then, much of the internet was powered by massive, high-end servers, notably Sun’s Enterprise line. These machines attempted to provide scalability and reliability in a single computer: many processors and massive amounts of memory were racked together, each part was engineered for high quality, and each function was provided by multiple redundant components, so the machine could keep running when individual parts failed. Entire sites, even large ones such as eBay, ran off a single such machine or a small cluster of them. If a machine went down, part or all of the site went down, but the quality of the machines was such that this was rare.

These machines were awesome to behold. They were also expensive. A fully-loaded Sun E10K cost well over a million dollars. And that’s in 1999 dollars…

What Jeff, Sanjay and others at Google and elsewhere realized in the late 90s and early 2000s was this: no matter how expensive any machine was, at some point it would fail, taking at least part of the site down with it. And as the site scaled up to meet the growth in users and content, these failures would become more frequent. This meant that there needed to be a way to provide site reliability in the face of machine failure.

But once you have the ability to handle machine failure without loss of service, you can not just tolerate machine failure but embrace it, allowing you to buy much cheaper hardware. That $1M box cost 1000x more than a commodity machine, but provided perhaps 50x the computing power. What you were paying the huge premium for was hardware reliability. However once you’re able to handle hardware failure, you may as well spend that $1M on a thousand cheap computers, and get much, much more computing power for your money. This insight led to the now familiar datacenter architecture of racks and racks of cheap commodity machines connected by a fast local network.

So how do you get reliability if you can’t provide it in hardware? You provide it in software. This pushes the burden of reliability onto the software engineers. The problem is that writing reliability-guaranteeing software for a distributed architecture is much harder than writing for a single “big iron” machine: every system has to knit together data and computation from multiple different machines, some of which might fail in mid-flight. At the time, few engineers had experience with distributed systems. To this day many don’t.

Most of Jeff and Sanjay’s work, therefore, has been about making distributed systems easier to write, by providing layers that abstract away the particulars of how data and computation are divided up among machines, and how failures are handled. Essentially, their work has been about providing reliability in software instead of in hardware, allowing application engineers to focus on the particulars of their application instead of re-inventing distributed wheels.

For example: most of you are familiar with the local filesystems on your laptops. You save a file under some name, and it’s stored on your machine’s disk drive, under some directory path. But in a distributed system, which machine should this file live on? What happens when that machine dies?

The solution was the seminal GFS (Google File System), which offered functionality that looked like that a regular file system, but under the covers split the file up into chunks, and stored each chunk on at least 3 different machines. If any machine died, the chunk was still available on the other machines. GFS dealt transparently with pushing the data around the network, figuring out where the data lives, rewriting chunks after machine failure in order to maintain 3 copies, reconstituting the chunks into a single file and so on. Any other application at Google can use GFS naively, without having to be aware of all those details, just as you use your local filesystem without having to be aware of exactly where on the magnetic platter your bits and bytes go.

These ideas spawned a host of proprietary and open-source systems and designs that are in wide use today everywhere from Google and Facebook down to the smallest startups. The web as we know it could not exist without the distributed systems innovations of the past 12 years, and this is why the names “Jeff and Sanjay” are still spoken with awe by engineers all over Silicon Valley.

Zinc and Pants

Thu, 16 Aug 2012 00:20:59 -0700

If you’re interested in Scala and/or in build systems, Typesafe just published a blog post about Zinc, the standalone incremental Scala compiler. Among other things, it mentions the work I’ve been doing on integrating Zinc into Pants (Twitter’s open-source build system; we’re moving towards using it at foursquare).