Triple silken pumpkin pie takes the prize
11/26/2008 11:00:00 AM
What is the first thing that most people associate with Thanksgiving? Well, probably turkey, but pumpkin pie comes in a close second. As I am not a fan of the traditional pumpkin pie, I set out on a quest to find a delicious alternative this year. My search led me to a robust cookbook sitting on my kitchen shelves. Sheri Yard's Desserts by the Yard is an amazing compilation of a pastry chef's career spanning from coast to coast. What I found in that book turned out to be the most fluffy, decadent, flaky, scrumptious pie I have ever tasted. And apparently my officemates liked it just as much -- the triple silken pumpkin pie and I took home first place in last week's bake-off at our New York office! So if you're looking for a holiday-perfect pie, I encourage you to try out the recipe (PDF file). It takes a little time to make, but it's so worth it.

Happy baking, and happy Thanksgiving!



Posted by Kayla Wilson, Recruiting Coordinator

Permalink Links to this post

Labels: googlers and culture, recipe

Sorting 1PB with MapReduce
11/21/2008 04:55:00 PM
At Google we are fanatical about organizing the world's information. As a result, we spend a lot of time finding better ways to sort information using MapReduce, a key component of our software infrastructure that allows us to run multiple processes simultaneously. MapReduce is a perfect solution for many of the computations we run daily, due in large part to its simplicity, applicability to a wide range of real-world computing tasks, and natural translation to highly scalable distributed implementations that harness the power of thousands of computers.

In our sorting experiments we have followed the rules of a standard terabyte (TB) sort benchmark. Standardized experiments help us understand and compare the benefits of various technologies and also add a competitive spirit. You can think of it as an Olympic event for computations. By pushing the boundaries of these types of programs, we learn about the limitations of current technologies as well as the lessons useful in designing next generation computing platforms. This, in turn, should help everyone have faster access to higher-quality information.

We are excited to announce we were able to sort 1TB (stored on the Google File System as 10 billion 100-byte records in uncompressed text files) on 1,000 computers in 68 seconds. By comparison, the previous 1TB sorting record is 209 seconds on 910 computers.

Sometimes you need to sort more than a terabyte, so we were curious to find out what happens when you sort more and gave one petabyte (PB) a try. One petabyte is a thousand terabytes, or, to put this amount in perspective, it is 12 times the amount of archived web data in the U.S. Library of Congress as of May 2008. In comparison, consider that the aggregate size of data processed by all instances of MapReduce at Google was on average 20PB per day in January 2008.

It took six hours and two minutes to sort 1PB (10 trillion 100-byte records) on 4,000 computers. We're not aware of any other sorting experiment at this scale and are obviously very excited to be able to process so much data so quickly.

An interesting question came up while running experiments at such a scale: Where do you put 1PB of sorted data? We were writing it to 48,000 hard drives (we did not use the full capacity of these disks, though), and every time we ran our sort, at least one of our disks managed to break (this is not surprising at all given the duration of the test, the number of disks involved, and the expected lifetime of hard disks). To make sure we kept our sorted petabyte safe, we asked the Google File System to write three copies of each file to three different disks.

Significantly improved handling of the so-called "stragglers" (parts of computation that run slower than expected) was a key software technique that helped sort 1PB. And of course, there are many other factors that contributed to the result. We'll be discussing all of this and more in an upcoming publication. And you can also check out the video from our recent Technology RoundTable Series.

Posted by Grzegorz Czajkowski, Systems Infrastructure Team

Permalink Links to this post

Our international approach to search
11/21/2008 11:46:00 AM
In previous posts in this series, you have read about the challenges of building a world-class search engine. Our goal is to make Google’s search be relevant to all people, regardless of their language or country. As my colleague Amit Singhal described, we use statistical data as the basis for making sweeping algorithmic changes. Many of these changes can be rolled out across all languages we support, but in some cases the unique characteristics of each language require some algorithmic considerations and tuning. And to make things really interesting, there are cases where the same language is different across countries. Obvious examples are "color" in the U.S. vs. "colour" in the U.K., or "camião" in Portugal vs. "caminhão" in Brazil.

My name is Daphne Dembo, and my focus is improving Google's international search. This is a tough challenge, since Google search is used in many countries and languages where our engineers have little personal knowledge. Initially, the international search improvements were done by Search Quality engineers who were passionate about their languages and countries: Lina from Sweden improved our parsing of compound words in German and Swedish; Dimitra from Greece introduced diacritical support; Ishai from Israel worked on transliteration corrections for Hebrew and Arabic; Trystan from Australia created methods for identifying local search results and ranking them together with foreign ones from the same language; Alex, a bilingual Ukrainian and Russian, introduced morphological understanding of these languages. As the importance of our international search grew, we solicited help from Googlers in all our offices. Finally, we are leveraging an international network of search specialists who help us understand search within the unique combination of their language and country.

Our first step in providing search support for a language is to train our language model on a large collection of documents in that language. This ensures that our language model is more precise and comprehensive — for example, it incorporates names, idioms, colloquial usage, and newly coined words not often found in static dictionaries. For instance, we recently started identifying Swahili, and used pages such as this one for the Parliament of Tanzania to train our system with the language's nuances. Having a trained language model helps to categorize documents during crawling and indexing of the web and to parse the user's query. Once this stage was complete, we launched Swahili search in countries such as Tanzania and Kenya, enabling local searches for the "Dar es Salaam stock exchange" [Soko la hisa dar es salaam], and "cure for Malaria" [Tiba ya malaria]. (As always, we are using square brackets to denote a search query. For example, you can search for "soccer" in Hamburg, Germany by clicking on [fußball in hamburg]).

We learn some things from our users, so as people start using our search engine, we can improve the way we rank in that language. Here are few examples:

* Spell corrections: We recently launched spell corrections in Estonian. If your Estonian is rusty, and you don't remember how to spell "smoke detector," we can suggest a spell correction for [suitsuantur], leading to better search results.
* Diacritical marks: Many languages have diacritical marks, which alter pronunciation. Our algorithms are built to support them, and even help users who mis-type or completely ignore them. For example, if you're a resident of Quebec, Canada and would like to know the weather forecast in Quebec City, we'll serve good results whether you type with diacritical signs [Météo à Québec] or without [meteo quebec]. Czech users can read the same excellent results for a popular kids' cartoon by searching for [krtecek] and [krteček]. On the other hand, sometimes diacriticals change the meaning of the word and we have to use them correctly. For example, in Thai, [ข้าว] is "rice," with completely different results than [ข่าว], which is "news"; or in Slovakia, results for "child" [dieťa] are different than results for "diet" [diéta].
* Synonyms: A general case of diacritical support is the handling of synonyms in different languages. Korean searches showed that "samsung" can be viewed as a synonym of "삼성", so that when users search for [samsung], they find results which have the company's name in Korean.
* Compounding: Some languages allow compounding, which is the formation of new words by combining together existing words. You can see a nice example in Swedish, where we return documents about a Swedish credit card for both compounded [Visakort] and non-compounded [visa kort] queries.
* Stemming: Google has developed morphological models that can receive compound words as queries, and return pages which contain their stem, possibly as part of a different compound. For example, when searching for cars in Saudi Arabia, you can search for [سيارة] and [سيارات] because both are variants of the same stem, and both return many common results. A Polish user can search for "movie" [film], and get back results that contain other variants of the stem, such as "filmów," "filmu," "filmie," "filmy." A user from Belarus will find results for all word forms of the capital, Minsk [Мінск]: "Мінску," "Мінска," "Мінскага."

In addition to these semantic factors, Google does even more to parse documents and queries. Understanding the details of language usage in a country is important. Notation of acronyms is different across languages: In Hebrew it is double quotes before the last (left-most) character, as in "prime minister" [רה"מ]; in Thai — a dot at the end of the word, as in police station [สน. ]; while in the U.S. — dots after each character, as in [I.B.M.]. Chinese users quote works of art with a "《", as in: [《手机》剧情], and denote dates with a "日", as in: [2006年1月13日].

Beyond the linguistic elements of a language, we consider how people enter a query. For example, some languages that do not have Latin scripts require keyboards with dual alphanumeric keys. The user can switch between language input modes by typing special keystrokes. In case the user forgets to type this sequence, the queries end up being gibberish. You can see correct handling of these mistakes in Arabic ([hgsuv] corrected to [السعر]) and ([حقثسهيثىفهشم ثممثؤفهخىس ] corrected to [presidential elections]), Hebrew ([vdrk, kuyu] corrected to [הגרלת לוטו]), and Cyrillic ([rehc ljkffhf] corrected to [курс доллара]).

Another way of avoiding the inconvenience of switching keyboard modes is by typing the phonetic sounds of the query in Latin characters. Recreating the correct query in the target language isn't trivial, since there might be many possibilities. We can see several such examples in which we suggest the same query in the intended language for Russian ([biskvitnyi rulet] to [бисквитный рулет]), "movies" in Chinese ([dianying] to [电影]), and "Bank of Attica" in Greek [trapeza attikhs] returns good results for "Τράπεζα Αττικής". Users of 8 Indic languages (such as Hindi, Gujarati, Telugu) can type the phonetic sound of the query, and choose the words in Hindi script:


Ease of typing and reading is also influenced by the language used. Since every Chinese word requires several keystrokes on a standard keyboard, we provide category browsing by Images and related searches so that people don't need to type as much. Similarly, we are now launching Google Suggest, or real-time completion of queries, in many languages.

So far I described how we improve the quality of search in a language. However, there is a strong effect of the location of the user, even if it is only approximated to the country, since in many cases local content is more relevant than global information. For example, searching for Spanish Yellow Pages [Páginas Amarillas] will result in several documents of global interest and several local results in Peru, Mexico, and Spain. Similar to that, searching for [Côte d'Or] in France will return results for that region, whereas searches in Belgium will return results about the chocolate maker.

Note that the display of information should conform to the standards in that country, so we display "," as a decimal notation for Croatian users who want to know how many millimeters are in an inch [inč u milimetrima], or for Italian users who are interested in currency exchange rates [50 euro in dollari]. Similarly, temperatures in Norway [Været i Oslo] will be displayed in Celsius, while in the U.S. — in Fahrenheit [weather Boston].

If everything else fails, we provide cross-language translations based upon Google's translation technology described in this blog post. We will translate your query to English, search English documents on the web, and translate the returned results from English back into the original query language. For example, Japanese users who are interested in viewing Halloween illustrations (Halloween is a holiday which originated in Ireland) can search for [ハロウィン イラスト]. You can then request a Japanese translation of the English pages (at the bottom of the page), which will bring up the translation page in the screenshot below. Similarly, Korean users can search for the latest on Harry Potter [해리 포터], and Arabic readers can search for the opening of the Sydney Opera house [افتتاح دار الاوبرا في سيدني]. (Click on the image to see a larger version.)



All in all, Google Search is being actively developed for more than 100 languages, in 150+ countries, with dozens of improvements launched each month. So far I've covered the basics of how international search works, but this is just the surface of all the international work we do. There are many other interesting topics that impact international markets like usability, homepage and results page layout, and connectivity. An understanding of real cultural and human factors is essential to creating a search engine that resonates with the people who use it. (Click on the image to see a larger version.)



(Update: Replaced example in the 4th bullet point.)

Posted by Daphne Dembo, Engineering Director

Permalink Links to this post

Labels: search quality

SearchWiki: make search your own
11/20/2008 04:36:00 PM
Have you ever wanted to mark up Google search results? Maybe you're an avid hiker and the trail map site you always go to is in the 4th or 5th position and you want to move it to the top. Or perhaps it's not there at all and you'd like to add it. Or maybe you'd like to add some notes about what you found on that site and why you thought it was useful. Starting today you can do all this and tailor Google search results to best meet your needs.

Today we're launching SearchWiki, a way for you to customize search by re-ranking, deleting, adding, and commenting on search results. With just a single click you can move the results you like to the top or add a new site. You can also write notes attached to a particular site and remove results that you don't feel belong. These modifications will be shown to you every time you do the same search in the future. SearchWiki is available to signed-in Google users. We store your changes in your Google Account. If you are wondering if you are signed in, you can always check by noting if your username appears in the upper right-hand side of the page.

The changes you make only affect your own searches. But SearchWiki also is a great way to share your insights with other searchers. You can see how the community has collectively edited the search results by clicking on the "See all notes for this SearchWiki" link.

Watch our lead engineer, Amay, demonstrate a few ways to use SearchWiki in this short video:



This new feature is an example of how search is becoming increasingly dynamic, giving people tools that make search even more useful to them in their daily lives. We have been testing bits and pieces of SearchWiki for some time through live experiments, and we incorporated much of our learnings into this release. We are constantly striving to improve our users' search experience, and this is yet another step along the way.

Posted by Cedric Dupont, Product Manager, and Corin Anderson, Software Engineer

Permalink Links to this post

Labels: search

Transgender Remembrance Day
11/20/2008 10:01:00 AM
We believe great ideas can come from anywhere and everyone. And we aspire to be an organization that reflects global diversity, because we know that a world's worth of perspectives, ideas and cultures leads to the creation of better products and services. We have more than a dozen employee-driven resource groups, from Gayglers to GWE (Google Women Engineers), that actively participate around the world in building community and driving policy at Google. The post below kicks off an occasional series, entitled Interface, about valuing people's similarities and differences in the workplace. For more information on how Google fosters an inclusive work environment, visit Life at Google on our Jobs site. – Ed.

November 20th marks Transgender Remembrance Day, which takes on a special significance in a world awakening to the need for unity among all people. In observing this day, the Gayglers — the Lesbian, Gay, Bisexual and Transgender (LGBT) group within Google — extend their wholehearted support to the LGBT community at large, as we reflect on the senseless violence perpetrated against transgender people around the world.

People who identify or express their gender differently than the one assigned to them at birth usually call themselves transgender or transsexual. All too often, they are subjected to a range of not-so-subtle prejudices and transphobia, from verbal abuse to physical violence. Imagine walking into a public restroom in a state of dread over a confrontation about your appearance. Imagine visiting a doctor and worrying about how far to "out" yourself to receive appropriate care. Studies suggest that transgender people are 16 times more likely to be killed than the general population -- earlier this month, in fact, a transgender woman in Tennessee was murdered -- and this is just the most recent of many such cases.

We're fortunate here at Google, where there are LGB and T people at all levels of the company, thanks to enlightened hiring and promotion practices that set aside sexual orientation or gender presentation. Ultimately, Google fosters a workplace where everyone has the ability to be themselves at work. For transgender employees in particular, that means everything.

On this Transgender Remembrance Day, take just a few moments to remember the trans siblings, parents, friends and lovers who lost their lives to gender-based intolerance and hatred. Let's all share in a future where tolerance and understanding transforms the world. And let's work to create a better place for everyone to live peaceably in an all-inclusive world community that merits our deepest pride.

Posted by Kai Johnson, Software Engineer

Permalink Links to this post

Labels: diversity

Lively no more
11/19/2008 05:55:00 PM
In July we launched Lively in Google Labs because we wanted users to be able to interact with their friends and express themselves online in new ways. Google has always been supportive of this kind of experimentation because we believe it's the best way to create groundbreaking products that make a difference to people's lives. But we've also always accepted that when you take these kinds of risks not every bet is going to pay off.

That's why, despite all the virtual high fives and creative rooms everyone has enjoyed in the last four and a half months, we've decided to shut Lively down at the end of the year. It has been a tough decision, but we want to ensure that we prioritize our resources and focus more on our core search, ads and apps business. Lively.com will be discontinued at the end of December, and everyone who has worked on the project will then move on to other teams.

We'd encourage all Lively users to capture your hard work by taking videos and screenshots of your rooms.

Posted by The Lively Team

Permalink Links to this post

The Final Inch
11/19/2008 11:16:00 AM
(Cross-posted from the Google.org blog)

Early readers of the Google.org blog may recall us embarking on a film project portraying public health heroes working in the field to eradicate polio. Gone from the modern world, new cases of polio continue to afflict mostly children under age 3 in the poorest regions of just a few countries — India, Nigeria, Afghanistan and Pakistan. When we first announced this project and the collaboration between Google.org and Vermilion Films, filming was underway primarily in India and Afghanistan, documenting the front lines of public health in some forgotten corners of our world.

David Heymann of the World Health Organization reminds us, "When you haven't seen a disease for quite a while, which is the case in the industrialized countries, you forget about the terrible disease that it really is." Polio is such a disease, as it can ruin the lives of children even before they are old enough to understand how to prevent it.

We're proud to announce The Final Inch, a 38-minute film about the historic global effort to eradicate polio. Here, the story told is as much about the messengers as the message. You'll meet Munzareen Fatima, one of the thousands of community "foot soldiers" across India working to sway reluctant families to vaccinate their children, and Dr. Ashfaq Bhat, who travels into the backwaters of India's Ganges Basin by boat and foot to detect emerging cases of polio. Martha Mason and Mikail Davenport bring us into their lives and describe the paralyzing challenges of childhood polio, reminding us how endemic polio once was in the United States.

Filmed in high-definition (HD) in cinematic style — wide open shots to give a strong sense of place — The Final Inch captures their stories, and we hope it is both a tribute and an inspiration of hope. With a final push, this is a disease that can, and should, be eradicated finally.

The Final Inch will air nationally on HBO in 2009. We invite you to check out TheFinalInch.org, where you can view clips from the film and learn more about the people and the organizations tirelessly working on this global effort. You can also check out the film trailer here:

Update on 1/22: The Final Inch received an Oscar nomination in the category of Best Documentary (Short Film). Check out our post on the Google.org blog for more information.

Posted by Gregory Miller, Managing Director, Google.org, and Irene Taylor Brodsky, Vermilion Films, Producer and Director of The Final Inch

Permalink Links to this post

Labels: google.org

LIFE Photo Archive available on Google Image Search
11/18/2008 08:17:00 AM
The Zapruder film of the Kennedy assassination; The Mansell Collection from London; Dahlstrom glass plates of New York and environs from the 1880s; and the entire works left to the collection from LIFE photographers Alfred Eisenstaedt, Gjon Mili, and Nina Leen. These are just some of the things you'll see in Google Image Search today.

We're excited to announce the availability of never-before-seen images from the LIFE photo archive. This effort to bring offline images online was inspired by our mission to organize all the world's information and make it universally accessible and useful. This collection of newly-digitized images includes photos and etchings produced and owned by LIFE dating all the way back to the 1750s.

Only a very small percentage of these images have ever been published. The rest have been sitting in dusty archives in the form of negatives, slides, glass plates, etchings, and prints. We're digitizing them so that everyone can easily experience these fascinating moments in time. Today about 20 percent of the collection is online; during the next few months, we will be adding the entire LIFE archive — about 10 million photos.




It has been a thrill for us to explore this archive, filled with images captured by LIFE's famous photographers. See masters like Alfred Eisenstaedt and Margaret Bourke-White documenting pivotal world events, capturing the evolution of lifestyles and fashions, and opening windows into the lives of celebrities and everyday people.

One of our favorites is this classic Eisenstaedt image of children watching a puppet show.



Alfred snapped this in 1963, at the climax of Guignol's "Saint George and the Dragon" in the Tuileries Garden in Paris. Just as the dragon is slain, some children cry out in a combination of horror and delight, while others are taken aback in shock. Every child is consumed with emotion, masterfully captured by Eisenstaedt's camera. These amazing photos are now blended into our Image Search results along with other images from across the web.

Once you are in the archive, you'll also notice that you can access a rich full-size, full-screen version of each image simply by clicking on the picture itself in the landing page. If you decide you really like one of these images, high-quality framed prints can be purchased from LIFE at the click of a button. Think of the holiday gift possibilities! It doesn't get much easier than that.

So please take a look for yourself and experience these great photos. Your exploration will be limited only by your imagination and your desire to keep on clicking. Be sure to check back often as more photos from the LIFE archive will be added regularly to Google Image Search. We hope that you enjoy them as much as we do!

Posted by Paco Galanes, Software Engineer

Permalink Links to this post

Labels: photos, search

Now you can speak to Google Mobile App on your iPhone
11/17/2008 05:43:00 PM
Have you ever been in a hurry and really needed to find an answer to something, but there was no one to ask? Like when you're grocery shopping and looking for the last item on your list, the kids are running around you in circles, you're holding a basket in one hand, and you have no idea what "fennel bulbs" look like.

That's why we've added voice search to Google Mobile App for the iPhone — and made it super easy to use. Once the app is running, you don't have to tap any buttons. Just hold the iPhone to your ear, wait for the beep, and say what you're looking for. For instance, last week when I was arm wrestling with fellow product manager Robert Hamilton, I said, "official arm wrestling rules" to Google Mobile App to settle a little dispute about his elbow placement. (After all, the middle of an arm-wrestling match is no time to be typing.) Turns out we were both disqualified because we were not using elbow pads.

Our passion for making search faster and easier goes further. When you do local searches, Google Mobile App can now automatically use your location to make results more relevant to where you are. That was really useful when I was in San Francisco last weekend and my daughter got a paper cut. Having no familiarity with the neighborhood I was in, I just searched for "pharmacies" and I was quickly on my way to the nearest place to buy a bandage. The day was saved.

Check out this video to see what other Googlers from Chicago, London, New York, and Mountain View are searching for.



To get the latest Google Mobile App for iPhone, go to the App Store on your iPhone and search for "Google Mobile App." (Note that voice search will be enabled by default for U.S. English users only.) Then, if you have a great voice search query to share, send us a video response to our video.

Learn more about the new Google Mobile App for iPhone on the Google Mobile Blog and by watching this overview video.

Posted by Gummi Hafsteinsson, Product Manager and disqualified arm wrestler, Google Mobile team

Permalink Links to this post

Labels: mobile, search

Ads in new places
11/17/2008 03:25:00 PM
At Google we're great supporters of experimentation because it's only by trying new things (even if some of them don't work out) that you discover better, more creative ways to operate. We've been testing different advertising formats for years (some have been more successful than others), and over the next few months, you'll see us continuing to experiment with new ads in new places.

If you're based in the U.S. you may already have spotted or clicked on the different text and image ads we’re testing on the results pages of Google Image Search. And last week you may have noticed we launched Sponsored Videos on YouTube — a great example of matching ads to content.

In addition, we are today launching text ads on Google Finance in the United States. We're also looking at how best to show display ads on Google Finance. And later, in the very near future, we will start testing text ads on a small number of news refinements within Google Search — so if, for example, you type "iPod" into Google.com and then click on the news link on the upper left-hand side, you might see text ads alongside those results.

Whenever we make changes like these, we carefully evaluate users' reactions to ensure we're holding true to our basic principles: that ads by Google should always be relevant and useful. Of course, these experiments benefit Google because they generate revenue from new sources — but by ensuring that we show the right ads at the right time to the right people, we'll add value for users too.