Hello World

Text is harder than it looks

All chapters

Part 1 · Foundations

Text is harder than it looks

Chapter 4 26 min read 14 sections

Someone signs up to your app as "José". The welcome email greets them as "José".

Nobody wrote a bug that adds an "Ã". The text was saved as UTF-8, and somewhere along the way something read it back as a different encoding. Every layer between the user's keyboard and their inbox has an opinion about what text is, and if one of them disagrees with the others, this is what you get. It even has a name, mojibake, from the Japanese for "character transformation".

It happens to big publishers too. At Future, which runs sites like TechRadar, article bodies went through PHP's HTML parser on their way to the page. Hand that parser a fragment of HTML with no charset declaration and it assumes Latin-1, so accented letters and curly quotes came out as mojibake. The text was fine in the database and fine in the template. One step in the middle guessed the encoding and guessed wrong, until someone told it to expect UTF-8.

Most developers learn to handle text in English, where one letter is one byte, every letter has an uppercase version, and alphabetical order is obvious. None of that holds once your users write in other languages. This chapter covers what text actually is to a computer, and the handful of things you need to get right before translation even comes into it.

Keep reading

The rest of this chapter is for readers of the book

Get the book to read the rest of this chapter and every other one, the field guide and the glossary, and to download the whole book as a PDF.

Still to come in this chapter

  • Characters, code points and bytes
  • Use UTF-8 everywhere
  • How long is a string?
  • The same text, stored two ways
  • Upper case and lower case aren't universal
  • Alphabetical order depends on the alphabet
  • Regular expressions assume English
  • Search needs to forgive
  • Slugs can change the meaning
  • Typing isn't one key per character
  • Keyboard shortcuts assume your keyboard
  • Test with text that breaks things
  • Checklist
  • Further reading