sodcalculator

10 Tech How-To

Encode JavaScript Text as UTF-8 Bytes

Guide 10-55845 · Updated · Step by step

Encoding JavaScript text as UTF-8 bytes is the process of converting a string of characters into a sequence of bytes using the UTF-8 character encoding. This is essential when you need to transmit or store text in a format that is universally understood, such as in HTTP requests, file I/O, or cryptographic operations. In this guide, you’ll learn multiple methods to perform this conversion in JavaScript, both in the browser and in Node.js, along with common pitfalls and troubleshooting tips.

What You’ll Need

  • A basic understanding of JavaScript and its data types.
  • A JavaScript environment: either a modern web browser (e.g., Chrome, Firefox) or Node.js installed on your machine.
  • For Node.js, you’ll need access to the built-in Buffer class and the TextEncoder global (available since Node.js 11).

Step-by-Step

Method 1: Using TextEncoder (Browser and Node.js)

  1. Create a TextEncoder instance. In both browsers and Node.js (v11+), the TextEncoder global is available. Simply instantiate it: const encoder = new TextEncoder();
  2. Encode your string. Call the encode() method on the encoder, passing your string as the argument. For example: const bytes = encoder.encode('Hello, world!');
  3. Inspect the result. The encode() method returns a Uint8Array containing the UTF-8 bytes. You can log it to the console: console.log(bytes); This will output something like Uint8Array(13) [72, 101, 108, 108, 111, 44, 32, 119, 111, 114, 108, 100, 33].
  4. Convert to a regular array (optional). If you need a plain array of numbers, use Array.from(bytes) or the spread operator: const byteArray = [...bytes];

Method 2: Using Buffer (Node.js)

  1. Use Buffer.from(). In Node.js, you can directly convert a string to a Buffer with UTF-8 encoding: const buffer = Buffer.from('Hello, world!', 'utf8');
  2. Access the bytes. The Buffer object is a subclass of Uint8Array, so you can access individual bytes via indexing or convert to an array: const bytes = [...buffer]; or const byteArray = Array.from(buffer);

Method 3: Manual Encoding (for educational purposes)

  1. Understand the UTF-8 mapping. UTF-8 encodes each code point (Unicode character) into one to four bytes. ASCII characters (U+0000 to U+007F) use one byte, which is the same as the character code. For characters beyond ASCII, you need to apply the UTF-8 bit-packing rules.
  2. Use a loop to encode. A manual implementation involves iterating over the string’s code points (using codePointAt()) and building the byte sequence. Here’s a simplified version:
    function toUTF8Bytes(str) {
      const bytes = [];
      for (let i = 0; i < str.length; i++) {
        let code = str.codePointAt(i);
        if (code > 0xFFFF) i++; // skip the second surrogate for astral characters
        if (code < 0x80) {
          bytes.push(code);
        } else if (code < 0x800) {
          bytes.push(0xC0 | (code >> 6), 0x80 | (code & 0x3F));
        } else if (code < 0x10000) {
          bytes.push(0xE0 | (code >> 12), 0x80 | ((code >> 6) & 0x3F), 0x80 | (code & 0x3F));
        } else {
          bytes.push(0xF0 | (code >> 18), 0x80 | ((code >> 12) & 0x3F), 0x80 | ((code >> 6) & 0x3F), 0x80 | (code & 0x3F));
        }
      }
      return bytes;
    }
        
  3. Test it. Call the function with a string containing non-ASCII characters, like console.log(toUTF8Bytes('é')); which should output [195, 169].

Common Mistakes

  • Assuming charCodeAt() gives UTF-8 bytes. charCodeAt() returns the UTF-16 code unit, which is not the same as UTF-8 bytes. For ASCII characters it matches, but for accented characters or emoji it will give a number greater than 255.
  • Using the wrong encoding in Buffer.from(). If you omit the encoding parameter, Buffer.from() defaults to UTF-8, but if you specify something else like ‘ascii’, non-ASCII characters will be truncated or replaced.
  • Not handling surrogate pairs. When manually encoding, you must handle code points above U+FFFF (like emoji) correctly. Simply iterating over string indices with charCodeAt() will treat surrogate pairs as two separate characters.
  • Forgetting that TextEncoder is not available in older Node.js versions. If you’re using Node.js < 11, you'll need to use Buffer or a polyfill.

Tips

  • Use TextEncoder for modern environments. It’s fast, standard, and handles all Unicode correctly. It’s the recommended approach in both browsers and Node.js 11+.
  • Convert to ArrayBuffer if needed. To get an ArrayBuffer, you can use bytes.buffer from the Uint8Array, but be careful: the buffer may be larger than the byte length if the Uint8Array is a view. Use bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) to get the exact portion.
  • For Node.js, Buffer is more flexible. Buffer allows you to easily convert back to a string using buffer.toString('utf8').
  • Test with non-ASCII characters. Always test your encoding with characters like ‘é’, ‘€’, or ‘😀’ to ensure your method handles multi-byte characters correctly.

Troubleshooting

  • Getting unexpected byte values. Double-check that you are using UTF-8 and not UTF-16. For example, the character ‘é’ should encode to [195, 169], not [233] (which is the Latin-1 byte).
  • TextEncoder is not defined. If you’re in an older Node.js environment, you can install a polyfill or use Buffer. In browsers, TextEncoder is supported in all modern browsers, but if you need to support very old ones, you might need a polyfill.
  • Encoding produces empty array. If your string is empty, the encoded result will be an empty array, which is correct. But if you’re getting an empty array for a non-empty string, check that you’re passing the string correctly and not accidentally passing an array or object.
  • Getting ‘Cannot read property of undefined’ when using Buffer. Make sure you’re in a Node.js environment, not a browser. Buffer is not available in browsers by default.
  • Manual encoding gives wrong results for emoji. Ensure you’re using codePointAt() and correctly skipping the second surrogate. Also, the length property of a string counts UTF-16 code units, so you need to adjust the loop index accordingly.

Frequently Asked Questions

What is the difference between UTF-8 and UTF-16?

UTF-8 encodes each Unicode character into 1 to 4 bytes, with ASCII characters using a single byte. UTF-16 uses 2 or 4 bytes per character. JavaScript strings are internally stored as UTF-16, but when you encode to UTF-8, you get a different byte sequence.

Can I decode UTF-8 bytes back to a string?

Yes. In browsers, you can use TextDecoder with the ‘utf-8’ encoding. In Node.js, use Buffer.from(bytes).toString('utf8').

Why do I need to encode text as UTF-8 bytes?

Encoding is necessary when sending text over a network, storing in files, or hashing. Most protocols and file formats expect UTF-8 bytes.

Does TextEncoder handle all Unicode characters?

Yes, TextEncoder supports the full Unicode standard, including emoji and rare scripts. It will correctly encode any valid JavaScript string.

Is there a performance difference between TextEncoder and Buffer?

TextEncoder is generally faster in browsers and modern Node.js. Buffer is also efficient but adds a bit of overhead due to its mutable nature. For most use cases, the difference is negligible.

How do I convert a Uint8Array to a regular array?

Use Array.from(uint8Array) or the spread operator: [...uint8Array]. This gives you a plain array of numbers.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →