Just the colored part. Most shops with a "Kitchen Gadget" section have what are called a "citrus zester". They are specially made to scrape just the zest off of lemons, limes and oranges. (Also grape fruit, pomelo and kumquat, if you want to get bizarre.)
You can Google them if you are interested. There are two kinds. One looks like a fine cheese grater with a handle ("fine" as in "tiny holes", not "superlative".) The other kind are small blades with 4 - 6 holes as big as the holes on the small side of a cheese grater at the end of the blade. They scrape very thin strips of zest off the fruit; the fact the holes are small keeps them from digging into the bitter pith. They range in price from $3.99 to $19.99. I like the "OXO" brand "Good Grips" model from Bed, Bath and Beyond, $7.99, but you can use a plain cheese grater if you don't have the time, money or space for another kitchen gadget. I didn't buy mine until late in life, and every time I use it I regret not buying one a lot sooner. I add zest to basically every recipe that calls for fresh lemon or lime juice; it adds a lot of flavor.
In a related subject, in a restaurant, if your water comes with lemon, or your beer with lime, squeeze the juice into same, but then, if you really like citrus, flip it over so the peel is down and squeeze. The lemon or lime oil from the peel will go into your glass.