Gems are used to permanently increase your stats on the character you consume them on. Upon death, you will lose all consumed gems and will start anew. When the stat is maxed, you will be unable to consume more gems on that specific character.
A list of Gem drop locations can be found here.