harfbuzz recently added hb_font_get_glyph_h_advances()
and some similar functions for other cases, eg see this commit :
https://github.com/harfbuzz/harfbuzz/commit/28e2f467e1c7dd5e82d105efa8c4580dbfc93960#diff-b71ebe9c54b0d6421ee7318cf0edadcd

I hope that if we use this API we can get some performance improvement
than one call per glyph.

It will require upgrading to the latest harfbuzz, but we will do that at
some point soon anyway.