It seems removing the row height disabled the extra space for each additional subcategory, even though the extra space seemed longer than 80vh. Was it adding the 80vh after the end of the thumbnail grid perhaps?
However, this means I can't use custom heights when using this setup. If there are no other issues going on, then this fix will work as well :-)