The disadvantage of a lower buffer size is that this puts more pressure on your CPU and makes the processors work harder. When you are tracking, you should lower the buffer size to at least 256 samples to reduce the amount of latency you hear when monitoring.
Here are a few scenarios of when you would want to adjust the buffer size: Are you hearing pops and clicks? Are you receiving CPU overload error messages? Hearing latency? Changing the buffer size could fix the issues above as this is the amount of time allotted for processing.