After fiddling with the settings for Caller ID on my FXO port, I finally found something that works:

Caller ID Scheme: SIN 227 – BT
FSK Caller ID Minimum RX Level (dB): -40
FSK Caller ID Seizure Bits: 96
FSK Caller ID Mark Bits: 55
Caller ID Transport Type: Relay via SIP From

If you have a Grandstream HandyTone 503 connected to a BT line on the FXO port then leave your configuration tips below!