For Mac you need to define plist entry to declare the URL scheme and later handle the AppleEvent.
For Windows it is more difficult. First you edit registry to declare your URL scheme. Than you need to handle the URL as start parameter. But as a second copy of your app may be launched, you need to talk yourself to the first instance via IPCSocket and pass URL command to main app.
Robert Beier from Dibomedia GmbH in Leipzig recently implemented a helper class to handle this and put it on githup:
Feel free to send feedback including bug fixes to him. And next time you meet him at a Xojo conference, you can of course invite him for a drink!