First, you need a 32-bit wineprefix to install ie8 with winetricks, because the custom launcher uses embedded internet explorer. (Currently Proton doesn't crash on loading a 32-bit wineprefix, so don't mess with Steam's proton system files.) Then, you need to set Windows XP mode in winecfg for the BYOND account login or registration button to work.

After you've successfully connected your BYOND account to Steam and the custom launcher is giving you the options to run the game, you can return winecfg to Windows 7 mode if you prefer. You're probably going to get incredibly poor performance either way. It doesn't seem like the launcher can properly enable hardware rendering.

If you know what BYOND is and can get it running better in Wine directly, the launcher should unpack the BYOND game file to "steamapps/compatdata/249550/pfx/drive_c/users/steamuser/My Documents/BYOND/MyHub/silkgames/nestalgia/", but only users who have purchased it via BYOND or Steam will be able to log-in and play (so it's not 100% DRM-free). Good luck.