I opened a page to experiment with it.
Added a button.
Clicked the X to close the tab and chose “Proceed anyway”.
Next I re-open the same page, and it opens with the button I had just added.
… So, it seemed like it was saving anyway.
BUT … I got suspicious and pressed CMD+R to reload, and the button is gone, so it didn’t really save it after all.
Confirmed and noted. This will be looked into!