The RE contained an error. This should work better:
ISBN[- ]((?:978[- ]?)[0-9- ]+)\b
Using \1
in the “Set ISBN to…” field of a smart rule works as expected, “ISBN” being defined as a single-line text-field.
But I’d rather go for a small script that standardizes the ISBN before storing it as a custom metadata field. As @meowky pointed out, ISBNs come in many variants, and if you want to use them eg for searching, you’d be better off with a unified format, containing only digits.