Note that any object used as a key (eg. in HashSet<T>.Add()) must have a suitable implementation of Object.GetHashCode() and should override Object.Equals() as described here.
If you’re using .NET Core then you can easily generate reliable hashcodes using System.HashCode.
If not, then you can copy it into your project from the .NET Core source.