You would be 100 percent correct with all you have stated in this. Often times there can be substances that leak down the joystick wire from drinks or human fluids or other outside elements.
You will want to slide the gray part of the harness then pull to disconnect it. If it is having issues coming off, try a little rubbing alcohol or let sit with some WD-40 to break away any substances.
If the connector was forced together by putting it upside down, then you may have to figure out other means of gripping the connector to separate them.
Once you have the connector loose you can take the seat right off.