You can prevent someone from connecting to your Bluetooth speaker by using a security code, buying a more advanced speaker, switching off the speaker when you're not using it, unpair unwanted devices, update your software, and switch off visibility.
Using KNOB to hijack Bluetooth speakers with iPhone or Android. To do this, you will need to do two things: Escalate the KNOB attack: Let the KNOB attack decrypt or weaken the Bluetooth traffic between the two devices. This gives you a window to take over the session.