Partially, the online speed test like Pingdom measure only a first page load (no browser cache) for non loggedin users (no loggedin cache) so you may find few difference, but your users will!
The best way to measure the performance, is to test by yourself the time to render a page in your browser.
Sure, cleaning the cache from Speed Cache automatic system or from Cache Cleaner extension will have the same result.
Usually yes, you do need to activate cache per user in this case. You may not include some URL that have fresh content every second to work properly.
To use speed cache on dynamic environnement, please check every case your users may encounter before validating the URL presence in Speed Cache.