The initial scanning/refresh is in general much slower than subsequent refreshes due to the system disk cache.
A manual View|Refresh will be needed, if you are performing SVN operations with another SVN client (i.e. outside of SmartSVN).
SmartSVN is only refreshing relevant parts and with changes to
(1) process .
(2) don't process these events (because they have been produced by SmartSVN itself and hence are redundant).
Given commonness of these use cases and performance reasons, we have have opted for (2).