It would be worth using the Inspector function in Safari (available when developer tools is enabled) to test a loading page. Use the outputted bar charts to gauge which files are the slowest to download. Items added to a page like Flash or large images can dramatically increase load times. Custom code added to a theme may also increase load times.

If your page files are still relatively small (under 200 KB) the problem might be with your internet connection or web host.