Mac OS X Hints has a post explaining “how to seperate icons in your Dock into groups, you can insert invisible spacers, much like the blank spaces which you can put into toolbars,“ in Mac OS X 10.5 Leopard, the feature that Apple missed to include. In the end you can add as many separators as you would like on either side of the Dock.

To add a spacer to the applications (left) side of the Dock, run the following in the terminal:

defaults write com.apple.dock persistent-apps -array-add '{tile-data={}; tile-type="spacer-tile";}'

To see your changes, restart the dock with killall Dock. To add a spacer to the documents (right) side of the Dock, run the following:

defaults write com.apple.dock persistent-others -array-add '{tile-data={}; tile-type="spacer-tile";}'

Again, killall Dock to see your changes. Once the spacer is in the Dock, you can drag it around to move it. You can even drag it between the application and document sections, if you wish. Run either command multiple times to add multiple separators.

Source:→ Mac OS X Hints