To enable pf insert the following lines in your kernel configuration and compile the kernel:

# needed for new packetfilter pf
device          pf                      # required
device          pflog           # optional
device          pfsync          # optional

# enable QoS from pf
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
#options         ALTQ_NOPCC      # Required for SMP build

tcpdump -n -e -ttt -i pflog0
tcpdump -A -s 256 -n -e -ttt -i pflog0

pfctl -sr

Install:

security/bruteforeceblocker (requires pf as the firewall)
or
security/denyhosts (uses tcp_wrappers and /etc/hosts.allow)
or
security/sshit (requires ipfw as firewall)

or http://www.pjkh.com/wiki/ssh_monitor

Enable pf in rc.conf:

# enable pf
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

Edit /etc/pf.conf:

ext_if = "em0"
set block-policy drop
# define table
table <ssh-bruteforce> persist file "/var/db/ssh-blacklist"

# block ssh known brute force
block log quick from <ssh-bruteforce>

# move brute force to block table
pass on $ext_if inet proto tcp from any to $ext_if port ssh keep state \\
 (max-src-conn 10, max-src-conn-rate 5/60, overload <ssh-bruteforce> flush global)

Create the blacklist file:

touch /var/db/ssh-blacklist
chmod 644 /var/db/ssh-blacklist

Restart pf with:

/etc/rc.d/pf restart
/etc/rc.d/pflog restart

http://www.daemonsecurity.com/pub/src/tools/cc-cidr.pl

To reduce priority for traffic:

altq on $ext_if cbq bandwidth 10Mb queue { def, mostofmybandwidth, notalot }
     queue def bandwidth 20% cbq(default borrow red)
     queue mostofmybandwidth 77% cbq(default borrow red) { most_lowdelay, most_bulk }
     queue most_lowdelay priority 7
     queue most_bulk priority 7
     queue notalot 3% cbq
[...]
block all
pass from $localnet to any port $allowedports keep state queue mostofmybandwidth
pass from $iptostarve to any port $allowedports keep state queue notalot

Example:

altq on $ext_if cbq bandwidth 100Kb queue { std, ssh }
queue std bandwidth 90% cbq(default)
queue ssh bandwidth 10% cbq(borrow red)

pass on $ext_if inet proto tcp from any to $ext_if port ssh keep state \
 (max-src-conn 10, max-src-conn-rate 5/60, overload <ssh-bruteforce> flush global) \
 queue ssh

pass out on $ext_if from any to any queue std

To see the live shaping:

pfctl -vvsq