Export and Import Content Filter Words or Phrases

In Exchange 2010 and Exchange 2007, you can add custom words or phrases as good or bad words to modify the Spam Confidence Level (SCL) assigned to messages. Messages with a good word or phrase are assigned an SCL of 0 and bypass other antispam agents that fire after the Content Filtering agent. Messages with a bad word are assigned an SCL of 9, and any configured action (delete/reject/quarantine) is taken based on the Content Filtering configuration.


Figure 1: Adding a custom word or phrase to Content Filtering configuration

To add a good or bad phrase to the custom words list using the EMC:

1. Go to Organization Configuration | Hub Transport | Anti-spam tab
   
2. Select Content Filtering and click Properties in the action pane
   
3. In Content Filtering Properties, select the Custom Words tab
   
4. Add a word or phrase in the following fields as required:
   • Messages containing these words or phrases will not be blocked:To add a good word or phrase, type it in this field
     
   • Messages containing these words or phrases will be blocked, unless the message contains a word or phrase from the list above: To add a bad word or phrase, type it in this field.
   
   

To add a word or phrase using the shell, besides the actual word or phrase, you must also specify the influence:

Add-ContentFilterPhrase "bad word" -Influence Badword

You can get a list of words or phrases added to Exchange by using the Get-ContentFilterPhrase cmdlet:

Get-ContentFilterPhrase | Select phrase,influence

Exporting and Importing Custom Words and Phrases
On the Edge Transport server, configuration information is stored in the local instance of Active Directory Application Mode (ADAM) on Windows Server 2003. In Windows Server 2008, ADAM is renamed to Active Directory Lightweight Directory Service (ADLDS). Unlike Exchange Server configuration information stored in Active Directory, which is replicated to all domain controllers in the AD forest, Edge Transport configuration information stored in ADAM/ADLDS is not replicated to other Edge Transport servers.

You can configure an Edge Transport server using a cloned configuration. See Using Edge Transport Server Cloned Configuration.

You can also export only the content filter phrases from one Edge Transport and import it to another Edge Transport server. To export the phrases, use the Get-ContentFilterPhrase cmdlet:

Get-ContentFilterPhrase | Select Phrase,Influence | Export-CSV "C:\MyFolder\CFPhrases.txt"

To import the phrases on another Edge Transport server, use the Add-ContentFilterPhrase cmdlet:

Import-Csv "C:\MyFolder\CFPhrases.txt" | foreach {Add-ContentFilterPhrase -Phrase $_.phrase -Influence $_.influence}

Troubleshooting Exchange Server Antispam: Disable Outlook Junk Mail

When troubleshooting antispam issues, particularly false positives (legitimate email incorrectly tagged as spam), frequently you run into scenarios where Exchange Server antispam features seem to be working correctly but you still see messages being delivered to the Junk Mail folder instead of the Inbox.

For instance, you whitelist a sender or sender domain, or add the sending IP address to the IP Allow List. You find message(s) from the whitelisted sender, domain or IP address still being delivered to the Junk Mail folder. You open the message and check the antispam headers - as expected, Exchange has stamped the message with a SCL of -1.

When troubleshooting Exchange antispam issues, it's best to turn off Microsoft Outlook's own antispam filtering. This is something Outlook does in addition to Exchange's server-side antispam agents. By default, Outlook's Junk E-mail Filter is turned on and set to Low.

To disable Junk Mail filtering in Outlook 2007:
1. Go to Tools menu | Options | Preferences tab | under E-mail section -> click the Junk Mail button.
2. In Junk E-mail Options, on the Options tab, select No Automatic Filtering...


Figure 1: Disable Outlook's own Junk E-mail Filtering when troubleshooting Exchange server antispam features

Related posts and links:

• About the Junk E-mail Filter
  

Disable Antispam agents on a Receive Connector

Exchange 2007's antispam agents are enabled for all Receive Connectors on a transport server. Is there a way to disable the agents on a particular Receive Connector?

Although not as simple as turning an agent off for each IP address or Receive Connector, Exchange 2007's new transport permissions model allows you to do this just as easily.

The ms-Exch-Bypass-Anti-Spam permission is what allows a sender to bypass antispam agents. By default, mail from authenticated senders is not filtered.

To allow unauthenticated/anonymous senders to bypass antispam filters on a particular Receive Connector, use the following command:

Get-ReceiveConnector "My Receive Connector" | Add-ADPermission -User "NT Authority\Anonymous Logon" -AccessRights ExtendedRight -ExtendedRights ms-exch-bypass-anti-spam

Related Posts and Links:

• Removing internal host names and IP addresses from message headers
  
• Why Get-TransportAgent doesn't agree with the Exchange console
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• Exchange 2007 Content FIlter: The Whitelist Is Here!
  
• Exchange 2007 Transport Permissions Model
  

Quarantine Mailbox: Displaying original SCL, sender and recipients

In Exchange Server 2007, messages delivered to the quarantine mailbox show up as DSNs sent by the postmaster address of the default domain. In HOW TO: Expose original senders and recipients of quarantined messages, we modified the QTNE.cfg form for Microsoft Outlook to reveal original senders and recipients.

Although the original sender and recipient fields were added, the original SCL stamped on the quarantined message wasn't visible. The OriginalScl property was exposed in Exchange 2007 SP1, and is now included in the updated form in that post. Installing the updated form exposes the original SCL for messages in the quarantine mailbox, as seen in Figure 1.


Figure 1: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form

Exchange Server Antispam Updates: Do I have the latest update?

Not sure if the Exchange 2007 or 2003 (IMF) updates on your system are the latest and greatest? Doubt if the automatic update process is working?

You can use the Microsoft Update Catalog web site to search for these. You can also subscribe to the RSS feeds for each update (search result on the web site).

• Exchange Server 2007 Antispam updates  
  
• Exchange 2003 Intelligent Message Filter updates  
  

You can also use the site to search for other Microsoft products and subscribe to the feeds.

PowerShell: Listing multi-valued attributes

In previous posts, we've taken a look at how to update multi-valued attributes and remove values from multi-valued attributes using PowerShell/Exchange Shell (EMS).

Multi-valued attributes have a special significance in AD, and interfaces/APIs used to access AD. Whereas single-valued attributes can be retrieved and updated quite easily, multi-valued attributes come with a twist. Values from a multi-valued attribute are returned as an array (of values). To evaluate values in a multi-valued attribute, you need to iterate through each one (using a foreach loop in most cases). Similarly, when updating a multi-valued attribute, we need to remember we're adding/updating one value of what could possibly be multiple items in an array.

With that out of the way, a real-word experience relates to how these values are listed in Exchange shell cmdlet output. For instance, the BypassedSenders property of ContentFilterConfig may have a few dozen safe senders that you do not want to subject to the Content Filter. If you list these bypassed senders using Get-ContentFilterConfig, the output will list a few bypassed senders. Note the trailing dots to indicate there are more values?

Using a format-list or fl (Get-ContentFilterConfig |select BypassedSenders | fl) doesn't help.


Figure 1: Output from Exchange shell cmdlets does not list all values in multi-valued attributes

BypassedSenders and Safelist Aggregation

The Content Filter Agent does not filter messages from addresses on its BypassedSenders property, regardless of the recipient. This should not be confused with a recipient's Safe Senders list (used by the Safelist Aggregation feature) to bypass mail for a recipient from the senders he/she adds to Safe Senders list in Microsoft Outlook. CFA's BypassedSenders is global in scope.

To get a list of all values in a multi-valued attribute such as BypassedSenders:

$senders = (Get-ContentFilterConfig).BypassedSenders; $senders

Alternatively, you can list them without adding them to a hash table ($senders in above example):

(Get-ContentFilterConfig).BypassedSenders

Figure 2: Listing all values in BypassedSenders multi-valued attribute

Similarly, multiple IP addresses or address ranges in a Receive Connector's RemoteIPRanges property:

(Get-ReceiveConnector "MyConnector").RemoteIPRanges

or formatted as a table with the required info:

(Get-ReceiveConnector "MyConnector").RemoteIPRanges | ft Lowerbound,Upperbound,RangeFormat -AutoSize

Figure 3: Listing all values in RemoteIPRanges multi-valued attribute of a Receive Connector

Related posts:

• HOW TO Update multi-valued attributes in PowerShell
  
• HOW TO: Remove values from multi-valued properties using Exchange shell
  

Quick antispam report or status check?

Having received an annoyingly higher proportion of spam in my Inbox this morning, I wanted to quickly check what the antispam agents are doing. Here's a quick cmdlet (besides the ones to check whether the antispam agents are enabled, checking the Content Filter SCL thresholds, etc.).

Get-AgentLog -StartDate "6/16/2008" | group action | ft name,count -Autosize

What you get back:

Name Count
---- -----
RejectCommand 520
AcceptMessage 39
RejectMessage 163
QuarantineMessage 11
DeleteMessage 21

The filters are still working. Perhaps it's one of those days when you wake up to high volume of spam.

Note to self: Create a quick monitoring script that provides more information from agent logs, antispam configs, and perfmon counters.

Related posts:
- Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007
- Exchange Server 2007: How are RBLs performing?
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

Assigning SCL to messages scanned by 3rd-party antispam filters

Here's a problem I had a hard time resolving on Exchange Server 2003. Exchange Server 2007's Transport Rules resolve this within minutes.

Pretend you're taking a Microsoft exam:
Scenario: "You are the Exchange administrator for your organization... ".

Exchange has the Content Filter Agent (CFA), and the Edge Transport Server role designed to be a non-domain-joined mail gateway, located in perimeter networks. However, the Unix/Linux/Security folks in your organization don't trust Exchange to do the filtering (or act as the mail gateway). They insist on using open source anti-spam software, such as SpamAssasin (yes Eric, I remember CRM114 - the Controllable Regex Mutilator... and all the cool stuff you can command it to do. Just to set the record straight, I was suitably impressed back then, and continue to be so till this day.. :) on the non-Exchange SMTP gateways. After tweaking it for a number of weeks, they are able to make it work the way they want it to, or are close to it. It's implementation time!

Their solution is to insert an X-header in messages that looks like this:

X-Spam-Status:yes

That's it. Their job ends there.

As the Exchange team/administrator, your job is to ensure messages with that header end up in users' Junk Mail folder.

Exchange Server 2003 did not provide any way, out-of-the-box, to be able to inspect message headers and stamp SCL. To add some contextual fun - back then, our "solution" was a doc that showed the end-users how to create an Outlook rule to move such messages to Junk Mail. Only if Exchange/Windows admins could code... I can hear you think.

 Creating a Transport Rule: Exchange Server 2007's Transport Rules functionality allows you to accomplish this easily. Here's how:
1. Fire up EMC | Organization Config | Hub Transport | Transport Rules tab
2. Click on New Transport Rule in the Action pane
3. Give the new rule a name, add a comment if you wish
4. In the Conditions page, select the condition when a message header contains specific words
5. In the Step 2 edit box, click on the message header link

6. Type X-Spam-Status | click OK
7. In the edit box, click on the specific words link
8. Type yes | click OK | click Next
9. In the Actions page, select the action set the spam confidence level to value
10. In the rule description, click on the 0 link and add a value that's above your SCLJunkThreshold | click Next
11. On the Exceptions page, click Next if you do not want any exceptions to this rule
12. Click New | click Finish to close the wizard

Or you can use the following commands:

$condition = Get-TransportRulePredicate HeaderContains
$condition.MessageHeader = "X-Spam-Status"
$condition.words = @("yes")
$action = Get-TransportRuleAction SetSCL
$action.SCLValue = 5
new-TransportRule "Stamp SCL" -condition @($condition) -action @($action)

 Disabling the Content Filter agent: Since you have a 3rd-party filtering solution running on your non-Exchange SMTP host(s), you can disable the Content Filter Agent. Messages exceeding SCLJunkThreshold will still be moved to Junk Mail folder.

Disable-TransportAgent "Content Filter Agent"

Alternatively, you can leave the CFA enabled, but disable the Delete, Reject and Quarantine actions.

Set-ContentFilterConfig -SCLDeleteEnabled $false -SCLRejectEnabled $false -SCLQuarantineEnabled $false

Send a test message with the X-header X-Spam-Status:yes. The message has the SCL value set by the Transport Rule. If it is above the SCLJunkThreshold, it should be delivered to the Junk Mail folder.

Why this doesn't work with Delete, Reject or Quarantine thresholds enabled?

The Content Filter Agent fires on the EndOfData SMTP event. On Hub Transport servers, the Transport Rules Agent fires on the OnRoutedMessage event. If the CFA is left enabled, along with any of the above actions, messages may get deleted, rejected or quarantined. The Transport Rules Agent never gets to see deleted and rejected messages at all.

What about quarantined messages?

It's understandable if the message is already deleted or rejected - there's nothing left for the Transport Rule agent to act on! However, why doesn't the agent act on quarantined messages? With Pipeline Tracing turned on, I spent some precious cycles trying to unravel the mystery, with little success. As it turned out (thanks to helpful Exchange team folks for pointing out... ), I was overlooking the obvious - Quarantined messages are wrapped in an NDR "envelope". If you've ever logged into the quarantine mailbox and looked at quarantined messages, you know what these look like. (Related post: "HOW TO: Expose original senders and recipients of quarantined messages") The Transport Rules Agent does not see the original message headers, including our X-Spam-Status header. As a result, the rule never fires.

Life on the Edge

If using an Edge Transport server in the mix, things change. The Edge Rule agent fires on the EndOfData event, unlike Hub Transport servers. When two transport agents fire on the same SMTP event, you can set the priorities of each accordingly so one fires before the other. This is described in "How to Make the SCL Value Available to Edge Transport Rules" in Exchange Server 2007 documentation.

However, it's a lot simpler to just disable the Content Filter Agent. This meets the requirements of this scenario, where anti-spam filtering is done by 3rd-party anti-spam filters and Exchange isn't required to do any filtering at all.

Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007

One of the more useful improvements in Exchange Server 2007 is the abundance of logging for different features and components (read previous post "Exchange Server 2007: How many logs hath thee?"). In particular, the antispam agent logs fill an important gap in monitoring, reporting and troubleshooting message flow as it relates to antispam agents (read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"). As a messaging/Exchange administrator, you want to be able to pin point what the antispam agents have been up to, and determine if particular messsages, or sending hosts, domains or email addresses have been blocked by any of the antispam agents. The antispam agent logs and the get-agentlog command allow you to do this quickly and efficiently.

In addition to the the get-agentlog command, Exchange Server 2007 also ships with a number of canned scripts that help you keep tabs on what the agents are doing. These scripts are found in the \Exchange Server\Scripts directory, where \Exchange Server is the path of the Exchange Server 2007 installation. Note, no documentation or support is available for these scripts - they are meant to be examples you can use to write your own scripts. Let's take a peek in the directory and see what we find.

Commandline parameters used by antispam scripts

Most of the following scripts take the same (optional) parameters.
-top n: Where n is the number of results to display. If not specified, the script defaults to (top) 10.
-StartDate: Start date/time
-EndDate: End date/time
-location: path of agent log files. If no path is specified, the agent works against the default agent log file location.

1. Get-AntispamFilteringReport.ps1: Takes one of the following values as a mandatory parameter: 1) connections 2) commands 3) messagesrejected 4) messagesdeleted 5) messagesquarantined, and displays statistics for each agent.
2. Get-AntispamSCLHistogram.ps1: Provides a breakdown of number of messages stamped with each SCL value.
3. Get-AntispamTopBlockedSenderDomains.ps1: Lists top ten sender domains from which mail was blocked.
4. Get-AntispamTopBlockedSenderIPs.ps1: Lists the top ten IP addresses blocked by antispam agents, and number of messages blocked from each.
5. Get-AntispamTopBlockedSenders.ps1: Lists the top ten blocked senders (SMTP email addresses) and number of messages blocked from each. The script can report on P1 (i.e. address in message envelope in the MAIL header) or P2 addresses (from headers in message body like FROM). Specify the option as a commandline argument (optional parameters -top n, -StartDate, and EndDate used in this example):

.\Get-AntispamTopBlockedSenders.ps1 P1 -top 20 -StartDate "12/1/2007" -EndDate "12/10/2007"

Replace the P1 in the above command with P2 to report on P2 senders.
6. Get-AntispamTopRBLProviders.ps1: Lists the top ten RBLs (aka "IP Block List Providers") and messages blocked by each (read previous post: "Exchange Server 2007: How are RBLs performing?").
7. Get-AntispamTopRecipients.ps1: Lists the top ten recipient addresses that receive spam. The addresses may or may not exist in your Organization - the reporting is based on actions taken by antispam agents on incoming messages. Recipient Filtering with Recipient Validation (dropping messages for recipients that do not exist in AD/GAL) is a great way to drop a large number of messages.

The following scripts are not used for reporting:
8. Reset-AntispamUpdates.ps1: Uninstalls any antispam updates and reinstalls the original "out-of-box" antispam data.
9. Install-AntispamAgents.ps1: By default, antispam agents are not installed on the Hub Transport server role (read previous post: "HOW TO: Install anti-spam agents on Hub Transport server"). This script installs them if you need to.
10. Uninstall-AntispamAgents.ps1: Uninstalls antispam agents.

Connection Filtering, RBLs and SMTP logs

Exchange Server 2003's Connection Filtering feature allows you to block connections from IP addresses explicitly added to the Global Deny List, or drop messages from IP addresses listed on a RBL (Real-Time Blackhole List / Real-Time Block List).

Note: The term "RBL" is commonly used to describe DNS Black Lists (DNSBLs), but it's a trademark of Mail Abuse Prevention System (MAPS), one of the first RBLs. It was acquired by Trend Micro in 2005. Exchange Server 2007 uses the term IP Block List Provider.

When connections from IP addresses listed on the Global Deny List are dropped, you don't find any traces of those dropped connections in SMTP logs [for more details about SMTP logging in Exchange Server 2003, read previous post "Logging SMTP protocol activity"].This is understandable - the IP addresses are explicitly added to the deny list by an administrator, and a SMTP session is never initiated. It's not as important to log these (though many would prefer to capture information about connections so blocked). If in doubt about which hosts are getting blocked, all one needs to do is look at this list.

When it comes to DNSBLs, the behavior is different. Connections are not dropped. Messages are dropped on RCPT TO: command. This allows you to receive mail for addresses on the Exception List. RFC 2821 requires senders to be able to contact the postmaster of a domain. You can also add some of the generic addresses from your domain to the Exception List if you don't want email to these addresses blocked.

Here's a demo of the SMTP session. As seen in the session, the sending host gets a 550 5.7.1 error code because the IP address is listed on a DNSBL, and the recipient (in the RCPT TO: command) is not listed on the Exception List.

Here's what the SMTP log shows (some irrelevant fields removed):

2007-07-25 19:04:54 172.31.1.10 0 HELO - - 250 0 49 4 0 SMTP - - - -
2007-07-25 19:04:59 172.31.1.10 0 MAIL - +from:foo@mydomain.com 250 0 41 26 50 SMTP - - - -
2007-07-25 19:05:11 172.31.1.10 0 RCPT - +to:jadams@exchangelabs.net 550 0 0 31 5017 SMTP - - - -
2007-07-25 19:05:15 172.31.1.10 0 QUIT - - 240 22943 72 4 0 SMTP - - - -

SMTP logs the response code 550, but not the enhanced status code 5.7.1. Having said that, this enhanced status code doesn't by itself allow you to narrow it down to DNSBLs.

Exchange Server 2007's Connection Filtering agent behaves differently - connections from IP addresses explicitly listed in the IP Block List (equivalent of the Global Deny List in Exchange Server 2003) are not blocked. This change in behavior allows such connections to be logged. Detailed information is available from the agent logs [read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"] about why a message was accepted/rejected by anti-spam agents.

One example of the kind of reporting/troubleshooting capabilities this provides: you can search the agent logs for messages blocked by Connection Filtering agent based on DNSBL lookups:

Get-AgentLog -StartDate "7/27/2007" | where {$_.Reason -eq "BlockListProvider"}

The Get-AgentLog output:

Timestamp : 7/27/2007 6:42:08 AM
SessionId : 08C99DA727C8EF31
IPAddress : 68.45.233.59
MessageId :
P1FromAddress : dbuuxwxjlm@bobach.com
P2FromAddresses : {}
Recipients : {bf5b7b59@mydomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on
a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

SCRIPT: Turning on Filter Junk Email

OWA users who never logon using Outlook do not have their Junk Mail filtering option turned on by default. Exchange MVP Glen Scales has a script here that allows you to turn this on programmatically for Exchange Server 2007 users.

Why Get-TransportAgent doesn't agree with the Exchange console

You disable a particular anti-spam agent - let's say the Content Filtering Agent - from the Exchange console.



Next, you use the Get-TransportAgent command to get the status of transport agents - and surprisingly the Content Filter Agent shows up as Enabled!



Why doesn't Get-TransportAgent agree with the Exchange console about the status of an agent?

Fellow MCT and Exchange geek David Elfassy blogged about it recently.

The Anti-Spam tab is located under the Organization Configuration | Hub Transport node in the console. Disabling an agent from the console prevents the agent on all transport servers in the Organization from taking any action. It's an Organization-wide or global setting.

The Get-TransportAgent command is transport server-specific. It shows the status of agents on a particular server. With the Org-wide configuration set to disabled, an agent enabled on a particular transport server behaves like a soldier instructed to fire but not kill. (May not be the best analogy - it doesn't exactly scare spam, and it doesn't do much else).

To draw a parallel to Exchange Server 2003, you can enable settings on a particular filter - e.g. Recipient Filtering, in Global Settings. However, you still have the flexibility of being able to enable/disable it on a per-SMTP virtual server basis. This is somewhat similar.

To get the global/Org-wide status of a particular filter, you need to use the corresponding command for that filter.
- For Content Filter Agent: Get-ContentFilterConfig
- For Recipient Filter Agent: Get-RecipientFilterConfig
- For Sender Filter Agent: Get-SenderFilterConfig
- For SenderID Agent: Get-SenderIDConfig
- For Protocol Analysis Agent: Get-SenderReputationConfig

If you've carefully looked at the above list, you're probably wondering why there's no mention of Connection Filtering Agent in it. The Connection Filtering Agent handles multiple tasks which are listed individually in the console.

To check the global setting for configuration of each, use the corresponding config command:
- For IP Block List: Get-IPBlockListConfig
- For IP Block List Providers: Get-IPBlockListProvidersConfig
- For IP Allow List: Get-IPAllowListConfig
- For IP Allow List Providers: Get-IPAllowListProvidersConfig

To "disable" an agent in Organization Configuration (equivalent of what you do in the console) using the shell, use the corresponding Set command, e.g.:

Set-ContentFilterConfig -Enabled $false

Next, let's take a look at how you can disable an agent on a transport server. Generally, if the Get-Blah command returns a property like Enabled: True, you can use Set-Blah -Enabled $false to disable it (You probably know where we're going with this by now... :). Not so for Get-TransportAgent. Its "Set" counter-part - Set-TransportAgent allows you to modify only one property -Priority. You can change the order in which these agents fire by changing their Priority.

You use the Disable-TransportAgent command to disable an agent on a transport server:

Disable-TransportAgent "Connection Filtering Agent"

No points for guessing the command used to enable a transport agent - Enable-TransportAgent

Header Firewall: Why spammers can't insert fake SCL (and other Exchange Organization) X headers

After almost a year and a half of working with Exchange Server 2007, I still stumble into features I haven't tested before. When it happens, it blows holes into my theories about how well I know E12. :)

It started when investigating why a certain message - likely not from an authenticated source, would have a SCL of -1 (seen in message headers as X-MS-Exchange-Organization-SCL: -1).

One suggestion that got me thinking: it may have been a fake header, perhaps inserted by a spammer. I tested this a few times in different combinations (by inserting SCL and PCL headers before, after and in between other X-headers, like the ones below:


Note: Exchange uses the PCL and SCL headers seen above, but the QCL header in the above screenshot is fabricated/fake.

When the above messages came in, all the headers with X-MS-Exchange-Organization* were removed by Exchange. Nice work, Microsoft, I said to myself, and asked for an explanation.



Someone from the team immediately pointed me to Header Firewall - a supercool feature (not so much in what it does but the fact that it exists). It removes protected Exchange Organization and Forest headers from messages that enter or exit the Exchange Organization. This is the reason you're not likely to see fake SCL, PCL, SenderID, and other such headers in messages received by Exchange Server 2007 Edge or Hub Transport servers.

Preserving Organization and Forest X headers: If you want these to be preserved for inbound mail from particular sources, it is recommended to do so using a separate Receive Connector, just as you would create one if you needed to allow unauthenticated relaying [read previous post "Exchange Server 2007: How To Allow Relaying"], and assign MS-Exch-Accept-Headers-Organization and MS-Exch-Accept-Headers-Forest permissions. Similarly, to send Organization and Forest headers to particular external mail systems, it is recommended to create a separate Send Connector for those address spaces, and assign the MS-Exch-Send-Headers-Organization and MS-Exch-Send-Headers-Forest permissions.

Getting back to the original question - when do you see a SCL of -1? Either when the sender is authenticated, or the sending host is on the IP Allow List.

Read more about the Header Firewall feature in the Exchange Server 2007 documentation. A list of Exchange Organization and Forest x-headers can also be found there.

Moving items to Junk Mail folder doesn't do much

Scenario: Junk mail makes it to your Inbox, without getting filtered by any of Exchange's anti-spam filters.
Action: User moves item to Junk Mail folder.
Result: This accomplishes little more than having the piece of UCE moved out of the Inbox. Contrary to what many may think and like to believe, moving a message manually to the Junk Mail folder doesn't really have Outlook or Exchange do anything at all with that item. No, the sender or sending domain or host do not get blacklisted or added to Microsoft Outlook's Blocked Senders List. Yes, you're not likely to see a similar message from same sender, domain or host get treated any differently based on that action.

To make Microsoft Outlook filter messages from the same sender, the sender needs to be added to the Blocked Senders list manually (by right-clicking message | Junk E-mail | Add Sender to Blocked Senders List).



Does the Blocked Senders List make it to the Exchange Hub or Edge server using the Safelist Aggregation feature? No. Safelist Aggregation is only used for aggregating the safe list, as the name of this feature correctly suggests. The Blocked Senders List is only used by Microsoft Outlook, without any server-side action being taken on messages from senders on this list.

Related Posts:
- Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder
- TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007
- Exchange 2007 Content Filter: The Whitelist Is Here!
- HOW TO: Expose original senders and recipients of quarantined messages
- Connection Filtering, RBLs and SMTP logs

Exchange Server 2007: Can the quarantine mailbox be an external smtp address?

In Tuesday's Birds of a Feather (BOF) session on Fighting Spam with Exchange Server 2007, someone from the audience wanted to know if an external (smtp) email address can be used as a quarantine mailbox. Exchange does no validation of the email address used for a quarantine mailbox - as long as it's a valid smtp address, it can be used.

However, the results may not be what one's looking for. The quarantined message appears as a NDR. Outlook knows how to treat it correctly using the Send Again functionality, which won't happen if you're forwarding quarantined messages to an external address. The From headers will not be rewritten to identify the original sender of the message.

So yes, it can be done - but what's the point?

TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007

Tuesday night was a Birds of a Feather (BOF) night, with plenty of BOF sessions and chalk talks scheduled. My session on Fighting Spam With Exchange Server 2007 (not a very creative title, I know... but it got the message across :) was scheduled for 7:45 PM. I was positive I'll probably be the only person in the room - 7:45 PM isn't exactly the time I'd want to attend a TechEd session on anything, with the exception of a few good speakers.

On a side note, keeping my expectations low also meant I could claim the attendance was more than I expected even if only one person showed up. Nevertheless, the room filled up as we got closer to the start time.

It was an interesting discussion on anti-spam in Exchange Server 2007, as well as Exchange 2003.

Some takeaways:

• Exchange's anti-spam features consist of more than just Intelligent Message Filter (IMF) - there's 1) Connection Filter: allows using IP Allow Lists, IP Block Lists, IP Block List Providers (RBLs) and IP Allow List Providers 2) Sender Filter: block sending domain or SMTP address 3) Recipient Filter: block internet messages to list of valid recipients, perform Recipient Validation to block messages for non-existent recipients 4) Sender ID Filter: looks up SPF records in purported sending domain's DNS zone, protects from address spoofing and phishing 5) Content Filter (aka "IMF"): Blocks messages based on content
• 3 Gateway Thresholds and Actions: Exchange Server 2007's Content Filter Agent supports 3 gateway thresholds and actions - 1) Delete 2) Reject and 3) Quarantine
• Quarantine functionality: Though the quarantine functionality in Exchange Server 2007 is a step forward compared to Exchange Server 2003's archiving feature, it's not as full-featured as that offered by many third-party products. End-users have no control over the quarantine - an administrator has to inspect quarantined messages and make the decision to deliver or delete. However, it's important to understand that a) it's not compulsory to use the quarantine and b) The end-user mechanism for accessing suspicious messages is the Junk Mail folder c) Quarantining is one more layer available to admins - between the gateway actions of Rejecting or Deleting mail and the Store action of delivering to Junk Mail
• Exchange Server 2007's Connection Filter agent does not immediately drop connections from IP addresses explicitly listed on the IP Block List. Exchange Server 2003 does this for IPs listed in its Global Deny List. This change in behaviour allows such "filtered" connections to be logged in the agent log. Some customers like to see these logged, whereas many want connections to be dropped immediately.
• Many customers prefer anti-spam appliances like Barracuda and IronPort in the perimeter, and consider these very effective, and generally lower cost compared to an Exchange 2007 Edge Transport server.
• No studies or test results are available yet to demonstrate how the Edge Transport server performs compared to such appliances. It would be great to have a feature and performance comparison of these.
• Many customers are interested in Greylisting - a technique that registers each connection's "triplet" of a) Sending host's IP address b) Sender's address c) Recipient's address and looks up a database. If not found, the connection is immediately dropped and the triplet added to the database. The technique is based on the idea that valid RFC-compliant SMTP hosts will attempt to resend the message after a set interval. The (receiving) SMTP server will find the triplet in the database upon the second attempt, and consequently accept the connection.
• Interested in learning more about Greylisting? Evan Harris has a whitepaper titled "The Next Step in Spam Control: Greylisting"
• Exchange Server 2007 SDK has the code for a Greylisting agent. This needs to be compiled into a DLL using Visual Studio 2005, and installed using the accompanying installation script. Since this is just a programming sample provided in the SDK, it makes sense to test this adequately in a test environment and not expect any production support
• Freeware Greylisting transport sinks/add-ins for Exchange Server 2003 are available on the web, like this one from Chris on Grynx.com
• Over a period of time, the effectiveness of techniques like Greylisting may decrease, but as of now it still eliminates 80-90% of spam.
• SMTP Tarpitting is a technique that inserts a delay in SMTP responses to SMTP clients that display suspicious protocol behavior. More details about tarpitting in a previous post titled "Enabled by default: SMTP Tarpit in Exchange Server 2007"
• Safelist Aggregation: This allows Exchange to aggregate senders marked as safe by (Outlook/OWA) users. If an EdgeSync subscription exists (in topologies with an Edge server), the safe list is replicated to the Edge as part of EdgeSync. Exchange's use of safe lists can reduce false positives significantly. It's not required to have an Edge server - Hub Transport servers with anti-spam agents installed can also use this feature.
  

TechEd 2007: Let's talk about fighting spam with Exchange Server 2007

If you're going to be at Microsoft TechEd 2007 in Orlando next week, and are interested in sharing your experiences in fighting spam using Exchange Server's anti-spam features, here's a Birds-of-a-feather (BOF) session just for you.

What: BOF session "Fighting Spam With Exchange Server 2007"
When: Tuesday, June 5th @ 7:45 PM
Where: Microsoft TechEd, Orlando, FL
What's a BOF session? : Birds-of-a-feather sessions are not presentations or lectures. These are meant to be interactive discussions, sans the PowerPoint slides and demos you typically see in breakout sessions.
Why so late?: Tuesday night is an all-BOF session night, with no other conference activities. Microsoft will be serving food and drinks.

Abstract:
“Exchange Server 2007 anti-spam features are a significant improvement over previous versions, and come with interesting deployment and topology questions. Are these effective in the fight against spam? This BOF session will be a round table for Exchange administrators to talk about their experiences with Exchange Server 2007 anti-spam features, including issues such as Edge servers, use of IP Block List Providers, and new features and techniques such as Sender Reputation, and Safelist Aggregation.”
Audience: IT Professional

Grimes Gripes About RBLs

InfoWorld columnist Roger Grimes gripes about Real-Time Block Lists (aka Real-Time Blackhole Lists, DNSBLs or DNS-based Block Lists. The Exchange Server 2007 term for RBLs is "IP Block List Providers").

The feelings towards RBLs are understandable - you won't be a fan of RBLs if your IP addresses are the ones getting listed, and your users breathing down your neck about bounced messages. It's important to monitor RBLs to determine if your IP addresses are listed in any. (Incidentally, Zenprise for Exchange does this automatically several times a day - configurable by the administrator - and alerts you if any of your IP addresses are listed on RBLs. Your external IP addresses are detected automatically based on messages sent to Zenprise's Zmail hosted service).

Once listed, some RBL providers make the process of getting IP addresses delisted a long and painful one. At times, and in spite of your best efforts and assurances by the provider that the IP address(es) will be delisted, they continue to live on in such lists. On the other end of the spectrum are RBL providers that make the delisting process as simple as entering the listed IP address in a form on their web site, without any checks in place. I would shy away from using the latter.

Regardless, if you are on the other side, as a victim of spam rather than a victim of RBLs, good/reputed RBL providers like Spamhaus (which has its detractors as well - read previous post: "Spamhaus, the E360Insight lawsuit and the future of RBLs"), can help in dramatically reducing spam at the mail gateway - by as much as 90+ percent. This further reduces the stress on mechanisms like content filters (like Intelligent Message Filter or IMF in Exchange Server 2003, and the Content Filter agent in Exchange Server 2007).

Whether you use RBLs to block inbound internet mail from black-listed addresses or not, it makes sense to monitor whether your IP addresses are listed or not on a regular basis.

Read Grimes' column titled "Why I hate RBLs" on InfoWorld.com.

HOW TO: Expose original senders and recipients of quarantined messages

The Content Filter Agent in Exchange Server 2007 allows you to quarantine messages above the SCLQuarantineThreshold. Messages so quarantined are delivered to a quarantine mailbox, specified in the Content Filter configuration. [Read previous post for more info, "Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder"]

Messages arriving in the quarantine mailbox look like NDRs. They appear to be sent from the postmaster address of the default domain. As such, you can't easily see the original sender and recipient's addresses, as shown in the screenshot below:


Figure 1: By default, messages in the quarantine mailbox show the envelope sender— the postmaster address

"How to Recover Quarantined Messages from the Spam Quarantine Mailbox" in the Exchange Server 2007 documentation shows how to use a custom Outlook form to expose the original sender - the doc has code that can be copied to a text editor like Notepad and saved as a custom form (QTNE.CFG), which can be used in Outlook to expose the relevant MAPI property as a field that can be added to the view.

This exposes the original sender, but doesn't expose the original recipient of the message. To expose the recipient, you need to modify the form from the above link and add the relevant MAPI properties.

 Download the modified QTNE.CFG file
Download the modified form file and save it in the correct Forms folder for your locale.

For US English, the path is: \Program Files\Microsoft Office\Office12\Forms\1033. You can find out the locale ID (LCID) for your locale from KB 221435: List of supported locale identifiers in Word..

What was added in my Modified Quarantine Extension Form:
Under [Properties]

Property02=OriginalRecipient

After [Property.OriginalSender] section:

[Property.OriginalRecipient]
Type=30
NmidInteger=0x0E04
DisplayName=Original Recipient

 Install the form in Microsoft Outlook

1. Tools | Options | Other tab | Advanced Options | Custom Forms | Manage Forms | Install.
2. Navigate to the path where you saved the QTNE.CFG file, and select the file.


Figure 2: Installing the Modified Quarantine Extension form (QTNE.cfg) in Microsoft Outlook

3. Click OK a few times till all the dialog boxes are closed. The form is now installed.


 Add the fields
With the form installed in Outlook, it's time to make the fields show up!

1. In the Outlook messages pane, right-click the column heading on the top | select Custom.
2. In the Custom View dialog box, click Fields | from the "Select available fields from:" drop-down (which says "Frequently-used fields" by default), select Forms [screenshot].
3. Select the Modified Quarantine Extension Form | click Add
4. In the Show Fields dialog box, select Original Sender and Original Recipient | click Add.


Figure 3: Adding the Original Sender and Original Recipient fields to the view

5. Additionally, you can remove unnecessary fields and order the fields the way you want [screenshot of fields I have selected].

Once you're done, the Original Sender and Original Recipient fields are displayed as shown in the screenshot below:


Figure 4: The updated view after adding the Original Sender and Original Recipient columns

When the message list pane is expanded in Outlook, this is what it looks like.


Figure 5: Expanding the columns in the updated view

Exposing these fields makes managing the quarantine mailbox easier.

Update: Adding the Original SCL column
08/19/2008 7:39 PM

Exchange 2007 SP1 also exposes the original SCL. The Outlook form file (QTNE.cfg) can be updated by adding the following.

- In the [Properties] section:

Property03=OriginalSCL

- After the [Property.OriginalRecipient] section:

[Property.OriginalSCL]
Type=3
NmidPropset={41F28F13-83F4-4114-A584-EEDB5A6B0BFF}
NmidString=OriginalScl
DisplayName=Original SCL

You can download the updated QTNE.cfg form or manually updated your own file. Now you can add the Original SCL column.


Figure 6: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form.

The previous version of QTNE.cfg is still available.

Exchange Server 2007: Protect users from spam from your own domain

One of the common complaints from users and many messaging folks is spam received from senders that appear to be from your own domain. SMTP mail is exchanged with anonymous internet hosts without any authentication. Headers are effortlessly spoofed. Rather than use an unregistered or invalid domain in the From: field, many spammers find it attractive to use an address from your domain - even an invalid one, or worse - a perfectly valid one. (It is advisable to not have resolution of anonymous senders enabled. Read previous post "A Late New Year's Resolution: Do Not Resolve Anonymous Senders")

Messages that appear to be sent from your domain probably have a better shot at making it through some spam filters, and certainly have a higher chance of being opened by recipients in your domain, once they do make it to the Inbox.

One way of tackling this menace is by using SenderID filter to drop messages that return a FAIL result (from a SPF record lookup). It's advisable to publish SPF records for your domain(s), if you haven't already done so. Explicitly designate SMTP hosts authorized to send for your domain(s). However, the impact of SenderID and whether you should consider dropping email based on it is an open debate, and the topic of another post.

In addition to the above, Exchange Server 2007 allows you to configure ReceiveConnectors to drop messages where the FROM address is from your own domain. It's a good idea to do so if you don't expect any legitimate messages sent without authentication (with your domain in the From: field). To configure a ReceiveConnector to reject such messages:

Remove-ADPermsission "ReceiveConnector-Name" –user “NT AUTHORITY\Anonymous Logon" –extendedrights “ms-Exch-SMTP-Accept-Authoritative-Domain-Sender”

Consider doing this only on the ReceiveConnector that receives inbound internet mail. There may be a need for certain - generally internal - senders like copiers/scanners, and other non-Exchange mail hosts to be able to send with an address from your domain in the FROM field, without authentication. As discussed in previous post titled "Exchange Server 2007: How To Allow Relaying", it is advisable to create an additional ReceiveConnector for such trusted/internal hosts.

Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

The anti-spam agents in Exchange Server 2007 log their actions in agent logs. By default, agent logs reside in \Exchange Server\TransportRoles\Logs\AgentLog. Each agent log is 10 Mb in size. The size of the directory is capped at 250 Mb, the age of a log to 30 days. (Logs are flushed when they "age" up to 30 days or if the directory size reaches 250 Mb - whichever happens first).

Additional agent log configuration parameters in SP1

Exchange Server 2007 RTM: In the RTM version, there's only one configuration option for the agent log - that to enable or disable it.

Exchange Server 2007 SP1: Additional parameters to control the max directory size, file size and age of agent logs added.

The config options available:
1) Enable/Disable agent log: Boolean value - TRUE/FALSE
2) Max directory size: In bytes. If not specified, the default is 250 Mb or 262144000 bytes.
3) Max file size: If not specified, the default is 10 Mb or 10485760 bytes.
4) Max age: If not specified, the default is 30 days.

The agent log configuration parameters can be controlled by editing the EdgeTransport.exe.config file, located in \Exchange Server\Bin folder on Edge and Hub Transport servers.

To disable agent logging, insert the following key under <appsettings></appsettings> in the config file:

<add key="AgentLogEnabled" value="FALSE" />

In the following example, we modify max directory size to 500 Mb, file size to 20 Mb, and age to 60 days, by creating new keys in EdgeTransport.exe.config:

<add key="AgentLogMaxDirectorySize" value="524288000" />
<add key="AgentLogMaxFileSize" value="20971520" />
<add key="AgentLogMaxAge" value="60.00:00:00" />

Parsing the agent log: You can parse the agent log using the Get-AgentLog command from the shell. By default, this parses the agent log residing in the default location. If you've copied a bunch of agent logs at an alternate location, you can specify the alternate location using the following syntax - in this example the agent logs have been copied to Z:\Antispam Agent Logs directory:

Get-AgentLog -location "Z:\AntiSpam Agent Logs"

Here's what an entry in the agent log looks like - note the different fields and their values:

Timestamp : 4/16/2007 12:39:49 AM
SessionId : 08C948C83FB951AC
IPAddress : 72.46.133.113
MessageId :
P1FromAddress : ret@noncornelan.com
P2FromAddresses : {}
Recipients : {foo@yourdomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

As seen in the above output, the logs provide adequate information for reporting on anti-spam activity, as well as for troubleshooting anti-spam issues like messages not being received/wrongly filtered out.

Getting to know the agent logs will make troubleshooting such issues much easier.

By default, the Get-AgentLog command returns all the entries in the agent logs. It can be constrained to a particular date and time - the recommended way to perform most agent log searches, unless you want to immerse yourself in 30 days (or 250 Mb) of anti-spam goodness! This is done using the -StartDate and -EndDate parameters:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007"

You can also constrain it further by adding time of the day:

Get-AgentLog -StartDate "4/17/2007 8:00 AM" -EndDate "4/17/2007 2:00 PM"

Though the Get-AgentLog command only takes these 3 parameters - location, StartDate, and EndDate, you can further filter the logs using most of its logged fields.

To filter the log to show messages to a particular recipient:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.recipients -like "foo@yourdomain.com"}

To search for messages from a particular sender:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "aqe@easymoney2u.com" -or $_.P2FromAddresses -like "aqe@easymoney2u.com"}

To search for messages from a particular domain:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "*somedomain.com" -or $_.P2FromAddress -like "*somedomain.com"}

To filter by the anti-spam agent that acted on a message, e.g. Connection Filtering Agent:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Agent -eq "Connection Filtering Agent"}

Similarly, you can filter by other agents that write to the agent logs: 1) Content Filter Agent 2) SenderID agent 3) Sender Filter agent 4) Recipient Filter agent and 5) Edge Rules agent.

To filter agent logs by the sending host's IP address, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.IPAddress -eq "72.46.133.113"}

The reason field in each log entry specifies the reason supplied by the anti-spam agent that takes the action. For instance, as seen in the agent log entry shown earlier in this article, the agent that acted on the message is the Connection Filtering Agent, the reason is BlockListProvider (i.e. "RBL" or "Real-time Block List", known as IP Block Lists in Exchange Server 2007). The ReasonData field gives you the name of the IP Block List Provider, as configured in Exchange. In the above agent log entry, it is "Spamhaus SBL-XBL". To constrain the search for messages blocked by IP Block List Providers:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Reason -eq "BlockListProvider"}

To get a list of all IP addresses blocked by IP Block List Providers:

Get-AgentLog -StartDate "12/21/2007" | where {$_.Reason -eq "BlockListProvider"} | ft Timestamp,IPAddress,ReasonData

You can also look for messages blocked by a particular IP Block List Povider:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.ReasonData -eq "Spamhaus SBL-XBL"}

For messages scanned by the Content Filter Agent, the Reason field contains details like SCLAtOrAboveDeleteThreshold, SCLAtOrAboveRejectThreshold, etc. The ReasonData field contains the SCL value assigned to the message. To get a list of messages above the SCLDeleteThreshold, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.reason -eq "SCLAtOrAboveDeleteThreshold"}

As shown in the above examples, you can use the Get-AgentLog command and pipe the data to filter it based on the fields logged. You can get more details about agent logs - including the fields logged, from the Managing Agent Logging section in Exchange Server 2007 documentation.


2/9/2008: Added sidebar about additional configuration parameters available in SP1.

HOW TO Update multi-valued attributes in PowerShell

Multivalued attributes are simply attributes that can have more than one value. Examples include members attribute of groups, or proxyAddresses attribute of recipients.

In GUI interfaces, it is simple to add/modify multivalued attributes. VBScript has control codes that allow you to control the interaction with multi-valued attributes - 1 = clear all entries, 2 = update all entries, 3 = append, and 4 = delete.

Exchange shell uses Add commands to add new entries to some multivalued attributes - e.g. Add-IPBlockListEntry to add another IP address or range to the IP Block List.

However, if you want to add more values to stuff like BypassedSenderDomains in ContentFilterConfig, there's no equivalent of an Add command. If you try to add a value to the attribute using the following syntax, the existing values will be replaced by the newer ones:

Set-ContentFilterConfig -BypassedSenderDomains "somedomain.com"

You could copy the existing values (e.g. microsoft.com,zenprise.com), and paste these along with the new value:

Set-ContentFilterConfig -BypassedSenderDomains "microsoft.com","zenprise.com","somedomain.com"

However, this copying and pasting can be tricky for attributes that have a lot of values. Another alternative (..thanks to Vivek for the suggestion):

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains +="somedomain.com"
$foo | Set-ContentFilterConfig

This is a great workaround... and it works!

However, it'd be much easier to have a switch that lets you append to such multi-valued attributes, in keeping with the shell's promise of allowing most operations using one-liners.. something like Set-ContentFilterConfig -BypassedSenderDomains +domainthree.com,domainfour.com to add values to the attribute, and some other combination to remove a particular value.

Update:
To remove a value from a multivalued attribute, for example from the BypassedSenderDomains property of the ContentFilterConfig used in the preceding example:

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains -="somedomain.com"
$foo | Set-ContentFilterConfig

Empower Your PowerShell with PowerGadgets

Exchange MVP Andy Webb showed off this great tool called PowerGadgets recently that can help you build cool gadgets - with or without Windows Vista. The gadgets take piped output from PowerShell commandlets and scripts (as well as SQL databases and web services).

Why I think this is so cool:
- it allows you to visualize data output by PowerShell commands and scripts. The samples shown here are simply displays, but you can create pretty fancy stuff like gauges and charts
- comparatively, it makes PerfMon - which is a useful tool, but so NT 4.0-ish - look like.. well, NT 4.0-ish :)
- critical or important data can be displayed in real-time or close-to-real-time on your desktop or Windows Vista sidebar

Andy created a gadget that takes the output from the get-antispamtoprblproviders.ps1 script [read previous post about this script - "Exchange Server 2007: How are RBLs performing?"]. The gadget runs the script and shows you number of messages blocked by the RBL provider Spamhaus.org (if you're using it with Exchange Server 2007's IP Blocklist Providers feature).

Yet another one of Andy's gadgets shows the Top 20 mailboxes by size. Very cool!

Thanks to input from Andy, and Vivek Sharma, I was able to effortlessly create a gadget that shows messages scanned by Exchange Server 2007's Content Filter agent.

I created another one that shows you number of messages dropped by Recipient Filter using Recipient Validation (drop messages for recipients that do not exist in AD).

Both the above gadgets use real-time output from perfmon counters captured by shell commands and refresh every minute.

You can create gadgets natively from within PowerShell, or use the PowerGadgets Creator GUI that walks you through creating one. Download PowerGadgets from www.powergadgets.com.

Here are the gadgets:
1. Andy's Spamhaus gadget
2. Andy's Top 20 Mailboxes gadget
3. Content Filter gadget - shows total messages scanned by Content Filter agent
4. Recipient Filter gadget - shows messages dropped because of invalid recipients

Update:
Right-click on the above links and save as/download.

If you've created a gadget using PowerShell, spread the word by adding a comment to this post.

Exchange 2007 Content Filter: How to move messages to Junk Mail folder

You've setup Exchange Server 2007, and configured the shiny new Content Filter agent (CFA), which is more than just a rewrite of the equally loved and hated Intelligent Message Filter (IMF) from Exchange Server 2003. The CFA has three thresholds - equivalent of Gateway thresholds and (gateway) actions in IMF. These are: 1) SCLDeleteThreshold 2) SCLRejectThreshold and 3) SCLQuarantineThreshold.

1. SCLDeleteThreshold: Messages with SCL equal to or higher than the SCLDeleteThreshold are deleted silently. To enable the SCLDeleteThreshold:

set-ContentFilterConfig -SCLDeleteThreshold 8 -SCLDeleteEnabled:$true

2. SCLRejectThreshold: Messages with SCL equal to or higher than the SCLRejectThreshold are rejected during the SMTP session, after the data is received. In this case, senders get a NDR. To enable the SCLRejectThreshold:

set-ContentFilterConfig -SCLRejectThreshold 7 -SCLRejectEnabled:$true

In the above case, Exchange doesn't accept the message. After the data is received, it responds with a 500 5.7.1 error and a rejection response (by default this response is: Message rejected due to content restrictions. This rejection message can be configured using the following command (response message used here is for illustration, not a real suggestion - Bharat) :

set-ContentFilterConfig -RejectionResponse "Stop spamming you *****!"

The actual NDR is generated and sent to the sender by the sending host. What the sending host will see after the message content is sent (and if you actually modified the rejection response based on my example :)
500 5.7.1 Stop spamming you *****!


Figure 1: Exchange Server 2007's Content Filtering agent can be configured with the equivalent of all 3 Gateway actions available in Exchange Server 2003's Intelligent Message Filter (IMF): 1) Delete messages 2) Reject messages and 4) Quarantine messages

3. SCLQuarantineThreshold: Messages with SCL equal to or higher than the SCLQuarantineThreshold are delivered to the quarantine mailbox, provided you have one configured. To enable the SCLQuarantineThreshold and configure a quarantine mailbox:

set-ContentFilterConfig -SCLQuarantineThreshold 6 -SCLQuarantineEnabled:$true -QuarantineMailbox:MyQuarantineMailbox@mydomain.com

This is an improvement over IMF, which had only one gateway action (and one corresponding gateway threshold). The Content Filter agent allows the flexibility of enabling all three actions on the gateway - the rule is: SCLDeleteThreshold > SCLRejectThreshold > SCLQuarantineThreshold.

To get a list of all three SCL values and whether each action is enabled or not, use the following command:

get-ContentFilterConfig | Select SCL*

So where's the equivalent of IMF's Store threshold? (to move messages to users' Junk Mail folders)?

It's called SCLJunkThreshold, and it resides in a different location - in the Organization configuration. It can be set using the set-OrganizationConfig command:

set-OrganizationConfig -SCLJunkThreshold 5

Setting SCLJunkThreshold not intuitive?

Before you jump to conclusions about this being counter-intuitive, or confusing - which it may be, consider this - it is in response to the different server roles in Exchange Server 2007.

The gateway actions - delete, reject and quarantine - can be thought of as message transport actions, and thus applicable to transport server roles (the Edge server, or the Hub if you have antispam agents enabled on it). Moving messages to users' Junk Mail folders can be thought of as something that happens at the Store, performed by the Mailbox Server role.

Another aspect to consider when setting the SCLJunkThreshold - if you're in a topology with an Edge server, the SCLJunkThreshold on the Edge doesn't impact anything. This needs to be set on your Exchange Org - the Edge server is not a part of it.

The Defaults: By default, the SCLJunkThreshold is set to 4. If you have an existing Exchange Server 2003 SP2 server installed, and you haven't tweaked the Store threshold, IMF v2's default Store threshold of 8 is used - this is what you'll see in the Org's SCLJunkThreshold. Given that the SCLRejectThreshold is set to 7 by default, messages will not move to users' Junk Mail folder unless the SCLJunkThreshold is lower than the transport thresholds (i.e. the Delete, Reject and Quarantine thresholds).

How the Junk Mail threshold is calculated: Unlike the transport actions of deleting, rejecting, and quarantining messages - which check for SCL equal to or higher than their respective thresholds, for moving messages to Junk Mail folder the Store checks for SCLs higher than the SCLJunkThreshold. This is consistent with the behavior of IMF in Exchange Server 2003 (read previous post: IMF Confusion - Store threshold rating text in UI).

If you want to disable rejection of messages with SCL of 7 or above, use the following command:

set-ContentFilterConfig -SCLRejectEnabled:$false

Related Posts:
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs
- Moving items to Junk Mail folder doesn't do much

Exchange Server 2007: Making SenderID work with non-Exchange smtp hosts

To make the SenderID filter work with non-Exchange smtp hosts acting as your mail gateways for inbound mail, you need to tell Exchange about them. If Exchange does not know about these, SenderID will not be able to determine the correct sending host, and you may see SenderID checks fail on all inbound mail.

This may have been a source of confusion for users in the past - which hosts do you consider as "internal" - the ones on your internal network? or only those located in perimeter networks (aka "DMZ")? how about those located at your ISP?

Exchange needs to know about all smtp hosts that do not belong to senders - the ones that will handle your inbound mail, whether at your ISP or in your perimeter or internal networks, before messages are delivered to the Exchange Org. (Start by taking a look at MX records in external DNS zones - do these point to non-Exchange servers? If yes, do these hosts route mail to Exchange servers?).

In Exchange Server 2003 this is done by entering internal smtp server addresses in Global Settings | Message Delivery properties | General tab | Perimeter IP List and internal IP Range Configuration.


Figure 1: Setting up internal or perimeter SMTP hosts in Exchange Server 2003's PerimeterIPList

On Exchange Server 2007, you can configure this from the shell:

set-transportconfig -InternalSMTPServers 1.2.3.4,1.2.3.5,1.2.3.6

Updates:
8/16/2007: SP1 Beta2 allows setting InternalSMTPServers parameter using the Exchange console, as shown below:


Figure 2: Setting up InternalSMTPServers using the Exchange console in Exchange Server 2007's new Global Settings tab | Transport Settings | properties | Message Delivery tab

Exchange 2007 Content FIlter: The Whitelist Is Here!

Messaging Hygiene features in Exchange Server 2003, including the Intelligent Message Filter (IMF), did not have a way to whitelist sending domains or SMTP addresses.

This is a follow up to a previous post, and one of the more popular ones on this blog— "IMF: Where's the whitelist?". ("IMF and whitelist" has for long been one of the most common search terms on this blog - Bharat).

Whitelists are common in most 3rd-party anti-spam tools. Adding domains or SMTP addresses of important senders like customers, vendors, or your CEO's home email address (almost always an AOL address... :) for instance, ensures messages from these domains or addresses do not get filtered by the anti-spam filter.

Bypassed Senders and Sender Domains: The Whitelist
The good news is— Exchange Server 2007's shiny new Content Filter Agent (or IMF v3 if you will) has whitelists! You can add SMTP addresses and domains to the Content Filter configuration, and have messages from these senders and domains bypass the Content Filter Agent. However, you need to resort to the Exchange shell (EMS) to manage it.

Use the following shell command to add sender SMTP addresses to the BypassedSenders list:

Set-ContentFilterConfig -BypassedSenders foo@somedomain.com,foo2@somedomain.com

Use the following command to whitelist the sending domain:

Set-ContentFilterConfig -BypassedSenderDomains somedomain.com,someotherdomain.com

Some whitelisting considerations
Before you start using whitelists, here are a few things you should consider:

• SMTP headers can be spoofed easily. If spammers spoof any of the addresses or domains you whitelist, your recipients may end up getting more spam as all of it will bypass the Content Filter.
  
• Use SenderID Filtering to detect and protect your mail system from header spoofing.
  
• Maintaining whitelists, just as maintaining blacklists, is a manual process that imposes its own management costs.
  
• Checking every inbound message against a list of whitelisted recipients imposes a performance penalty - miniscule as it may be. Use the whitelists sparingly.

Nevertheless, many IMF users have repeatedly demanded this functionality and it's great to finally have it in what some folks call IMF v3.0.

Bypassed Recipients: The Exception List
The Content Filter can also be configured with an exception list - to not apply the filter to inbound messages for particular recipients. This can be done from the console by going to Hub Transport | Anti-spam tab | Content Filtering -> properties | Exceptions. This list is limited to a 100 recipients - you can add generic recipients that you want to exempt from the Content Filter, such as sales@yourdomain.com, info@yourdomain.com, etc.

To add recipients to the exception list using the Exchange shell:

Set-ContentFilterConfig -BypassedRecipients sales@yourdomain.com,info@yourdomain.com

Related Posts:

• BypassedSenders, BypassedSenderDomains, and BypassedRecipients are multivalued attributes. The following post shows how to modify multivalued attributes:
  HOW TO Update multi-valued attributes in PowerShell
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• HOW TO: Install anti-spam agents on Hub Transport server
  
• Quick antispam report or status check?
  
• HOW TO: Expose original senders and recipients of quarantined messages

IMF: Updates not showing up in Add/Remove Programs?

Some users report that recent updates to Microsoft's Intelligent Message Filter (for Exchange Server 2003) do not show up in Add/Remove Programs in Windows. Double-checking on some servers, I've noticed this - the last update that shows up on those in Add/Remove Programs is dated 8/10/2006.

If you look at the registry key for this update (all IMF Updates show up as update for KBA 907747) under HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall\KB907747, the DisplayVersion value - which is my best guess at what should show up in Add/Remove Programs or an indicator of which update was installed last - shows 2006.11.30. This was the update prior to the current one that was released in January.

One way to determine the version of IMF update currently installed on your system is to go to the \exchsrvr\bin\MSCFV2 directory (\exchsrvr is the install path for Exchange server), and look at the sub-folder that starts with the IMF update version - e.g. 6.5.xxx.x - with the latest date. The following screenshot shows the latest update that was downloaded on 1/18.