[Esd-l] Re: Fw: On e-mail sent with header: Content-Transfer-Encoding: base64

Tue Feb 3 20:55:55 PST 2004

I cannot thank you more.

Please see my note below.

----- Original Message ----- 
From: "John D. Hardin" <jhardin at impsec.org>
To: "Min-Soo Kim" <minsukim at jikji.org>
Cc: "Email Security Discussion list" <Esd-l at spconnect.com>
Sent: Tuesday, February 03, 2004 11:14 PM
Subject: Re: Fw: On e-mail sent with header: Content-Transfer-Encoding: base64

> On Tue, 3 Feb 2004, Min-Soo Kim wrote:
> 
> > As it looks like you're blocking my server's IP, I try to reach
> > you.
> > 
> > Feb 3 15:34:51 jikji-home sendmail[14666]: i136YXbF014664:
> > to=<esd-l at spconnect.com>, ctladdr=<minsukim at jikji.org> (1000/0),
> > delay=00:00:15, xdelay=00:00:15, mailer=esmtp, pri=34112,
> > relay=merlin2.spconnect.com. [204.96.236.14], dsn=4.0.0,
> > stat=Deferred: 452 4.2.1 mail i136YpP28780 from 211.239.126.51
> > temporary greylist embargoed
> 
> You'll have to talk to the list owner. I don't administer the mailing
> list.
> 
> I'll CC the list so that he sees this.
> 
> > ----- Original Message ----- 
> > From: "Min-Soo Kim" <minsukim at jikji.org>
> > 
> > Yes, I'm a Korean, but I really hate spams all over the world, and
> > I'm using your Sanitizer and spambnc quite successfully to block
> > spam e-mails among other tools I've been used.
> 
> I maintain the sanitizer, and it's not intended to be an antispam
> tool. What is "spambnc"? I don't recognize it.

please refer to http://www.spambouncer.org/

> 
> > allocated by international organization, but also because we
> > cannot block UTF-7/UTF-8 encoding from each other for better
> > communication in near future.  It's not a matter of whether one
> > country is notorious for sending / relaying spams, but how we're
> > going to prevent myself / my kids / my friend / my neighbor being
> > exposed to that 4-letter spammers and threir e-mails is what
> > matters and is more important.
> 
> You should be able to check for specific text strings in procmail
> regardless of their character set. You just need to be careful if the
> character maps to an ASCII character that is syntactically meaningful
> to procmail.
> 
> Scanning for text in a base-64 encoded body is more difficult,
> though...
> 
> > I found that Sanitizer is quite useful tool to make the goal I had
> > set more realistic.
> > 
> > Here is my most recent problem. (I'm not a programmer, but an
> > end-user) With this mail header below, images - entire HTML
> > contents - can be seen and not filtered, and the continuing
> > problem is I have to use Korean encoding for my daily life and
> > should accept this transfer-endocing according to
> > http://support.microsoft.com/default.aspx?scid=kb;EN-US;323489 .  
> > God feeling is this is not a proper e-mail, but I do not know how
> > to block them.
> 
> > > E-mail header is like this:
> > ===================
> > MIME-Version: 1.0
> > X-Security: MIME headers sanitized on jikji-home.jikji.org
> > See http://www.impsec.org/email-tools/sanitizer-intro.html
> > for details. $Revision: 1.139 $Date: 2003-09-07 10:14:23-07 
> > Content-Type: multipart/mixed;boundary= "----=_NextPart_000_0058_18DCD15F.9AF77D72"
> > X-Priority: 3
> > X-MSMail-Priority: Normal
> > X-Mailer: Microsoft Outlook Express 6.00.2462.0000
> > X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2462.0000 
> > X-SpamBouncer: 1.8 (1/26/04)
> > X-SBPass: No Freemail Filtering
> > X-SBScore: 0 (Spam Threshold: 5) (Block Threshold: 2)
> > X-SBClass: OK
> > X-Folder: Default
> > X-UIDL: )n7!!3h`!!dY`"!<0T!!
> > 
> > ------=_NextPart_000_0058_18DCD15F.9AF77D72
> > Content-Type: text/html; charset= "ks_c_5601-1987"
> > Content-Transfer-Encoding: base64
> > 
> > vsiz58fPvLy/qSC/5MHyIMitwabAxyC/7rW/uLjAuLfOIMiutOu1x7TCIMbktM+9uiDHwbfO
> > ......
> > ===================
> 
> Yeah, base64-encoded plain text and HTML text is an annoyance. Add to
> that a non-english character set...
> 
> I think your problem is going to be resolved within spambnc. I would
> wager that it is capable of decoding the message body and scanning it
> for spam content (SpamAssassin is). The problem is it probably doesn't
> know how to recognize spam in Korean! You may be able to help out here
> if you can work with the spambnc people to develop recognition strings
> for Korean-language spams. Unfortunately this may require some
> programming skills in addition to being literate in Korean...
> 

That really makes me feel very sorry :-(, 
At this moment of time, at my age, I'll focus what I'm doing now.  
However, I'll keep in mind your input when I have change to talk with my fellow Korean programmers.  I'll be (let them) back. 

> > I've searched to get some hint, and what I found is as follows,
> > 
> > 1. http://news.spamcop.net/pipermail/spamcop-list/2003-February/031732.html
> >  "The messages _do_ contain valid HTML, and when I remove the header line,
> > the message is correctly parsed."
> 
> Which header line are they referring to?

I've searched blindly, and I assumed this header is "Content-Transfer-Encoding: base64".

I knew this message was bit old but even with spambnc and Sanitizer I could not block the text with content-transfer-encoding:base 64.
I have put three of my e-mail messages in here : http://www.jikji.org/base64_examples.txt

>  
> > 2. http://wpbl.pc9.org/procmailrc
> > "# The following will catch Mimail.Q, MiMail.R, Mydoom, Novarg, Shimg
> > # and automatically drop them after recording the sender for WPBL
> > # Last updated 2004-01-27 21:30 CST"
> 
> ...which is looking for specific strings in the base64 body,
> essentially a very lightweight antivirus tool.
> 
> > While I'm not sure whether this procmailrc is the one that blocks
> > the spam mail I'm getting, I put this recipe into my .procmailrc
> > with a hope, and now I'm wondering if you could make this thing
> > working with existing local-rules.procmail, as I said before, I'm
> > not goot at hacking.
> 
> As far as email worms ar concerned I'm not sure it's needed. The
> reason the sanitizer doesn't block certain worms is that they aren't
> using bare executable file attachments, but instead are wrapping the
> attack in a .ZIP file. Since historically this has been the proper
> method to get a legitimate file past an email security system, we now
> have a problem: do we start considering all .ZIP files as suspicious,
> or do we attempt to identify specific evil .ZIP attachments?
> 
> The local-rules is an attempt to do the latter. If at some point
> worms' usage of .ZIP files becomes unmanageable through local rules,
> then I will add .ZIP to the list of extensions to mangle. You can also
> add "ZIP" to your local list of mangled extensions if you feel that is
> an appropriate security policy for your site.
> 
> I hope to be able to add .ZIP attachment unwrapping per a suggestion
> made earlier, which may help as well.
> 
> But, note: The sanitizer is not an anti-spam tool and not an antivirus
> tool, and I will resist attempts to make it either.
> 

I've been utilizing Sanitizer to block worms and defang images and potentially harm attachment.  
I wrote to this group as you have made me feel cozy here, as one can see from your lengthier response ;-), additionally I'm not an active subscriber to spambnc nor SpamAssassin mailinglist. I was just in the same way of thinking to extend us usage of Sanitizer to defang images, which is sometimes more harmful than any attachment, and only difference I saw from messages I got with DEFANG_IMG and not was  "Content-Transfer-Encoding: base64" and I was not able to say whether it would be better to handle this e-mail with content scanning tool like SpamAssassin, Spambnc or your Sanitizer.  Your response made me clear what's the best approach might be.  Thank you again.

> > It's been a long mail, but the point is simple.
> 
> My response was longer... :)
> 
> >  1. Thank you so much Mr. John Hardin for the Sanitizer.  You
> > saved a lot of my energy for better use.
> 
> You're very welcome.
> 
> >  2. Appreciate if you could add Mimail.Q MiMail.R Mydoom, Shimg
> > into your distribution.  I could send you some e-mails if you need
> > them.
> 
> The .ZIP variants of those attacks are the subject of the discussion
> above. Perhaps it should be discussed specifically on-list.
> 
> As for spam detection on base-64-encoded text, there are several ways
> to proceed:
> 
> 1) write a tool that decodes base64-encoded text body parts. I has
> considered doing this as a "sanitizing" step in the next generation of
> the sanitizer, but I have not carefully explored the side effects.
> There may be situations where base64-encoding text is *critical* to
> reliable delivery of that text. Anybody have and comments or examples
> of this?
> 
> Anyway, if you did this, then non-base64-aware text scanners (e.g.
> procmail rules) would be able to see the message body.
> 
> 2) decode the body in a separate file, scan it for spamminess, and
> assign the result back to the original message. (Note how I leave
> "scan it for spamminess" as a black box :)
> 
> 3) have procmail scan for the encoded spammy strings in the message
> body. This will be less obvious (what word of phrase does that string
> of gibberish represent?) and less efficient (one phrase will generate
> 3 (I think) search strings based on the byte alignment of the
> encoding). It's also more difficult to generate the search
> expressions.

This is actually part of my problem, well my entire problem as a non-ASCII based text user.  
http://www.iana.org/assignments/character-sets; 
    KS C 5601 
    ISO-2022-KR
    EUC-KR
    UTF-8
There is some historic reason I sometimes find myself difficult to explain with my own language :-(, but these are 'four' charsets Koreans normally use to put Korean text in an e-mail/internet document.  That means that I need to 'decode' at least 4 times for spambnc to actually scan the content.  Eventually I'll use UTF-8, but it's a headache and chaotic at the moment.

Now I'm having a better undestanding why spambnc has default setting I need to re-configure in my  ~./procmailrc.

    BASE64BLOCK=no          # I want to get unicoe & Korean e-mails
    KOREAN=yes              # Needless to say

For you guys it's pretty easy, but....

> 
> Free project idea for the list: write a tool that will take a simple
> text string (or list thereof) and generate procmail search expressions
> that will detect that string in an encoded text body part. Extra
> credit: support simple regular expressions. :)
> 
> The all you'd need to do would be to type all your nasty words into a
> text file using your character set of choice, run the tool over it,
> and put the result into your /etc/procmail/antispam.procmail file.
> 
> > With regards, Min-Soo Kim.
> 
> Regards. Sorry that you're suffering so disproportionately from the
> spam problem.
> 

Thank you for your thoughtful concern and walking on my shoes.

> --
>  John Hardin KA7OHZ    ICQ#15735746    http://www.impsec.org/~jhardin/
>  jhardin at impsec.org                        pgpk -a jhardin at impsec.org
>  key: 0xB8732E79 - 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
>   "Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never
>   does quite what I want. I wish Christopher Robin was here."
> -- Peter da Silva in a.s.r
> -----------------------------------------------------------------------
>    61 days until the Slovakian Presidential Election
> 