TheThoughtfulCoder.com

Create absolute URLs using ASP.NET MVC

Wed, 25 Jan 2012 20:58:34 GMT

At times it is desirable to create an absolute URL for a HTML element's href or src attribute. I failed to find a shrink wrapped mechanism to generate them in ASP.NET MVC, and some of the methods I found on the web were not quite what I wanted. So I created extension methods for the Uri class, which will generate absolute URLs relative to the current context's Request.Url property.

The class definition follows:

public static class UriHelperExtensions
{
   // Prepend the provided path with the scheme, host, and port of the request.
   public static string FormatAbsoluteUrl(this Uri url, string path)
   {
      return string.Format( 
         "{0}/{1}", url.FormatUrlStart(), path.TrimStart('/') );
   }

   // Generate a string with the scheme, host, and port if not 80.
   public static string FormatUrlStart(this Uri url)
   {
      return string.Format( "{0}://{1}{2}", url.Scheme, 
         url.Host, url.Port == 80 ? string.Empty : ":" + url.Port );
   }
}

The following snippet of code from an ASP.NET MVC Razor view demonstrates how to generate an absolute URL given the relative URL /images/img.jpg.

   <img src="@Request.Url.FormatAbsoluteUrl("/images/img.jpg")" alt="Alt text" />

For my blog, the HTML generated for the img element will be:

I hope this helps :-)

Domain name siphoning, how to protect a website

Sat, 21 Jan 2012 23:45:15 GMT

Some time ago I was going through web logs, and I noticed a number of requests being served on a domain that did not belong to my employer. It ends up the entire website was indexed in the search engines under this other domain.

It is simple for anyone to pull this off, all it takes is to point a DNS entry to some other website's IP address, and now all their pages will appear under that domain if the target site is not protecting itself. In a sense the domain name is siphoning off the content of another website.

It ends up there were a number of domains doing this to my employer's web site. I can only guess at the motivation. It might be a free way to build up SEO keyword relevance before it is switched over to other content. It might be a build up to a social engineering hacking attack. Regardless, it is not a desirable situation to allow for many reasons.

Your pages may be found in search engines under the siphoning domain, which would be very confusing to your end users.
This is a disaster from an SEO perspective, as the siphoning domain is a complete duplicate of your domain's content.
The siphoning domain has the potential to use a lot of bandwidth.

There are probably many ways to deal with this issue, but I decided to use rewrite rules to protect against this situation. I implemented the concept of authorized domains for a website.

Rewrite rules for Apache's mod_rewrite or Helicon Tech's Ape

I've used Helicon Tech's ISPAI Rewrite, and now Ape, for years now under IIS, but I've tested these rules using mod_rewrite on Apache 2.2 as well.

I implemented two things that should make attackers steer clear of targeting a domain: return a robots.txt that tells the search engines not to index any content and return a HTTP 410 Gone for any other page request. These two things tell any bot or user that nothing exists on a siphoning domain targeting your website. Valid content will only be returned from domains you authorize.

I created a rewrite map which lists authorized domains. For example, assume the domains www.domain-test1.com and images.domain-test1.com are the domains you use for your site. Create a text file named AuthorizedDomains.txt with the following content:

www.domain-test1.com -
images.domain-test1.com -

Place a file called robotsDisallow.txt in your website's root with the following content.

User-Agent: *
Disallow: /

The robotsDisallow.txt will be served for robots.txt requests on an unauthorized domain.

The following rewrite rules should be placed in your httpd.conf file.

RewriteMap  lower int:tolower
RewriteMap AuthorizedDomainsMap txt:AuthorizedDomains.txt

# Serve a robots.txt file which tells search engines not to index unauthorized domains.
RewriteCond ${AuthorizedDomainsMap:${lower:%{SERVER_NAME}}|NOT_FOUND} NOT_FOUND
RewriteRule ^/robots\.txt$ /robotsDisallow.txt [NC,L]

# Return a HTTP 410 page for unauthorized domains.
RewriteCond ${AuthorizedDomainsMap:${lower:%{SERVER_NAME}}|NOT_FOUND} NOT_FOUND
RewriteRule .? - [G]

For Apache, I put the AuthorizedDomains.txt file in the Apache installation folder on Windows, but you can specify the path to the file as well.

A few notes:

This may be obvious, but make sure to thoroughly test these rules. You do not want to accidentally muck up your website.
I recommend using authorized domains in your development environment to make sure the rules are working as expected.
I used a map file as my team uses a lot of domains in our development environment. This makes for easier management of authorized domains, as well as allowing for the use of the same httpd.conf for both development and production, but different authorized domain configurations.
Make sure to put your authorized domain rules after rules that redirect subdomains you wish to send to your primary domain(s).

Rewrite rules for IIS's URL Rewrite Module

You can implement a functionally equivalent set of rules using IIS's URL Rewrite Module. Again, I created a map for the authorzied domain look up, but unlike mod_rewrite, the map is not stored in a separate file. As in the previous example, assume the domains www.domain-test1.com and images.domain-test1.com are your authorized domains.

Place a file called robotsDisallow.txt in your website's root with the following content.

User-Agent: *
Disallow: /

The robotsDisallow.txt will be served for robots.txt requests on an unauthorized domain.

The following rules go in the website's web.config file.


<rewrite>
   <rules>
       <rule name="robots for unauthorized domain" stopProcessing="true">
           <match url="^robots\.txt$" />
           <conditions logicalGrouping="MatchAll">
               <add input="{Authorized domains:{SERVER_NAME}}" pattern="-" negate="true" />
           </conditions>
           <action type="Rewrite" url="/robotsDisallow.txt" />
       </rule>
       <rule name="Authorized domain check" stopProcessing="true">
           <match url=".?" />
           <conditions logicalGrouping="MatchAll">
               <add input="{Authorized domains:{SERVER_NAME}}" pattern="-" negate="true" />
           </conditions>
           <action type="CustomResponse" statusCode="410" 
              statusReason="Gone" 
              statusDescription="The requested resource is no longer available" />
       </rule>
   </rules>
   <rewriteMaps>
       <rewriteMap name="Authorized domains">
           <add key="www.domain-test1.com" value="-" />
           <add key="images.domain-test1.com" value="-" />
       </rewriteMap>
   </rewriteMaps>
</rewrite>

Like the mod_rewrite rules, these rules will return a robots.txt file that tells search engines not to index any content on an unauthorized domain and to return a HTTP 410 Gone for all page requests on an unauthorized domain.

Another options is to use host headers in IIS to restrict the domains for a website, but this mechanism should be used as a last resort. Host headers in IIS have a number of limitations, so avoid them if possible.

Wrap up

To check for this situation, review your website's web logs and look for any unfamiliar domain names. If you find that a website has already fallen victim to this attack and is indexed by the search engines under the siphoning domain, you can implement the use of authorized domains. Then it will take time. It can take quite a while for your pages on the siphoning domain to start dropping out of the search engines, but it will take effect eventually. If I remember correctly, it took a few months for the search engines to start dropping the pages.

It is surprising how many websites are open to this attack. I checked a number of major websites, and all but one was open to this issue. I'm sure there are legal ways to stop someone from doing this, but with a fairly small technical implementation, this can be averted in a much cheaper way.

If you have any questions please let me know. I hope this helps.

.htaccess subdomain redirect

Wed, 18 Jan 2012 00:22:25 GMT

A common desire is to redirect all subdomains to a single domain for all website requests. The following rewrite rule will 301 redirect all subdomains other than www to the www subdomain.

RewriteCond %{HTTP_HOST} (?<!www\.)domain\.com$ [NC]
RewriteRule (.*) http://www.domain.com/$1 [R=301,L]

I used this rule in IIS 7.5 using HeliconTech's Ape, but I've tested it with Apache's mod_rewrite as well. It can go in your .htaccess file or httpd.conf. If put in httpd.conf, it will work as is using RewriteBase /, otherwise remove the trailing slash in the RewriteRule's substitution string.

I hope this helps :-)

Skullcandy stencil usage examples

Wed, 11 Jan 2012 01:09:59 GMT

I bought a set of Skullcandy headphones to replace my decade old, falling apart ones. The Skullcandy headphones are definitely better than my old set, not quite as good as I expected for the price, but good nonetheless. Included in the package is a Skullcandy stencil.

My favorite part of the stencil is the sidebar pictures showing possible stencil usage.

Snowboard - okay makes sense.
The back of a bald guys head while reading a newspaper - Judge Judy wouldn't be happy with these antics.
On your cat - would someone's cat actually hold still long enough?

I guess I'm too old to be the target demographic for these headphones. I'm more likely the target of the stencil on the back of head, but I still find it humorous.

Okay, who wore a pair of Famous Monsters underwear?

Wed, 11 Jan 2012 01:09:59 GMT

I was going through an old Famous Monsters of Filmland magazine, issue# 132 printed in 1977, and one thing really stood out to me; the advertisements. They are like an old horror movie: campy, funny, and simply awesome :-)

Nothing quite says super nerd virgin like a pair of these underwear. Okay, okay, maybe some women would really dig these. Well, probably not. Not even a Star Trek convention girl could likely accept an Eerie underwear wearing nerd.

I'm not sure I'd want my underwear to say Eerie or Creepy anyway. That might not be quite what you'd like to advertise. On the other hand, sporting Godzilla with Famous Monsters printed on your bikini underwear, well, you might just want to have the goods to back that one up!

I love how it states, PLEASE BE VERY SURE YOU ORDER THE CORRECT SIZE. Hopefully they didn't take returns. The nerd cooties on a used pair of these would push the wearer beyond the boundaries of any possible social connection with a human being.

I've never thought of having something printed on my underwear. I mean, how do you show these off? Do you just drop trou in front of your friends to show off your awesome undergarments? These are not really appropriate for the locker room, as you might get a good old fashioned nerd induced butt kicking from a Boogeresque kind of jock.

At least the guy appearing in the advertisement is realistic. Putting some muscle bound hunk in a pair of these babies wouldn't really help sell them. It just wouldn't be believable.

Don't get me wrong, these things are awesome. I just need to know, did anyone really buy, or better yet, wear a pair of these? Come on, be honest.

Reduce bleed through for double sided document scan

Tue, 10 Jan 2012 21:10:53 GMT

I scanned some old documents which have print on each side. When scanned, there was significant bleed through from the reverse side in the image output. A couple example pieces of the image follow.

The print on the reverse side comes through pretty strong. The inside of the scanner cover is white, which will certainly reflect more light. I decided to put black construction paper on top of the document to see if it would help.

The scan with the black backing significantly reduces the bleed through of the text on the reverse side. The black backing does darken the scanned image, but it can be brightened up using an image editing tool.

The top two images show the scan results with the black backing. The next two images are the scans without the black backing. It is not perfect, but I'm happy with the result.

I hope this helps someone else :-)

Godzilla Fan Club fan letters from 1977. A forgotten treasure.

Fri, 06 Jan 2012 02:47:21 GMT

I was going through some of my old stuff and I came across a folder of old forgotten childhood treasure. First a little backdrop, the 70's were a different era; access to media at that time was very limited. We had 3 or 4 television channels, depending on the antenna's rabbit ear configuration and a little luck. Movies and magazines were the other main entertainment media outlets in that day.

As a kid, I was a huge Godzilla fan. Once in a while I'd get to see the latest Godzilla film in the theater. Otherwise, I'd flip through the TV Guide from cover to cover, hoping to find a Godzilla film showing on TV.

The other main medium for Godzilla was magazine articles, mainly appearing in Famous Monsters of Filmland magazine. I'd always hope for the big G's appearance on the cover of the next Famous Monsters. In issue 132 of Famous Monsters, I came across the announcement of the Godzilla Fan Club. I remember being so excited about the prospect of a Godzilla focused periodical! Sweet!

I recall being so excited to get the fan letters, and so disappointed when it was abandoned. They were a prized possession of mine.

Having recently come across them, I decided to search the internet to see what existed about these fan letters. To my amazement, I could find very little mention of their existence, let alone copies of them. I think this is the first time I've searched for something on the internet that couldn't find. I actually have something that doesn't exist on the internet :-)

I decided to see if I could get in contact with the creator of the Godzilla Fan Letters, Richard Campbell. Luckily enough, I was able to contact him, and had a very pleasant interaction with him. He was very gracious that I had these copies; he no longer has copies of the fan letters he authored.

He was only 17 or 18 years old when these were created. That's quite a feat given the technology available back then. Typewriters were not the best word processors :-) Mr. Campbell was happy that these letters can now be shared with all Godzilla fans. I thank him for creating them in the first place, they made a kid very happy.

Mr. Campbell mentioned that he created another Godzilla publication named Godzillamania, of which there were two issues. One of my hopes is to see if anyone still has copies of the Godzillamania fanzines. I'd love to read them. Along with these fan letters, they would be a very cool piece of Godzilla fan history.

There were only 5 issues of the Godzilla Fan Letter, totaling 21 pages printed front and back, including my welcome letter. I've scanned them all and created various formats, which I'll make available on this blog post. The letters are quite worn and have yellowed with age. I've made some black and white versions, along with the current form of the letters. I have blacked out any personal addresses and signatures that were on the letters. It is a different day now, I want be careful in that regard.

You can view the letters on my blog using the following link: Godzilla Fan Letters from 1977.

I'm also providing the letters in comic book format (CBZ) and PDF. If you are not familiar with CBZ readers, then you probably just want the PDF. If you haven't heard of comic book readers, you might want to check out ComicRack. It is a very cool program. The CBZ format has the highest quality images.

Black and white version of the Godzilla Fan letters in CBZ format Godzilla-Fan-Letters-BW.cbz
Black and white version of the Godzilla Fan letters in PDF format Godzilla-Fan-Letters-BW.pdf
Color version of the Godzilla Fan letters in CBZ format Godzilla-Fan-Letters-Color.cbz
Color version of the Godzilla Fan letters in PDF format Godzilla-Fan-Letters-Color.pdf

Enjoy :-)

Cannot import registry script (.reg file) Windows 7 error

Wed, 28 Dec 2011 21:31:28 GMT

I created and tried to import a .reg file but I received the error: Cannot import myfile.reg. The specified file is not a registry script. You can only import binary registry files from within the registry editor. The dialog box follows:

My file was UTF-8 encoded. I exported a registry key and the file encoding was UTF-16LE. I created a new file with UTF-16LE encoding with my registry changes and it imported successfully. I'm not sure if other encodings will work, but UTF-16LE seems to do it.

I hope this helps :-)

Combine and minify JavaScript (JS) and CSS using AjaxMin

Wed, 30 Nov 2011 22:09:07 GMT

Rendering a page on my blog requires 12 JavaScript files and 3 CSS files. This is is a lot of HTTP request overhead. I decided to use AjaxMin to combine and minify my JS and CSS files. Now my pages only requires one JS and one CSS file.

AjaxMin can use an XML file to specify which files to minify and combine along with the target output file. The input file for my blog:


<root>
   <output path="..\..\web\site.js">
      <input path="..\..\web\syntax\scripts\shCore.js" />
      <input path="..\..\web\syntax\scripts\shLegacy.js" />
      <input path="..\..\web\syntax\scripts\shBrushCSharp.js" />
      <input path="..\..\web\syntax\scripts\shBrushXml.js" />
      <input path="..\..\web\syntax\scripts\shBrushPlain.js" />
      <input path="..\..\web\syntax\scripts\shBrushVb.js" />
      <input path="..\..\web\syntax\scripts\shBrushPowerShell.js" />
      <input path="..\..\web\scripts\jquery-1.7.1.min.js" />
      <input path="..\..\web\scripts\json2.js" />
      <input path="..\..\web\scripts\jquery.validate.min.js" />
      <input path="..\..\web\scripts\jquery.form.js" />
      <input path="..\..\web\scripts\FoxBlog.js" />
   </output>
   <output path="..\..\web\site.css">
      <input path="..\..\web\syntax\styles\shCore.css" />
      <input path="..\..\web\syntax\styles\shThemeDefault.css" />
      <input path="..\..\web\content\site.css" />
   </output>
</root>

The file name for the XML input file will be passed through the -xml argument to AjaxMin.

I'm running AjaxMin out of a folder off the root of my checkout tree, so the paths to all the files are relative paths that resolve to my web project folder structure. The folder structure of interest is shown below.

AjaxMin is run out of the 3rdParty\AjaxMin folder. The input file is also located in the AjaxMin folder. The command to run AjaxMin:

ajaxmin -clobber -xml AjaxMin.xml

After executing AjaxMin, the files site.css and site.js will be created in the Web folder. These are the files that will be referenced in the website code.

AjaxMin can be run as a build event in Visual Studio, although technically it would only be necessary to run it if any of the CSS or JavsScript changes. A build event command for the above project structure:

$(ProjectDir)..\3rdParty\AjaxMin\ajaxmin -clobber -xml $(ProjectDir)..\3rdParty\AjaxMin\AjaxMin.xml

It might be better to set up a external tool in Visual Studio to run AjaxMin, a shortcut, or whatever is most convenient.

Keep in mind, some sort of versioning needs to be done with the CSS and JS files so the browser will download the latest version of the file. I have yet to implement a versioning mechanism for my blog, but I need to get to it at some point. Maybe a blog post about it as well :-)

Check for null in select when using LINQ to Objects left outer join

Sat, 26 Nov 2011 14:47:29 GMT

Make sure to check for null in the select of a LINQ to objects query when using a left outer join.

The following example illustrates how to get a method value out of class that is the target of the outer join. The code can be run in LINQPad as a C# program.


class Foo
{
    public int FooKey { get; set; }
    public string FooVal { get; set; }
}

class Bar
{
	public int BarKey { get; set; }
    public string BarVal { get; set; }
}

void Main()
{
	List<Foo> aListOFoo = new List<Foo> {
        new Foo { FooKey = 1, FooVal = "FooVal1" },
        new Foo { FooKey = 2, FooVal = "FooVal2" },
        new Foo { FooKey = 3, FooVal = "FooVal3" } };
    
    List<Bar> aListOBar = new List<Bar> {
        new Bar { BarKey = 1, BarVal = "BarVal1" },
        new Bar { BarKey = 2, BarVal = "BarVal2" } };
        
    var q =
        from f in aListOFoo
            join b in aListOBar on f.FooKey equals b.BarKey into fbJoin
        from bLeftJoin in fbJoin.DefaultIfEmpty()
        select new { 
            f.FooVal, 
            BarVal = bLeftJoin != null ? bLeftJoin.BarVal : null };
    
    q.Dump();
}

The output for the above code in LINQPad:

If the select used bLeftJoin.BarVal without the ternary check for null, then the query would return Object reference not set to an instance of an object.

I'm not a huge fan of LINQ outer join syntax, but LINQ brings me much happiness nonetheless :-)

Use LINQ to page through a collection in memory with LINQ to Objects

Sat, 26 Nov 2011 00:00:57 GMT

It is not uncommon to find myself needing to page through a collection of .NET objects in memory. LINQ to objects provides a great way to accomplish paging using the Skip and Take extension methods.

The following example code illustrates the pattern in C#.


List<int> list = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
int chunkSize = 3, index = 0;
IEnumerable<int> chunk = list.Take(chunkSize);

while( chunk.Count() > 0 )
{
	// Do something with the chunk of data.
	chunk.Dump(); // This shows the data in LINQPad.
	
	index += chunkSize;
	chunk = list.Skip(index).Take(chunkSize);
}

The results of the above code in LINQPad:

A common scenario that arises is the need to run a database query with an in clause across a large amount of data in memory. A where clause with a condition comparing to a list in an ORM tool normally results in an IN clause with bound parameters. Since SQL Server can have a limited number of entries in an IN clause, somewhere around 2,000, processing can easily be done by paging through the data in memory.

Happiness :-)

.NET code to help with XML sitemap generation

Wed, 23 Nov 2011 00:48:33 GMT

I created some code which might save some time for others who want to generate XML sitemaps from .NET code. It is written in C#, but would likely be easily ported to other .NET languages. This code is used by my ASP.NET MVC blog engine to generate the sitemap for my blog entries. It could just as easily be used in ASP.NET WebForms, a console application, or from any .NET code framework.

The code for the helper class can be found here. You might want to change the namespace in the file.

The class SiteMapData is used to hold the data for each sitemap node.

[Serializable]
public class SiteMapData
{
   public string Loc { get; set; }
   public DateTime? Lastmod { get; set; }
   public string Changefreq { get; set; }
   public decimal? Priority { get; set; }
}

A list of SiteMapData instances is passed into the SiteMapHelper class function GenerateSiteMap, which will transform the list of SiteMapData objects into an XDocument.

public XDocument GenerateSiteMap(List<SiteMapData> dataRows)
{
   var xmlNodes =
      (from x in dataRows
         select CreateSiteMapUrlNode(x));

   XDocument siteMap = new XDocument(
      new XDeclaration("1.0", "utf-8", "yes"),
      new XElement(_xmlns + "urlset",
         new XAttribute(XNamespace.Xmlns + "xsd", "http://www.w3.org/2001/XMLSchema"),
         new XAttribute(XNamespace.Xmlns + "xsi", "http://www.w3.org/2001/XMLSchema-instance"), xmlNodes));

   return siteMap;
}

The resulting XDocument can be converted to a GZip'ed stream of bytes by calling the SiteMapHelper function GZipSiteMap.

public MemoryStream GZipSiteMap(XDocument siteMap)
{
   using (MemoryStream ms = new MemoryStream())
   {
      siteMap.Save(ms);
      ms.Seek(0, SeekOrigin.Begin);
      using (MemoryStream compressed = new MemoryStream())
      {
         using (GZipStream zip = 
            new GZipStream(compressed, System.IO.Compression.CompressionMode.Compress))
         {
            ms.CopyTo(zip, (int)ms.Length);
         }

         return new MemoryStream(compressed.ToArray());
      }
   }
}

The following code is an example of how to use the SiteMapHelper class to generate a sitemap from an ASP.NET MVC controller action. This is the code used to generate the sitemap for my blog.

public ActionResult SiteMap()
{
   EntityCollection<BlogEntryEntity> entities = 
      ServiceFactory.Beget<IEntityGenericService>().FetchAllEntities<EntityCollection<BlogEntryEntity>>();

   var mapData =
      from e in entities
      orderby e.DateInserted descending
      select new SiteMapData { 
         Loc = FormattingUtils.FormatUrlStart(Request.Url) + 
            Url.RouteUrl("Blog entry with id and title", new { BlogEntryId = e.BlogEntryId, blogEntrytitle = FormattingUtils.NormalizeTitle(e.Title) }), 
            Lastmod = e.LastTimeModified, 
            Priority = 0.5m, 
            Changefreq = "Weekly" };

   SiteMapHelper helper = new SiteMapHelper();

   XDocument siteMap = helper.GenerateSiteMap(mapData.ToList());
   MemoryStream compressedBytes = helper.GZipSiteMap(siteMap);

   FileStreamResult result = new FileStreamResult(compressedBytes, "gzip");
   result.FileDownloadName = "sitemap.xml.gz";
   return result;
}

The code used to fill in the SiteMapData list would change according to each specific situation. The use of the SiteMapHelper to the end of the function would be the same for most situations. Hopefully this illustrates the general idea.

One thing to keep in mind is that XML sitemaps can have at most 50,000 URLs and can be at most 10MB in size. So, if the list of SiteMapData nodes would generate a file that is larger than the given limits, then the list would have to be processed in chunks to generate separate sitemap files.

I share this with hope that it helps save someone a bit of time :-)

The entire SiteMapHelper code is shown below. It can be downloaded from the link at the top of this blog entry as well.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Xml.Linq;

namespace FoxBlog.Shared
{
   [Serializable]
   public class SiteMapData
   {
      public string Loc { get; set; }
      public DateTime? Lastmod { get; set; }
      public string Changefreq { get; set; }
      public decimal? Priority { get; set; }
   }

   public class SiteMapHelper
   {
      XNamespace _xmlns = "http://www.sitemaps.org/schemas/sitemap/0.9";

      public XDocument GenerateSiteMap(List<SiteMapData> dataRows)
      {
         var xmlNodes =
            (from x in dataRows
               select CreateSiteMapUrlNode(x));
   
         XDocument siteMap = new XDocument(
            new XDeclaration("1.0", "utf-8", "yes"),
            new XElement(_xmlns + "urlset",
               new XAttribute(XNamespace.Xmlns + "xsd", "http://www.w3.org/2001/XMLSchema"),
               new XAttribute(XNamespace.Xmlns + "xsi", "http://www.w3.org/2001/XMLSchema-instance"), xmlNodes));

         return siteMap;
      }

      public MemoryStream GZipSiteMap(XDocument siteMap)
      {
         using (MemoryStream ms = new MemoryStream())
         {
            siteMap.Save(ms);
            ms.Seek(0, SeekOrigin.Begin);
            using (MemoryStream compressed = new MemoryStream())
            {
               // The zip stream has to be closed to write out all the bytes. The underlying stream
               // is closed as well. To get the bytes of the the closed stream ToArray must be called.
               using (GZipStream zip = new GZipStream(compressed, System.IO.Compression.CompressionMode.Compress))
               {
                  ms.CopyTo(zip, (int)ms.Length);
               }

               return new MemoryStream(compressed.ToArray());
            }
         }
      }

      private XElement CreateSiteMapUrlNode(SiteMapData data)
      {
         XElement node = new XElement(_xmlns + "url", new XElement(_xmlns + "loc", data.Loc));

         if ( data.Lastmod.HasValue )
         {
            node.Add(new XElement(_xmlns + "lastmod", data.Lastmod.Value.ToString("yyyy-MM-dd")));
         }
         
         if ( data.Changefreq != null )
         {
            node.Add(new XElement(_xmlns + "changefreq", data.Changefreq));
         }

         if ( data.Priority.HasValue )
         {
            node.Add(new XElement(_xmlns + "priority", data.Priority.ToString()));
         }

         return node;
      }
   }
}

.NET GZipStream must be closed or it produces corrupt data

Tue, 22 Nov 2011 23:22:40 GMT

I encountered a situation in which I was missing bytes from my compressed data when using .NET's GZipStream class. The short of the story is the GZipStream must be closed, otherwise it does not write everything to the underlying stream that holds the compressed bytes.

For example, the following will produce an incorrect result:

string test = "This is a test string to compress";
using(MemoryStream source = new MemoryStream(UTF8Encoding.UTF8.GetBytes(test)))
{
   using ( MemoryStream compressedBytes = new MemoryStream() )
   {
      using ( GZipStream gzip = 
               new GZipStream(compressedBytes, CompressionMode.Compress) )
      {
         source.CopyTo(gzip);
         File.WriteAllBytes("out.txt.gz", compressedBytes.ToArray());
      }
   }
}

In this case, the out.txt.gz file will not contain the correct content. Using gzip.exe to decompress the file results in the error, unexpected end of file. When I use a tool like PowerArchiver, the file decompresses, but it has a bunch of garbage on the end.

Moving the use of the compressed bytes outside of the GZipStream's using statement will give the correct results.

string test = "This is a test string to compress";
using(MemoryStream source = new MemoryStream(UTF8Encoding.UTF8.GetBytes(test)))
{
   using (MemoryStream compressedBytes = new MemoryStream() )
   {
      using ( GZipStream gzip = 
               new GZipStream(compressedBytes, CompressionMode.Compress) )
      {
         source.CopyTo(gzip);
      }
      File.WriteAllBytes("out.txt.gz", compressedBytes.ToArray());
   }
}

Now the decompression of the file works. It is key to close or dispose the GZipStream in order to flush out all the bytes.

Another issue of note is ToArray should be used instead of GetBuffer on the stream holding the compressed bytes. In general the compressed bytes buffer will have a greater capacity than the length. In the previous example, using compressedBytes.GetBuffer() will result in in a file which decompresses successfully, but the message decompression OK, trailing garbage ignored will be given by gzip.exe.

All these issues might seem fairly obvious, but the situation in which this came up for me was a more complex scenario and it took a bit to figure out. Hopefully this might save some time for someone else :-)

How to remove the IIS ETag HTTP header in IIS 7 / 7.5 (75)

Thu, 10 Nov 2011 22:46:02 GMT

If you prefer to not send the ETag HTTP header for files served from IIS 7 or 7.5, it is a bit problematic. To the best of my knowledge, IIS does not provide a straight forward way to remove this header. This post shows a way to do it using Microsoft's URL Rewrite Module 2.0.

I'm not going to go into the what and why of the ETag. Suffice it to say, many web developers and web masters would prefer not to send it.

The ETag can be removed using an outbound URL rewrite rule. You can read more about it at the URL Rewrite Module 2.0 Configuration Reference .

The rewrite rule that can be put in your web.config follows:

   
<rewrite>
   <outboundRules>
      <rule name="Remove ETag" >
         <match serverVariable="RESPONSE_ETag" pattern=".+" />
         <action type="Rewrite" value="" />
      </rule>
   </outboundRules>
</rewrite>

This graphic shows the settings for the ETag removal rewrite rule in IIS Manager. In fact, if you look at the headers for the image, you'll see there is no ETag :-)

If you happen to be using Helicon's Ape, then this is very simple as well. You can use a mod_headers rule like:

      Header set ETag ""

I hope this helps :-)

SteelSeries Shift keyboard - Not recognized by SteelSeries Engine software

Thu, 10 Feb 2011 23:48:08 GMT

This is a quick post that hopefully might help someone else.

We just installed a SteelSeries Shift keyboard, along with the World of Warcraft Cataclysm keyset, on a machine running Windows 7 64-bit. The SteelSeries engine software would not recognize the keyboard and gave the message, “please connect a steelseries engine compatible device to start”.

Between my wife and I, we uninstalled and reinstalled the software many times and followed the directions on the SteelSeries website to no avail. To make a long story short, it ends up I needed to be logged in with an account which has administrator rights. I have logins for the family and I which do not run with admin privileges. I lock down my machine to keep my son from not going where I don’t want him to go and keep him from installing anything. To install any software, we have to enter the admin password when the UAC window pops up.

When installing the SteelSeries engine, the UAC window pops up so the administrator account login and password can be entered, but for some reason this was not enough. I had to do the install while logged in to the admin account. This usually isn’t an issue with most software, but it made the difference in the case.

Probably most people do not login with non-admin accounts, but I posted this just in case this might help someone out

Shut down or restart Windows from the command line

Sat, 04 Dec 2010 01:00:48 GMT

Often I need to restart a virtual machine that I’m connected to through RDP. For whatever reason this can be problematic at times. When connected through RDP, the option to shut down or restart may not be available on the start menu. Sometimes choosing shut down from the start menu on a virtual machine just doesn’t work either, but doing it from the command line works.

You can issue these commands from the command prompt or a Windows PowerShell prompt.

To restart from the prompt use:

shutdown –r –t 0

To shut down from the prompt use:

shutdown –s –t 0

If you do not supply the –t argument, there is a delay. If I’m trying to shut down or restart, I really don’t need a timer.

I’m pretty sure this works for all flavors of Windows from XP to Windows 7, and the server versions as well.

The IE 8 install, Checking your computer for malicious software

Sat, 04 Dec 2010 00:28:10 GMT

One of the steps during the Internet Explorer 8 install is, Checking you computer for malicious software:

If you have malicious software on your machine, isn’t it a little to late to be checking at this point? Is this really necessary? Isn’t this the goal of virus and malware scanners? Maybe malicious software is a code phrase for other browsers, or more likely IE 6

“You can now disconnect from the Internet” - Really?

Sat, 04 Dec 2010 00:16:56 GMT

I’m not sure why I found this funny; during the installation of the .NET Framework 3.5 the installer informs you, “You can now disconnect from the Internet”, after the download is complete.

It made me wonder, “When am I ever not connected to the internet?”. Pretty much, never. I suppose if someone travels around with a laptop out of usable Wi-Fi range. For me, I’m always connected.

I sent this to a friend with the joking subject, “I didn’t know this was possible”. He responded with the following image:

For some reason this made me literally laugh out loud I guess this would disconnect your machine from the internet for a bit as you’d have to search for new Cat5 or a spare RJ-45 connector and a crimper. Although, who uses Cat5 anymore? I only use it from my router to the cable modem.

The ease of being connected to the internet is such a change from the old dial-up days.

Google AdWords seller ratings extension – Some answers to some questions

Thu, 07 Oct 2010 00:30:38 GMT

Google AdWords seller ratings are a set of stars shown with an AdWords ad as follows:

If you are investigating seller ratings for your employer or your own company, I have a few tips that were not obvious to me at the outset.

Is a Google Merchant Center account or product listings in Google Product Search required for a seller rating to show in AdWords?

It ends up, a Google Merchant Center account is not needed and a site’s products do not need to be listed in Google Products to obtain a seller rating in AdWords. Much of the documentation provided by Google may lead to the assumption that products need to be listed in Google Products in order to have a rating, but it ends up this is not the case.

The key to qualifying for ratings in your AdWords ad is shown in this FAQ. An excerpt from the FAQ follows:

The FAQ mentions that seller ratings come from Google Product Search, but does not make it clear that a seller’s products do not need to be listed in Product Search. The FAQ does mention that a Google Merchant Center account is not needed, but it could be a bit more clear about the relationship.

How do I view a website’s seller rating?

To view a website’s rating in Google Product Search use a URL of the form:

http://www.google.com/products/seller?zmi=<base domain name of the website>

For example, to view the rating for www.amazon.com use the URL:

http://www.google.com/products/seller?zmi=amazon.com

I have a rating in Google Product Search above 4.0, why is the rating not showing in AdWords?

If a website has less than 30 ratings of 4 stars or above, then the website does not qualify for seller ratings to appear in their AdWords ads.

Does a seller rating in an AdWords ad affect click through rate?

I have not found any public empirical data to show how the seller rating affects click through rate. On the job though, it appears that click through rates are definitely affected by the lack of a seller rating when other ads do show a seller rating. Presumably, users are drawn to ads which have a rating and may think an AdWords listing without one may not be as desirable.

Google does state that click through rate is affected by seller ratings in their opt out form:

How long does it take for ratings from relevant ratings websites to show up in Google Products?

I can say from personal experience, it can take a bit over two weeks for ratings to start showing up. Of course, this may change over time as Google updates their systems.

If your company decides to pursue obtaining seller ratings, then customers will have to be directed to one or more of the ratings services used by Google. This is not too difficult to implement, and I personally would advise any e-commerce website to consider this task in their list of priorities.

Of course Google might decide to discontinue showing seller ratings, but until then it needs to be on the radar of any e-commerce website who relies on AdWords.

I hope this blog post helps answers someone’s questions as they look into Google’s seller ratings ;-)

Keep your World of Warcraft account from being hacked – Some tips to keep your WoW account safe

Wed, 06 Oct 2010 22:50:17 GMT

Unfortunately with World of Warcraft, it seems to be a very common problem that players find themselves with a hacked account. I think hackers like to attack WoW accounts because there is no legal repercussions from such an act. Unlike hacking a persons financial accounts, after hacking a WoW account, the hacker does not have to worry about the FBI showing up at their door.

Sure, WoW accounts are not as important as something like a banking website account, but players invest a lot of time in their WoW toons, and it would certainly be very unpleasant. Blizzard does seem to help players restore their items and gold lost, but I’m sure this is not guaranteed.

Keep in mind these are only tips to follow. Even following this advice does not mean an account is invulnerable, but every bit of vigilance helps.

Never install an interface addon that is an executable. Even if you think it is trustworthy, I would suggest it never be done as it is never necessary. World of Warcraft add ons are only text files and images that can be unzipped into the WoW Interface/Addons folder.
Never use your account user name or password for any other logins. Especially WoW related websites. In general you should be very careful not to use the same user name a password for critical websites like banking and other non-critical sites.
- To manage multiple logins and passwords, I recommend using a program like Password Manager XP or Password safe which is free.
- Never use a password that is in the dictionary or proper noun. Always include upper and lower case letters and least one or more digits in your password.
Run some sort of anti-virus package that has an internet security option.
Setup a Blizzard authenticator with your Battle.net account. Blizzard offers a mobile device version as well as a physical authenticator which you can purchase on the Blizzard Store.
- Keep in mind the authenticator is not foolproof. If you’ve been hacked with a key logger, the hacker can still get the code used to login to WoW, but it reduces the window of time they have to do it.
- The authenticator will help in the case a password and login is used on a malicious website as the hackers will not be able to get in without the authenticator code.
Never use an out dated web browser.
- If you are using Internet Explorer 6, then upgrade now. It is old and unsupported and more prone to be hacked.
- I personally recommend using a browser like FireFox with the NoScript add-on, Google’s Chrome, the latest Safari, or Internet Explorer 8.
Always be wary of emails with links. Even if an email looks like an official blizzard email, do not click on the links. Instead, go to worldofwarcraft.com or battle.net directly and navigate to account management to check your account status.
I’ve noticed a lot of whispers sent through WoW chat declaring your account is in violation. The whisper will declare your account will be shut down unless you visit a provided website address. Probably most players realize this is a scam, but it must work since it is being done.
Make sure you operating system is updated with the latest patches. If your OS is not patched, you are leaving it open to hacking attacks.
If you are a Microsoft Windows user and running Windows XP then I personally recommend upgrading to Windows 7. Although not perfect, Windows 7 provides features which make it more secure than XP.

Keep in mind, by taking these precautions the risk of a hacked account is certainly less, but not impossible. Always approach everything on the internet with caution and don’t trust much out there. These practices are not just specific to World of Warcraft either, it’ll help keep you safe on the internet in general.

If others have additional tips or ideas to share, please do. This is meant to help and I’ll try to update the list as I come across other precautions to take.

Happy questing :-)

Internet Explorer 8, compatibility view, and the local intranet zone

Thu, 23 Sep 2010 23:19:30 GMT

When developing a website on a local machine using localhost or some other non-public domain name, IE 8 will run in Local Intranet Zone mode. By default, Internet Explorer runs in compatibility view mode against any local intranet website. So, if your website does not expect to render in compatibility view mode, you may likely see the website render differently if you are running against the same website in Internet zone and Intranet zone.

You can change this setting as follows:

Open the Page drop menu on the IE 8 toolbar

Choose the Compatibility View Settings option.

On the Compatibility View Settings dialog uncheck the Display intranet sites in Compatibility View checkbox.

Close the dialog and now any website in the intranet zone will display as you might expect.

Initially when encountering this issue it was very confusing, so hopefully this helps someone figure out the issue more quickly :-)

A way to return a permanent 301 redirect result in ASP.NET MVC - RedirectResult returns a temporary 302 redirect

Thu, 23 Sep 2010 22:30:14 GMT

ASP.NET MVC 2.0 supplies a RedirectResult which returns a HTTP 302 temporary redirect. In many cases you really want a HTTP 301 permanent redirect result.

The following class will do the job:

using System.Web.Mvc;

/// 
/// Redirects using a 301 permanent redirect HTTP status code.
/// 
public class PermanentRedirectResult : RedirectResult
{
   public PermanentRedirectResult(string url) : base(url)
   {
   }

   public override void ExecuteResult(ControllerContext context)
   {
      base.ExecuteResult(context);
      context.HttpContext.Response.StatusCode = 301;
   }
}

I included this class in the TTC Tools which you can find on github.

It can be used in the same manner as RedirectResult.

What to check when ClientBuildManager.GetCompiledType returns null

Fri, 06 Aug 2010 22:53:45 GMT

The ClientBuildManager is a .NET BCL class that facilitates compiling ASP.NET markup files outside of IIS. I did run into an issue using this class which others may hit as well.

I compiled some ASP.NET MVC .aspx views and tried to get the type using ClientBuildManager.GetCompiledType, but the GetCompiledType function returned null. Hmm…I could see the compilation was working because the assemblies were seemingly generated correctly.

It ends up that the type would not load because the calling project did not reference all the assemblies needed to load the type. The ClientBuildManager compiles the web components in a separate application domain. When trying to retrieve the type, the assembly is loaded into the calling application domain, and then the type retrieval is attempted. If the type can not load no error is thrown, instead you get a null return value.

So make sure the application using ClientBuildManager references all assemblies referenced by the web application you are compiling.

I hope this saves someone some time. It took a bit of head banging to figure this one out. I had to debug down into System.Web to figure out what was going on :-(

Navigate from interface to implementation in Visual Studio using CodeRush

Mon, 02 Aug 2010 23:41:58 GMT

One feature of CodeRush from DevExpress which I use a lot when navigating code is the ability to navigate to a function/method/property implementation from a variable that is of an interface type. I do a lot interface programming (OK OK I get a bit interface happy) and this navigation feature rocks.

The default shortcut keys to bring up the Navigate/Jump To menu is Ctrl + Alt + N. I prefer to bind the Navigate command to Ctrl + E, as it is a bit quicker to type when your fingers are in home position on the keyboard. I show how to update the binding in the screen cast embedded with this blog post.

With the cursor on a function/property of a variable of some interface type, bring up the Jump To menu.

If you pick Implementations from the popup menu, you’ll be taken to the class implementation of the interface function/property. If multiple implementations exist, then a popup menu displaying the various choices will be displayed.

I created a screen cast to show this feature in action:

I’d suggest experimenting with the other Jump To options as well as they come in handy when navigating code.

I hope this helps!

Extract and implement interfaces in Visual Studio using CodeRush

Mon, 02 Aug 2010 23:33:18 GMT

A CodeRush feature that comes in very handy from time to time is the ability to easily extract and implement interfaces in Visual Studio.

This tip involves using the Refactor command included with CodeRush/Refactor Pro. The default shortcut key is Ctrl + ` . Personally I prefer to map the Refactor command to Ctrl + R for a more convenient key stroke. I show how to do the mapping in the screen cast embedded later in this blog post.

Extract an interface from a class

You can easily extract interfaces from an existing class. With the cursor on a class, bring up the Refactor menu and select Extract Interface.

This will create an interface that contains the public properties and methods of the class. You can then move the interface to it’s own file by using the Refactor menu once again and select Move Type to File.

Create an interface implementation

You can very quickly create and interface implementation. With the cursor on an interface, bring up the Refactor menu and select Create Implementer. You can choose to create an implicit or explicit implementation.

You can quickly move the implementation to a separate file the same way as shown above for extracting an interface.

I created a screen cast to show this in action:

CodeRush/Refactor Pro is a rock’in Visual Studio add-on from DevExpress.

I hope this helps!

Shortcut to popup intellisense in Visual Studio

Thu, 29 Jul 2010 22:13:32 GMT

Visual Studio tip: if the cursor is located after a variable followed by a dot without intellisense showing, press Ctrl + Space Bar to bring up intellisense. One common use of this shortcut is if the wrong property or function is chosen you can quickly select the property or function, delete the selection, and press Ctrl + Space Bar to popup intellisense to pick the correct one.

I created a short demonstration to show the shortcut in action:

I hope this helps!

Shortcut to popup function/method parameter help in Visual Studio

Thu, 29 Jul 2010 21:35:50 GMT

Visual Studio tip: if your cursor is located between the parenthesis in a C# function/method you can use use Ctrl + Shift + Space Bar to bring up the function parameter help popup.

I created a short video to demonstrate how the shortcut works:

I hope this tip helps!

Selecting text in Visual Studio with CodeRush

Sat, 24 Jul 2010 01:18:15 GMT

I created a screen cast to show how to use CodeRush’s SelectionExpand and SelectionReduce commands. This is one of those “I can’t live without” features that I use when editing code. These keyboard commands allow quick selection of blocks of code which speed up your code editing time.

Use it and code on!

Quickly change function/method visibility using CodeRush for Visual Studio

Thu, 22 Jul 2010 23:35:08 GMT

Using CodeRush from DevExpress you can very quickly change function/method visibility without typing the visibility text :-)

In a code file, if your cursor is anywhere within or on a function definition, you can use the left Alt key plus the up and down arrow keys to cycle through the various visibility modifiers.

I created a short screen cast to show it in action.

I hope this tip helps :-)

Visual Studio unit test runner using CodeRush from DevExpress

Wed, 21 Jul 2010 23:45:08 GMT

CodeRush is one of my can’t live without tools :-) I created a short screen cast to demo the unit test runner that comes with CodeRush from DevExpress.

I use it to run my unit tests at home and work :-D