Index pages for Realtors - what you need to know for great SEO
Before I get into recommendations for how to ensure and maximize indexing of pages (especially low value pages & IDX pages) I would like to clarify a few things.
#1: Unless “specifically blocked” Google is able to index (store a copy of) any “built in” IDX system on the market. For a system to be “built in” it means that the IDX and website vendor reside on the same platform and a dynamic (data base driven) system is outputting pages that appear as “static” (html pages) to Google. REW (Real Estate Webmasters) is one such system, the very first, and most advanced system of its kind.
#2: Google “indexes” pages (becomes aware of them) in one of only 2 ways. (A) they follow links to your pages (they crawl) or (B) they are provided with page locations via the Google Webmasters Tools (GWT) xml site map initiative.
#3: Just because Google “indexes” your pages (or is able to index them), does not guarantee the pages' inclusion in the index (short or long term).
#4: Too many indexed pages is “ALMOST ALWAYS” a bad thing.
I am sure that folks are going to want to discuss #4 first, so let’s deal with that.
The important take away from this article on indexed pages is that having “indexed pages” by themselves is not a good thing. It’s only a good thing if those pages that you are able to “get indexed” are also able to “get ranked”.
If indeed pages are too numerous, not only will they not be able to get ranked, they can impact your high value pages' ability to get ranked by diluting the available “PageRank” to each page.
Here is a simple
Pagerank is a ranking factor used by Google in order to determine a portion of the authority of a domain or a specific page.
Without enough “PageRank” even if a page is in Google’s “index” it will never be selected as a preferred result to show users after the search for something. (And let’s call “never” - nothing on the first couple of pages of results. Nobody looks past that point - unless it’s you looking for where your pages rank of course :)
A domain has a certain “finite”
amount of Page Rank that is received from external links pointing
to your domain. You can distribute this page rank to your internal
pages (by linking to them) in any way you wish, but no matter what
you do, you will never be able to create “more” page rank than your
maximum internally. (I could get into a discussion of how to
augment your page rank using internal linking, but it is far too
complex for this article and not necessary to understand the
So for now, just trust me - no matter what you do, unless you go out and get additional page rank from additional sources (or the page rank of your existing additional sources increases) you have a maximum that you can distribute to your pages.
Now let’s talk about rankings. In order for a page to rank, it needs a lot of things not related to page rank (on-page factors) such as titles, filename, on-page content, spiderable html, etc that pretty much all “built in” IDX vendors have (though some are better than others) but what a page also needs in order to be able to rank is “PageRank” (you know, that stuff you only have so much of?).
Pagerank is “shared” between all pages. If you have 100 points of pagerank and 100 pages linked to - each of those pages gets 1 point (well technically .85 points due to the dampening factor but don’t worry about that).
So here is some simple math (used for educational purposes only - there are more complex models to be discussed when it comes to how pagerank is actually distributed).
If you have 100 points of “PageRank” and you are in a space (competition wise) that requires 10 points of Pagerank (assuming all other factors are present) in order to rank - you can legitimately rank….. You guessed it - only 10 pages!
So how many pages should you link to internally / have on your site? (and have in the index?)
You guessed it, only 10!!! You just don’t have the power to rank any more.
But what happens if you create 100 pages, or 1000 pages? You guessed it, your pagerank goes down to 1 point per page or even .1 points per page. At 1 point, you “might” have enough power to keep your pages “in the index” but we already know you need 10 points to rank, so those pages are not ever going to be seen.
At .1 points per page, even if you DID manage to get those pages spidered / indexed initially - with such low pagerank, Google will eventually purge those pages from its index as they are clearly not important enough to keep. Noone (not even you) is linking to them in any significant way and they simply don’t have any juice.
^^^This is what is happening^^^ to Realtors who are trying to get thousands (or tens of thousands) of pages indexed in Google leveraging IDX data.
They have a quick and easy way to generate thousands of pages of dynamic content, it’s totally spiderable so they map / submit it all in the hopes that they will get the traffic from it. Google comes along, initially spiders these pages (so they show in the index) but then over time and in comparison to other similar content already spidered with higher authority - Google just sees these pages as insignificant and removes them from the index. You see a spike, then you see a drop over time.
^^^This is especially true with “new” sites^^^ where Google gives them an initial boost in authority (almost a "this is where you would be if you had lots of juice" preview). This boost to “new” websites has been observed and reported thousands of times across many different verticals including real estate.
Now let’s talk about YOU and your authority.
We will use a log 10 scale in order to demonstrate pagerank numbers (actual thresholds are a tightly held Google secret) - but this is relatively accepted as a metic.
Pagerank 1: 0-10
Pagerank 2: 11-100
Pagerank 3: 101-1000
Pagerank 4: 1001-10,000
Pagerank 5 10,001-100,000
Pagerank 6 101,000-1,000,000
There are no legit Realtor websites out there with a pagerank 7 or above so no need to continue to 10.
Now for most of you reading this article you will fall into a pagerank 1-4 so you have 0-10,000 points to distribute (very few will actually have 10,000)
You are targeting keywords with varying degrees of competition:
Main keywords, example “area real estate” or “area homes” all the way down to medium to low competition keywords, example “property address x city” via a blog post or IDX page.
Your main keywords are hugely competitive (though there is some variance from market to market obviously). So let’s just say you need “at least” 100 points in order to rank for those terms (that is 100 points EACH PAGE).
For your low to medium competition keywords you likely need 10-50 points “per page” in order to rank those pages.
So if you are one of those lucky 1000-10,000 point Realtors, you can rank 10-100 pages effectively (but no more than this). Or if you have 10 point competition you can rank 100-1000 pages.
Hopefully this makes sense.
Now here is where “too many” linked pages hurts you.
Let’s say you do have 1000 points and your competition really does require 100 points “per page” in order to rank.
If you link to just 10 pages, each of those pages should rank very well because they each receive 1000 / 10 or 100 points each.
But what happens to the rankings for those pages if you add just 2 more links to the page distributing the Page Rank? Your 10 becomes 12 and 1000 / 12 = only 83 points (this is not enough to rank for your money terms!).
You’ve lost rankings for keywords that potentially bring thousands of searches - maybe you dropped to #3 or #5 on the first page.
Add 5 more pages, now you have 1000 / 17 = 58 points, whoops, you just dropped right off the first page altogether.
Ok - so you have 1000 points right? Now add 20,000 IDX listings - at .05 points per page do you think Google will consider “any” of those pages worthy of keeping in the index? Absolutely not. But worse yet, you sacrificed all your good ranking pages (which brought actual traffic and leads) because you worried more about a number (indexed pages) that does not have any real value by itself at all!
Hopefully this simple example helps you understand why having “more indexed pages” can actually hurt you, not help you.
But wait! You sell spiderable IDX and you advocate getting IDX pages spidered don’t you?
Yes I do, and of COURSE I do.
I do because I understand that there is a way to leverage true value from thousands of indue pages (like Zillow and Trulia do) - the way to do that is to grow your indexed pages in tandem with your Pagerank.
Go back to our example of 100 points being needed for ranking competitive pages and 10 points being needed to rank non competitive pages.
Now build your authority to 1,000,000 weight points or more!
That’s 10,000 legitimately competitive pages all ranking!
That’s 100,000 low competition pages ranking!
So the part you are missing (pretty much ALL of you) is that you have not developed enough authority to your websites (which is hard and takes time) and you added way too many pages to your site too early (because it was quick and easy) and now NONE of your pages are ranking, and many of those initially indexed pages are dropping out.
Do yourself a favor - reduce your pages to ONLY the property types, price ranges and MOST important neighbourhoods you focus on and go out and BUILD YOUR PAGERANK.
Don’t worry, your other traffic can find the rest of the listings via your search, all the properties are still in there.