Cliff Niersbach NAR response to Google spidering listings in Mibor
Many of you read over at Agent Genius about how MIBOR (The local board of Realtors in Indianapolis) has demanded that all IDX vendors stop Google and other search engines from spidering any page of a Realtors website that has IDX listings or any part of the data referencing listings (such as property descriptions, MLS #'s or addresses) - this is with the exception of just the agents own listings of course.
Naturally as one of the largest custom IDX vendor in the world we are more than concerned about this issue as should all Realtors as it has the potential to dramatically impact their ability to get their listings competitive exposure unless they go through a third party aggregator such as Realtor.com and pay through the nose for premium advertising.
Also as the IDX vendor for Paula Henry http://www.hometoindy.com (the agent who first blogged about this) and the IDX vendor for her broker Mike Taylor http://www.reddoorindy.com I have first hand knowledge of the situation, in fact I am the person at REW that was handling the communication back and forth between NAR, REW, our client and MIBOR.
In the end - both NAR and MIBOR failed to see that the complaint lodged and the decision they were making had incredibly serious consequences that only served to do harm to their members and the listing clients they represent.
Over the next few days I am going to be documenting the communication between myself, NAR, MIBOR and my client (I have already spoken to him over the phone and he supports me in this decision) in hopes that we can raise enough awareness regarding this issue and have it nipped in the bud before this local (board level) situation spreads to other ill informed / uneducated boards.
"Mr Nierbach (if you would be so kind) can you please address
"Google" specifically and your policy regarding MLS listings
Mr Renkert has asserted that it is your position that Google may not spider any MLS listed property - is this in fact correct?
He also asserts that it is ok for Google to have MLS listings spidered as long as the broker has placed the listings on the site being spidered (example Trulia) which is a contradiction. It sounds like he does NOT have a problem with Google indexing listings rather he wants to be able to dictate how they get there.
The bigger issue (I beleive) is whether or not this policy (as it pertains to the single largest source of internet business for brokers - ie Google) has been interpretted for this particular situation. It most certainly is not being universally communicated or enforced as evidenced by the hundreds of thousands if not millions of agent websites, broker websites, IDX vendor websites and third party listing sites that currently allow (or rather encourage) Google to visit all of their inventory pages in an attempt to drive their traffic to the listing they have searched for.
So again - if brokers ONLY wanted their listings to be able to be accessed at thier website (this is the road we are going down) why have IDX in the first place?
I do hope this issue can be discussed (and resolved) quietly as I really think that it has the potential to become something a lot bigger than it needs to be. The solution - (rather a solution) do not consider major search engines (Google, Yahoo, Live etc) "scraper sites" (because they truly are not) and thus the rule being quoted would not (and should not) apply. "
To which mr Niersbach replied
"Good afternoon Mr. Carey:
I respectfully advise NAR policy does not expressly focus on Google or any search engine operator in particular but rather on the rights and obligations of MLS Participants when they are afforded the privilege of advertising other Participants' listings pursuant to the IDX policy.
Real estate brokers are prohibited under the Code of Ethics and under the licensing laws or regulations of most states from advertising other brokers' listings . Under IDX, listing brokers can, but are not required to, give other participants authority to advertise the listing broker's listings on the other participants' websites. Absent the listing broker's consent, other participants have no authority to advertise those listings in any medium, including the Internet.
When an MLS Participant who is not the listing broker, pursuant to a grant of authority from the listing broker, advertises one or more of the listing broker's listings on their website, he or she does so subject to the IDX-specific rules of the MLS which they participate in. Those rules are based on the IDX policy of the National Association of REALTORS®.
The IDX policy provides, in relevant part, that "Participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent 'scraping' or other unauthorized accessing, reproduction, or other use of the MLS database." This is the provision referenced in my earlier correspondence with the Metropolitan Indianapolis Board. Simply put, when an MLS participant advertises another participant's listing pursuant to IDX, the participant doing the advertising must take reasonable steps to prevent that other participant's listing from being scraped. It is important to note that this requirement applies only to the listings of other participants. The website operator is free to do whatever he or she chooses to do with his or her own listings, including allowing their own listings to be scraped, published on third party aggregator sites, etc.
Cliff Niersbach "
To which I responded
Thanks for your response - I am in fact aware of the IDX policy and specifically to the prohibition of advertising another brokers listings anywhere but on the IDX approved brokers website. I am also aware of the requirement of us (the IDX vendor) to employ appropriate data protection measures to ensure that data is not "scraped" - by "scraper sites"
The "NEW" arguement here seems to be that Google (a search engine) is being lumped in as a scraper site - and our contention is that Google is NOT a scraper site, rather a technology (just like a browser) with a significant user base with which the public access websites. Google simply tells the user where to go - they are not creating web pages of their own out of the data they index. I am sorry, but I just simply do not see how NAR could consider a search engine to fit under the same designation as a "scraper site" -
And then Cliff from NAR replied with
"Good afternoon Mr. Carey:
As noted in my earlier email, the NAR policy requires that participants displaying other particpants' listings on their websites make reasonable efforts to prevent scraping. It does not limit the obligation to prevent scraping to scraping done by entities you refer to as "scraper sites", but rather applies to scraping in general.
I believe the issue isn't whether MIBOR is characterizing Google or any other search engine as a "scraper site". Rather, the issue appears to be whether any information provided by other participants and displayed on an IDX site can be scraped by anyone.
I hope this responds to your request.
And then finally - this exchange:
Good evening Mr. Carey:
Please find my responses to your questions and comments below. I appreciate your follow-up and the opportunity to respond.
By copy of this email, I will share your concerns with the REALTOR® leadership of the NAR Multiple Listing Issues and Policies Committee. I will also review your comments and my responses with the head of the Center for REALTOR® Technology ("CRT") and with the NAR General Counsel. If they have different opinions, I will follow up with you and with the Metropolitan Indianapolis Board.
Perhaps I am not phrasing my question correctly - I will pear it down to as simple a form as I can:
Does NAR consider "indexing by search engines" to be the same as "scraping"
Response: The CRT advises that "indexing" is a form of scraping.
Because if it doesn't then the rule being cited does not apply.
We at Real Estate Webmasters do not feel that it is the same thing nor does our customer Mike Taylor (NAR and MIBOR member)
A "scraper site" stores and publishes content on their domain generally for the purpose of misleading or profitting from the publication of specific data that has been manipulated (generally) in some way.
A "search engine" indexes the location of data and "directs users to the original author of the content" which is not advertising, or malicious in any way. Nor is there any manipulation of data in the indexing process, it's simply a way for that particular service (in this case Google) to answer the question of "where is x" for it's users.
Response: NAR's IDX policy doesn't distinguish between "malicious" scraping and scraping that might (for lack of a better word) be described as non-malicious or even beneficial.
This is an incredibly important issue to address as we feel that the particular rule being cited out of context and is being used innappropriately as a "catch all" and interpreted and enforced arbitrarily to discriminate against our client. This board is NOT enforcing their "rule" against all members equally nor is NAR enforcing this rule (with respect to search engine indexing) anywhere else on the internet (there are literally hundreds of thousands of NAR member websites with IDX's that are indexable by search engines) - are you saying that all of these websites must be changed?
Response: The Metropolitan Indianapolis Board has a long and deserved reputation for integrity and fairness. If a REALTOR® has concerns about MIBOR's rules, or how those rules are enforced, the appropriate step is to engage the staff and REALTOR® leadership in constructive dialogue.
The fact of the matter is - NAR has an obligation to enforce it's rules fairly and accurately.
Response: Enforcement of MLS rules is the responsibility of local associations of REALTORS® and their MLSs, NAR's role is developing policies and procedures that permit MLSs to serve their participants in effective, efficient and legally defensible ways.
We understand that and take no issue with the "protect against scraping" clause - in fact we agree with it and actively participate in the defense against scraper sites (we don't like being scraped either)
NAR has an obligation to it's members to protect their rights and freedoms and ensure that they are treated fairly and equally and defend against discrimination (one issue at stake here)
Response: NAR understands that responsibility. NAR's concern for its members is manifested in the body of MLS policy that has been developed and implemented over the past four decades.
NAR finally and perhaps most importantly has an obligation to protect the interest of the seller. To deny a seller representation of their properties in search engines is to deny them one of the most significant opportunities to sell their home via a qualified professional. If NAR were to make the huge mistake of stating that search engines should be blocked from indexing IDX participant websites then the only results left (because it won't stop users from searching for them) will be the very scraper sites that you are trying to defend the public against be exposed to.
Response: I respectfully disagree with this conclusion. The basis of seller representation is the exclusive listing agreement that sellers enter into with listing brokers. It is those listing agreements that spell out whether the listing will, for example, be submitted to MLS; whether information about the seller's property will be submitted to third-party aggregator sites like Realtor.com; whether other brokers will be afforded the courtesy of displaying information about the seller's property on their sites, etc, As a practical matter, the fact that a search engine cannot index or scrape information a listing broker allows to appear on other brokers' sites to then direct consumers to those other brokers' IDX sites doesn't alter the fact that consumers viewing those sites will have access to every listing on those sites. The purpose of the IDX policy is to give listing brokers a method to consent to display of their listings by other participants on their websites. Absent such consent, advertising (whether on websites or otherwise) by other brokers is prohibited by the Code of Ethics and by the license laws of many states.
But websites that scrape content for their own gain or malicious purposes are not the same as search engines and should not be mentioned together in the same sentence - they are different and this needs to be acknowledged by NAR. I humbly request that you consider this issue and offer a final decision in this matter allowing participants to have their websites spidered by search engines if they so choose.
Response: I can only reiterate that NAR's IDX policy mentions neither websites that scrape for their own gain or for malicious purposes - or search engines that scrape for other reasons. I agree "scraper sites" and search engines are different - and must again point out that neither has any authority to scrape listing brokers' information from other brokers' IDX sites.
That said, as indicated above I will review your concerns with the head of the CRT and the General Counsel and if I have misstated the IDX policy in any way, I will respond to you and to the MIBOR. I hope that you and the leadership of the MIBOR realize that NAR staff has no authority to modify policies established by the NAR Board of Directors. If the IDX policy - or, for that matter, any other NAR policy - requires clarification or amendment, that would require action by the Board of Directors.
Please let me know if you have other questions.