On the surface, this seems to be a big win for all of us using or looking to use custom IDX systems, but can anyone see any negatives to this new rule?

__________________
Steve Castaneda, Realtor
Keller Williams Realty
The MyHomeHouston Team

Specializing in Houston Real Estate. Feel free to visit my Houston Real Estate Blog or read about Real Estate Technology for agents.

If your plan is to access the RETS interface directly for searches on your website versus using a framed interface your MLS may provide, then the negatives are:

1. RETS only standardizes the transport and exchange of data. It does almost nothing to standardize the format of the data. I've seen where the MLS converts their poor-quality data from CSV format to RETS, which means they've only changed the wrapper and delivery of the same poor-quality data. In other words, you can't purchase/develop a RETS client interface and expect to be able to interface to any MLS. You still need to customize your application to the data format your MLS uses. (i.e. column names, attribute codes, etc.)

2. For those that "consume" (import) an IDX feed, RETS is far less efficient a transport protocol than say, FTPing a zipped CSV file. Expect to use more bandwidth and processing power to consume a RETS-based feed.

3. Because of #2 above, many RETS implementations restrict the number of records that can be retrieved at one time during "business hours", which is quite silly. You have to do multiple calls to the server to pull the additional records, creating more load on the server than if you had pulled them all in one call.

4. Without getting into the details, the way photos are retrieved via RETS and redisplayed is programmatically quite complicated and inefficient.

5. Different RETS vendors can and do add "extensions" to the RETS protocol. Therefore, to get the full capabilities, you still need to program additional features beyond the RETS "standard" to get the full benefit.

From the perspective of the IDX-feed consumer, the only possible benefit that I can see that RETS offers is the ability to remotely search the MLS database and retrieve a subset of records in real time, without having to maintain your own local mirror database of listings and photos. However, the amount of code that needs to be written to retrieve the listings and photos and reformat them for display on a web page in real time to your site visitors seems to be a dubious proposition. Maintaining your own database from daily FTP feeds of text files would seem to be more reliable, efficient and offer better response times to your site visitors.

I'd love to hear from other developers' experience implementing a RETS interface. Also, given all the smart people in this forum, I'm sure someone will be able to find some real benefits that RETS brings to the consumer over "traditional" IDX feeds even though I can't think of any.

Paul Cormier

__________________
Paul Cormier
WinCorp Software, Inc.

http://www.BestHomePro.com

I think it goes a little further than you're aware. Right now, RETS defines StandardNames for fields for interoperability between different RETS servers. While I'll certainly admit that these aren't used as much as they should be, they do exist. The upcoming Schema payloads for RETS take this concept to a whole new level.

Even with that in mind, this isn't a negative over FTP because you have to do these things anyway. Right now, it's just not as big of a positive as it could be, but, like I said, with the upcoming Schema payloads, it will be a lot better.

When you consider everything that FTP doesn't do, it's a very acceptable trade-off. The additional bandwidth and processing power is minimal when you consider the benefits.

For example, an MLS can make their RETS server have complete access to their database including sold listings and other things IDX vendors aren't typically used to seeing. When Joe IDX logs in, the RETS server can apply a "master" query to your searches so you only receive Active listings, while Joe Broker could be logged in with more access to everything on the server at the same time. The RETS server can also limit what fields certain types of users get back in their results so IDX vendors feel better about the data they're getting (i.e. don't give me lockbox codes, please).

I think this is for a different reason and goes back to my #2 response. When you have an MLS that gives certain types of users the ability to query and return hundreds of thousands of records at a time, there needs to be something in place to make the system responsive in a decent amount of time. Using the RETS "Offset" parameter isn't much different from an SQL "Limit" parameter.

Also, I think most MLSs would agree that the "preferred" way for IDX vendors to use RETS is to pull a full, daily RETS download and then, throughout the day, run incremental queries checking for new updates since your last query (maybe once every few hours). That allows you to offer your site users quicker information about new homes or changes to homes on the market. Since you can't tell what's been deleted, the full download once a day is necessary so you can essentially start over.

I think I disagree with this. In your GetObject query, you get back either a URL to an image or a multi-part response with the actual image data for several images. I've only run into 1 MLS that didn't provide the photos directly in the RETS GetObject response and it wasn't that hard to work with.

Again, being able to pull individual photos due to incremental queries throughout the day is a very powerful feature. I've worked with several plain FTP IDX feeds that didn't provide daily zip files of photos and I had to retrieve them manually (often doing incremental guesses at filenames rather than having something tell me that, yes, -this- photo exists and -this- is where it's at).

Getting all of the images back for a listing within a single, multi-part response could also be seen as increased efficiency. The photos are meant to be downloaded to a local machine and not served directly from the MLS.

Keep in mind that this isn't a negative. If we're still talking about the negatives to IDX vendors, this has no impact. Often times these extensions are for statistics and reports that the MLS can provide to RETS users and these aren't the types of things IDX vendors are used to getting via FTP anyway.

In my opinion, it's best if you -do- maintain your own local copy of the database. One thing is that RETS doesn't provide a sorting mechanism. If you pull a full download each night and an incremental download a few times a day, you serve all of your website's users off of your local copy which is a lot faster and it allows you to provide more data in interesting ways (a RETS server can limit which fields you can query on where your local database wouldn't).


Some of these things may seem a little tougher to do at first, but with the right tools and understanding, they are easily re-used between different RETS servers so it certainly doesn't go to waste.

__________________
Troy Davisson
http://blog.troydavisson.com/rets

Troy, thank you for the additional insights. BTW, I have implemented a RETS client interface from scratch, so I am quite familiar with the standard and its practical implementation. Here are a few additional comments:

1. I was aware of the StandardNames, which unfortunately hasn't yet been leveraged enough. It is certain that the RETS standard will continue to evolve towards the standardization of the data format in addition to the querying and transport of the data, but unfortunately, we're not there yet.

2. RETS vs FTP really has nothing to do with what is contained within an IDX feed. Sold data, limited columns, actives-only, etc. can all just as easily be done via custom FTP feeds as it can using RETS.

3. You seem to concur on the preferred method being incremental updates of a local database of listings. There really are no advantages to pulling uncompressed RETS data vs FTPing a zipped file other than the ability to do granular incremental updates via RETS. That's a great feature, but like you mentioned, RETS lacks the ability to determine when listings are specifically no longer active. However, the uncompressed RETS data would easily be orders of magnitude larger in size than the same information ZIPped. It could be argued that on-the-fly gzip compression could be used, but again, would require more processing power to deflate than a zip file.

4. The methods available of photo distribution all-around aren't very smoothly executed. Unfortunately because the RETS standard doesn't include content-length headers for each section of the multi-photo retrieval, it makes it very difficult (and actually impossible in some environments) to attempt to parse out the binary photo data from the text separators. I still think it's easier to FTP download a zipped file of 20,000 photos and extract them to a local directory than it is to make 20,000 calls to a remote server and individually parse the responses to save the files locally.

5. I'm not sure I correctly understood your statement:

(a RETS server can limit which fields you can query on where your local database wouldn't)

You can formulate a SQL query on your local database to retrieve only certain fields just as you can with RETS.

Though there are some aspects of today's RETS standard that could be interpreted as being an improvement over the traditional FTP feed, there are also some aspects that are worse, and other aspects that are simply different. I would really like it if there were actually benefits for RETS over FTP that could be enumerated like "easier to implement" or "faster data downloads" or "standardized data formats", but I just don't see the benefit to the data consumer of "different".

I think the overall promise being made of RETS, "the easy transfer of data between systems" really is not possible today unless that transfer of data means only the handful of the most basic parameters with StandardNames.

BTW, I'm not a big sour-puss on RETS. There are MANY benefits from the MLS side to implement RETS. I support and encourage the continued development and evolution of the RETS standard so that one day, it can achieve what it promises. Unfortunately, it has a ways to go yet in creating a benefit for the data consumer.

Any other developers out there that have implemented a RETS interface? I think everyone can benefit from our experiences, so please comment.

__________________
Paul Cormier
WinCorp Software, Inc.

http://www.BestHomePro.com

Again, if we're talking about negatives of RETS over something like FTP, you're completely blind processing a FTP download where at least StandardNames give you something to work off of. How much of an advantage RETS gives you in this specific case depends on how often they used StandardNames. If they didn't use them at all, you don't get any advantage and a developer has to sit and do more data mapping which is the same thing that happens now via FTP.

But how often do you see that? I've maybe seen 1 feed where the download was somehow specific to my use and even then, it included fields they didn't really want me to have. Also, don't forget the live data access part. This is important for many brokers so their agents can have up-to-date information and can provide a timely service to their customers. If someone uses your service and knows about a house a day before someone that uses my service, that's a big deal.

I don't know specifics between gzip and zip and why one would require more processing power than the other, but we're talking about decompressing plain text which is negligible unless you're on a really old system. Also, you can specifically choose which fields you want returned in the results so you can further decrease the size by excluding data you'll never use.

The RETS server can limit what fields you search on (the metadata has a Searchable boolean value). If you do a more "open" query for the RETS data and save it locally, you can query whatever field you want on your local system.

When you're in a position of setting up dozens of IDX feeds, you can absolutely appreciate the interoperability of RETS. RETS could have just stopped and tried to standardize an FTP download but instead continued on to the transport of it which takes care of a lot of the steps involved.

Yes, if you're just implementing RETS on 1 or 2 feeds, it may not seem worth it, but the ability to reuse that same code across many MLSs (and with the mandate mentioned in the OP, many many more soon), that's a huge advantage by itself. There are free programs and libraries available (both open source and not) that can take care of almost every RETS related piece of that process.

As you mentioned, there are a lot more advantages on the MLS side but there are advantages on the client side as well.

Among other things, RETS gives you access to both the data and the format of the data. Pulling metadata information allows you to see, automatically, when fields have changed and to adjust accordingly. If you're suddenly getting a new TotalRooms field (and your system can use it), it can automatically know how to handle that and what to do with it. Maybe it makes a new field in your table and saves the data so that your program can automatically use it. I can't even tell you how many times I've either gotten printed field format information sent to me in the mail or in PDF/Word format via email. Not only do you have to re-key all of those field names into your tables (because the people that send you the printed copy can never be the "easy" ones and include less than 10 different property types) but most of the time (from my experience), the documentation is outdated anyway and it doesn't reflect the data you're getting and it certainly doesn't help that the data itself doesn't include a first line of field names for reference.

__________________
Troy Davisson
http://blog.troydavisson.com/rets

a developer has to sit and do more data mapping which is the same thing that happens now via FTP

I agree: different, not better.

Also, don't forget the live data access part. This is important for many brokers so their agents can have up-to-date information and can provide a timely service to their customers. If someone uses your service and knows about a house a day before someone that uses my service, that's a big deal.

I totally agree that granular intra-day updates is an advantage that RETS holds over FTPed IDX data.

but the ability to reuse that same code across many MLSs (and with the mandate mentioned in the OP, many many more soon), that's a huge advantage by itself.

What you're stating is the dream of RETS, not the reality. Because of the incomplete use of StandardNames, the task of Data Mapping still exists whether the transport is RETS or FTP. Sure, you can use 3rd-party RETS clients to pull the data, but because data mapping is required either way, it's only "different" not better. It's more likely that the consolidation of MLSes will help with standardization more than the mandated implementation of RETS. (How many MLSes will simply port their existing data structures to RETS vs restructuring to StandardNames?) I've implemented 8 MLSes so far and no 2 have been the same. (Not even close.)

Pulling metadata information allows you to see, automatically, when fields have changed and to adjust accordingly. If you're suddenly getting a new TotalRooms field (and your system can use it), it can automatically know how to handle that and what to do with it.

I agree with you that access to the MetaData is a significant advantage over how most MLSes currently provide that information. However, it would be a pretty cool trick to have your code automatically know when and where to display a new field. Unless you plan on implementing some AI to have your code automatically understand what "TotalRooms" means, you still have to write some code to use and display the contents of that field. Again, if an MLS is meticulous in informing consumers of an FTP IDX data structure change, doing it via RETS MetaData is just different, not necessarily better.

__________________
Paul Cormier
WinCorp Software, Inc.

http://www.BestHomePro.com