First IDX / MLS Project

[IronEdgeHosting.com]

Hi Everyone,

I'm working on my first IDX / MLS project and figured I'd post here before I got too deep into it.

The initial project is to provide live enriched listings for a couple of real estate firm clients. It's going to start off small with just live for a few realty sites listings and eventually grow to encompass a lot more.

I'm deploying on one of my dedicated Windows 2003 servers. Probably will use Microsoft Sql Server 2005 for the database and the application will be written in .Net/ASPX/C#

I'm receiving FTP feeds from North Florida MLS and I wrote a job to uncompress them into data CSV files.

The first issue is the data structure itself. The schema supplied by NFMLS has a huge number of columns in each of the data structures. 300+ in the smallest one. I'm used to working with highly normalized schemas with a lot of referential integrity. The data feeds I'm getting from NFMLS are giant denormalized basically flat files. Don't know what solutions you've come up with but tables with hundreds of columns don't seem like a good idea. And in the case of Sql Server they exceed the max row size by a fair bit. I'm not opposed to moving to MySql if that database would handle these huge row widths better.

The client isn't going to want to spend the money required for me to normalize this whole thing so I was looking for a simple fairly quick solution.

My one idea was to pre-parse the MLS data and cherry pick the 50 columns of interest into a tighter table containing the information that you'd need 90% of the time and then maybe stream the other stuff into XML and stuff the XML doc into a blob and then run XPATH queries on the XML DOM to present the more obscure stuff (like if you can waterski behind the place or other nonsense like that)

The only problem is that this solution boxes me into the handful of columns that I pick as important right off the bat.

If anyone out there has any ideas I'd love to hear them.

Thanks for your time.

[webmaster]

You definitely want to handle the data with a parser BEFORE you create your database so that you can get rid of all the crap, and normalize the data as much as your budget allows.

Without getting to much into it, we typically leave a lot of the database structure in tact (The parts we kept after parsing) and then we have a complex config that is quite standardized that makes dealing with creating the solution (Front end) a lot more palpable. you could also attack it with a series of abstraction layers blah blah blah

In terms of mssql or small and limp (Oops I mean microsoft) can't help you there, all REW developers are LAMP so we don't run into the same issues.

[webmaster]

NFMLS is Jacksonville right? http://www.kristafracke.com/idx/ this one?

[IronEdgeHosting.com]

Yeah I think thats really the only answer. I'm going to cull the data to a reasonable table size but I think I'm also going to convert the entire data structure into XML and store it in a blob table. This way if anyone wants to expose that data we can get to it with an xpath query.

All the REW developers are LAMP? Not anymore. I try to use the best technology available and not engage in group thinking or whats popular. To say the Microsoft solutions are slow and weak is just exposing your ignorance. For example the MySql max record length is basically identical to Sql Server record length. So the same problem exists in both databases. The Linux + MySql solution is def the low cost leader. Does that make it the best solution? In some cases. You don't see any bank databases running MySql they do however use Sql Server...

On the other side of the religious debate the hard core OO Java, C#, C++ guys would call PHP a script kiddy language designed for noobs with no coding skill. Yet PHP runs some of the highest traffic sites on the internet. So it must be more than a toy language...

Who is right? the one that doesn't take an extreme position and understands that all technologies are a series of tradeoffs. I have one project that has a MySql database, an open source Hibernate ORM with a ASP.NET/C# front end. I like it because it horrifies the Microsoft and Linux fanboys equally.

You definitely want to handle the data with a parser BEFORE you create your database so that you can get rid of all the crap, and normalize the data as much as your budget allows.

Without getting to much into it, we typically leave a lot of the database structure in tact (The parts we kept after parsing) and then we have a complex config that is quite standardized that makes dealing with creating the solution (Front end) a lot more palpable. you could also attack it with a series of abstraction layers blah blah blah

In terms of mssql or small and limp (Oops I mean microsoft) can't help you there, all REW developers are LAMP so we don't run into the same issues.

[BrandonSternfield]

M$ and Tux don't like each other. It's going to be that way until one buys the other, or one burns down. Either way, the consumer is the one that benefits from such a degree of competition.

[RonnieG]

NFMLS is Jacksonville right? http://www.kristafracke.com/idx/ this one?

Wow! Small world blast from the past! Krista was my next door neighbor, and a stay-at-home housewife, when I had just started in real estate. I talked her into trying real estate, though she had considered it already, and I understand she has done pretty well in her first few years. A few months after she got her license, we moved from Florida back to Colorado, and she eventually found the buyer for our Florida house.

The Jacksonville area MLS is actually NEFMLS, a service of North East Florida Association of Realtors (NEFAR).

[IronEdgeHosting.com]

I kinda ride the fence on the whole technology thing and I try not to get into the religious debate. We work on so many different projects in different technologies I see advantages and warts in all of them.

This blatent Microsoft sucks thing that is so pervasive in the nerd world bugs me. It's just not true. Windows 2003 is a great OS and just as reliable as our Linux servers and they are a whole lot easier to configure and maintain.

We have a mix of Linux and Windows servers both in critical roles. They are both solid, both perform well. Linux is more labor intensive and Windows costs more. The real advantage that I've seen with the Windows servers are that they can run all of the Microsoft stuff and all of the open source stuff. We have a client with a mix of; MySql, Sql Server, ASP, PHP, and ASPX all on the same server. Can't run all of that stuff on Linux.

[BrandonSternfield]

Personally, I have macs, windows, and linux in my house. Each has their use to me though

[webmaster]

Lol - I didn't mean we don't run into issues being LAMP, I meant we didn't run in to the "same" issues. I actually have nothing against microsoft, I just think small and limp is funny, I heard it somewhere and have never not been able to write it

What I meant by all REW developers - I was referencing REW's staff, not all members of this forum - your not any more led me to believe you thought I meant everyone here, I just meant everyone on my staff.

[IronEdgeHosting.com]

One of the Microsoft fanboys that works at one of the client sites I visit reguarly knows we have some Linux stuff and my background is Java/J2EE by spounting off "Java Sucks" every time he sees me. I'm all; "well you love C# so much it's a direct Microsoft copy of Java it must not be that bad."

He knows I'm not bigoted in either direction but he is always trying to bait me.

Lol - I didn't mean we don't run into issues being LAMP, I meant we didn't run in to the "same" issues. I actually have nothing against microsoft, I just think small and limp is funny, I heard it somewhere and have never not been able to write it

What I meant by all REW developers - I was referencing REW's staff, not all members of this forum - your not any more led me to believe you thought I meant everyone here, I just meant everyone on my staff.